This paper introduces an automated skill acquisition framework in reinforcement learning which involves identifying a hierarchical description of the given task in terms of abstract states and extended actions between abstract states. Identifying such structures present in the task provides ways to simplify and speed up reinforcement learning learning algorithms. These structures also help to generalize such algorithms over multiple tasks without relearning policies from scratch. We use ideas from dynamical systems to find metastable regions in the state space and associate them with abstract states. The spectral clustering algorithm PCCA+ is used to identify suitable abstractions aligned to the underlying structure. Skills are defined in terms of the transitions between such abstract states. The connectivity information from PCCA+ is used to generate these skills or options. The skills are independent of the learning task and can be efficiently reused across a variety of tasks defined over a common state space. Another major advantage of the approach is that it does not need a prior model of the MDP and can work well even when the MDPs are constructed from sampled trajectories. Finally, we present our attempts to extend the automated skills acquisition framework to complex tasks such as learning to play video games where we use deep learning techniques for representation learning to aid our spatio-temporal abstraction framework. Proceedings of the 33 rd International Conference on Machine Learning, New York, NY, USA, 2016. JMLR: W&CP volume 48. Copyright 2016 by the author(s). 1. Motivation and Introduction The core idea of hierarchical reinforcement learning is to break down the reinforcement learning problem into subtasks through a hierarchy of abstractions. Typically, in the full reinforcement learning problem, the agent is assumed to be in one state of the Markov Decision Process at every time step. The agent then performs one of several possible primitive actions. Based on the agent’s state at time t, and the action it takes from that state, the agent’s state at time t + 1 is determined. For large problems, however, this can lead to too much granularity: when the agent has to decide on each and every primitive action at every granular state, it can often lose sight of the bigger picture. However, if a series of actions can be abstracted out as an abstract action, the agent can just remember the series of actions that was useful in getting it to a temporally distant useful state from the initial state. This is typically referred to as an option or a skill in the reinforcement learning literature. A good analogy is a human planning his movement for a traversal from current location A to a destination B. We identify intermediate destinations Ci to lead us fromA toB when planning fromA, instead of worrying about the exact mechanisms of immediate movement at A which are abstracted over. Options are a convenient way of formalising this abstraction. In keeping with the general philosophy of reinforcement learning, we want to build agents that can automatically discover options with no prior knowledge, purely by exploring the environment. Thus, our approach falls into the broad category of automated discovery of skills. In order to exploit task structure, hierarchical decomposition introduces models defined by stand-alone policies (also known as temporally-extended actions, options, or skills) that can take multiple time steps to execute. Skills can exploit representing structure by representing subroutines that are executed multiple times during execution of a task. Such skills which are learnt in one task can be reused ar X iv :1 60 5. 05 35 9v 1 [ cs .L G ] 1 7 M ay 2 01 6 Submission and Formatting Instructions for ICML 2016 in a different task as long as it requires execution of the same subroutine. Options also make exploration more efficient by providing the decision maker with a high-level behaviour to look ahead to the completion of the corresponding subroutine. Automated discovery of skills or options has been an active area of research and several approaches have been proposed for the same. The current methods could be broadly classified into sample trajectory based and partition based methods. Some of them are: • Identifying bottlenecks in the state space, where the state space is partitioned into sets and the transitions between two sets of states that are rare can be seen as introducing bottleneck sets at the respective points of such rare transitions. Policies to reach such states are cached as options (McGovern & Barto, 2001). • Using the structure present in a factored state representation to identify sequences of actions that cause what are otherwise infrequent changes in the state variables: these sequences are cached away as options (Hengst, 2004). • Obtaining a graphical representation of an agent’s interaction with its environment and using betweenness centrality measures to identify subtasks (Simsek & Barto, 2008). • Using clustering methods (spectral or otherwise) to separate out different strongly connected components of the Markov Decision Process (MDP) and identifying access-states that connect different clusters (Menache et al., 2002). While these methods have had varying amounts of success, they have certain deficiencies. Bottleneck based approaches don’t have a natural way of identifying the part of the state space where options are applicable without external knowledge about the problem domain. Spectral methods need some form of regularization in order to prevent unequal splits that might lead to arbitrary splitting of the state space. We present a framework that detects well-connected or meta stable regions of the state space from a MDP model estimated from trajectories. We use PCCA+, a spectral clustering algorithm from conformal dynamics (Weber et al., 2004) that not only partitions the MDP but also returns the connectivity information between the regions. We then propose a very effective way of composing options using the same framework to take us from one metastable region to another, giving us the policy for free. Once we have these options, we can use standard reinforcement learning algorithms to learn a policy over subtasks to solve the given task. Specifically, we show results using SMDP Qlearning on the 2-room domain. For our attempt at extending it to higher dimensional state space tasks such as Atari 2600 video games, we append the learnt options to the set of primitive actions using Intra-Option Value learning to learn a policy solving the given task. One major advantage of the approach is that we get the policy for the options for free while doing the partitioning by exploiting the membership functions returned by PCCA+. Our approach is able to learn reasonably good skills even with limited sampling which makes it useful in situations where exploration is limited by the environment costs. It also provides a way to refine the abstractions in an online fashion without explicitly reconstructing the entire MDP. More importantly, we extend it to the case where the state space is so large that exact modeling is not possible. In this case, we take inspiration from the recent work on forward prediction to learn the model (Oh et al., 2015) to use Deep Convolutional Neural Networks (CNN) and Long Short Term Memory (LSTM) to learn spatio-temporal representations of the state space. Using the learnt representation, we perform state-aggregation using clustering techniques and estimate our transitional model for the abstract space on these aggregated states. We list the advantages of this approach below: • Skills are acquired online, from sampled trajectories instead of requiring a prior model of the MDP. • Instead of looking for bottleneck states, we look for well connected regions and hence, the discovered options are better aligned to the structure of the state space. • The approach returns connectivity information between the metastable regions which can be used to construct an abstract graph of the state space, combining spatial and temporal information meaningfully. • The clustering algorithm provides a fuzzy membership for every state in belonging to a particular metastable region, which provides a powerful way to compose options naturally. We organize the rest of the paper as follows: We first explain the Option Generation Framework that we propose, by delving on the important aspects of the spectral clustering algorithm PCCA+ and how PCCA+ can be used to generate options. We show results on the 2-room domain for this framework. The next part of the paper focuses on our attempt to extend this framework for more complex tasks such as playing video games like Seaquest on the Atari 2600 domain with options. We explain the motivation and usage of deep networks, the clustering algorithms used for state-aggregation, followed by the model used and Submission and Formatting Instructions for ICML 2016 initial results. We then conclude with a discussion on the challenges in this approach and also comparison with other recent attempts at Hierarchical reinforcement learning for higher dimensional tasks. 2. Option Generation Framework We divide this section into two parts: We first explain the spectral clustering algorithm PCCA+ and motivate its usage for spatial abstraction. The second part discusses the option generation using PCCA+. 2.1. Spatial Abstraction using PCCA+ Given an algebraic representation of the graph representing a MDP we want to find suitable abstractions aligned to the underlying structure. We use a spectral clustering algorithm to do this. Central to the idea of spectral clustering is the graph Laplacian which is obtained from the similarity graph. There are many tight connections between the topological properties of graphs and the graph Laplacian matrices, which spectral clustering methods exploit to partition the data into clusters. However, although the spectra of the Laplacian preserves the structural properties of the graph, clustering data in the eigenspace of the Laplacian does not guarantee this. For example, k-means clustering (Ng et al., 2001) in the eigenspace of the Laplacian will only work if the clusters lie in disjoint convex sets of the underlying eigenspace. Partitioning the data into clusters by projecting onto the largest k-eigenvectors (Meila & Shi, 2001) does not preserve the topological properties of the data in the eigenspace of the Laplacian. For the task of spatial abstraction, the proposed framework requires a clustering approach that exploits the structural properties in the configurational space of objects as well as the spectral subspace, quite unlike earlier methods. Therefore, we take inspiration from the conformal dynamics literature, where (Weber et al., 2004) do a similar analysis to detect conformal states of a dynamical system. They propose a spectral clustering algorithm PCCA+, which is based on the the principles of Perron Cluster Analysis of the transition structure of the system. We extend their analysis to detect spatial abstractions in autonomous controlled dynamical systems. In this approach, the spectra of the Laplacian L (derived from the adjacency matrix S) is constructed and the best transformation of the spectra is found such that the transformed basis aligns itself with the clusters of data points in the eigenspace. A projection method described in (Weber et al., 2004) is used to find the membership of each of the states to a set of special points lying on the transformed basis, which are identified as vertices of a simplex in the R subspace (the Spectral Gap method is used to estimate the number of clusters k). For the first order perturbation, the simplex is just a linear transformation around the origin and to find the simplex vertices, one needs to find the k points which form a convex hull such that the deviation of all the points from this hull is minimized. This is achieved by finding the data point which is farthest located from the origin and iteratively identify data points which are located farthest from the hyperplane fit to the current set of vertices. Figure 1. Simplex First order and Higher order Perturbation Algorithm 1 PCCA+ 1: Construct Laplacian L 2: Compute n (number of vertices) eigenvalues of L in descending order 3: Choose first k eigenvalues for which ek−ek+1 1−ek+1 > tc (Spectral Gap Threshold). 4: Compute the eigenvectors for corresponding eigenvalues (e1, e2, · · · , ek) and stack them as column vectors in eigenvector matrix Y . 5: Let’s denote the rows of Y as Y(1),Y(2), · · · ,Y(N) ∈ R 6: Define π(1) as that index, for which ||Y (π(1))||2 is maximal. Define γ1 = span{Y (π(1))} 7: For i = 2, · · · , k: Define πi as that index, for which the distance to the hyperplane γi−1, i.e., ||Y (πi) − γi−1||2 is maximal. Define γi = span{Y (π1), · · · , Y (πi)}. ||Y (πi) − γi−1||2 = ||Y (πi)− γ i−1((γi−1γ i−1)γi−1)Y (πi) )|| The PCCA+ algorithm returns a membership function, χ, defining the degree of membership of each state s to an abstract state Sj . The connectivity information between two abstract states (Si, Sj) is given by (i, j) entry of χLχ while the diagonal entries provide relative connectivity information within a cluster. The connectivity information is utilized to learn decision policies across abstract states which is described in the next section. There is an intrinsic mechanism to return information about the goodness of clustering of states from the presence of sharp peaks (indicates good clustering) in the eigenvalue distribution. Submission and Formatting Instructions for ICML 2016 2.2. Option generation from PCCA+
Output: Hierarchical Reinforcement Learning using Spatio-Temporal Abstractions and Deep Neural Networks