Title: Creating Multi-Level Skill Hierarchies in Reinforcement Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Reinforcement Learning

Hierarchical Reinforcement Learning

RL

HRL

Skill Discovery

Skill Hierarchies

Graph-Based

Graphs

Graph Clustering

Graph Partitioning
Abstract: What is a useful skill hierarchy for an autonomous agent? We propose an answer based on a graphical representation of how the interaction between an agent and its environment may unfold. Our approach uses modularity maximisation as a central organising principle to expose the structure of the interaction graph at multiple levels of abstraction. The result is a collection of skills that operate at varying time scales, organised into a hierarchy, where skills that operate over longer time scales are composed of skills that operate over shorter time scales. The entire skill hierarchy is generated automatically, with no human input, including the skills themselves (their behaviour, when they can be called, and when they terminate) as well as the dependency structure between them. In a wide range of environments, this approach generates skill hierarchies that are intuitively appealing and that considerably improve the learning performance of the agent.

Creating Multi-Level Skill Hierarchies in Reinforcement Learning

Abstract

What is a useful skill hierarchy for an autonomous agent? We propose an answer based on a graphical representation of how the interaction between an agent and its environment may unfold. Our approach uses modularity maximisation as a central organising principle to expose the structure of the interaction graph at multiple levels of abstraction. The result is a collection of skills that operate at varying time scales, organised into a hierarchy, where skills that operate over longer time scales are composed of skills that operate over shorter time scales. The entire skill hierarchy is generated automatically, with no human intervention, including the skills themselves (their behaviour, when they can be called, and when they terminate) as well as the hierarchical dependency structure between them. In a wide range of environments, this approach generates skill hierarchies that are intuitively appealing and that considerably improve the learning performance of the agent.

1 Introduction

One of the most important open problems in artificial intelligence is how to make it possible for autonomous agents to develop useful action hierarchies on their own, without any input from humans, including system designers and domain specialists. Before addressing this algorithmic question, it is useful to first consider a conceptual one: What constitutes a useful action hierarchy? Here we focus on this conceptual question with the aim of providing a useful foundation for algorithmic development.

Our primary contribution is a characterisation of a useful action hierarchy. In defining this action hierarchy, we use no information other than a graphical representation of how the interaction between the agent and its environment may unfold. When this information is not known a priori, it would be discovered naturally by the agent as it operates in its environment. Beyond this interaction graph, no particular domain knowledge is needed. Hence, our approach is applicable broadly.

Multiple strands of earlier work have used the interaction graph as a basis for defining collections of actions. The main novelty in our approach is our use of modularity maximisation as a central organising principle to expose the structure of the interaction graph at multiple levels of abstraction. The outcome is an action hierarchy that enables the agent to explore its environment efficiently at multiple time scales.

Our approach yields an action hierarchy with four desirable properties. First, it contains actions that operate at a wide range of time scales. This is necessary to solve complex problems, which require agents to be able to act, learn, and plan at varying time scales. Secondly, the actions are naturally organised into a hierarchy, with actions that operate over longer time scales being composed of actions that operate over shorter time scales. This hierarchical structure offers substantial benefits

over unstructured collections of actions. For example, it allows an agent to learn about not only the action it is currently executing but also all lower-level actions that are called in the process. In addition, a hierarchical structure allows actions to be updated in a modular fashion. For instance, any improvements to an action would be immediately reflected in all higher-level actions that call it. Thirdly, the action hierarchy is fully specified. This includes when each action can be selected for execution, how exactly it behaves, and when it terminates. It also includes the number of levels in the hierarchy and the exact dependency structure between the actions. Fourthly, the action hierarchy is generated automatically, with no human intervention.

In a diverse set of environments, the proposed approach translates into action hierarchies that are intuitively appealing. When evaluated within the context of reinforcement learning, they substantially improve learning performance compared to alternative approaches, with the largest performance improvement observed in the largest environment tested.

An important question for future research is how such an action hierarchy may be learned when the agent has no prior knowledge of the dynamics of the environment. We present an initial exploration of how this may be achieved, with positive results.

2 Background

We use the reinforcement learning framework, modelling an agent's interaction with its environment as a finite Markov Decision Process (MDP). An MDP is a six-tuple ( S , A , P , R , D , γ ) , where S is a finite set of states, A is a finite set of actions, P : S × A × S → [0 , 1] is a transition function, R : S ×A×S → R is a reward function, D : S → [0 , 1] is an initial state distribution, and γ ∈ [0 , 1] is a discount factor. Let A ( s ) denote the set of actions available in state s ∈ S . At decision stage t , t ≥ 0 , the agent observes state s t ∈ S and executes action a t ∈ A ( s t ) . Consequently, at decision stage t +1 , the agent receives reward r t +1 ∈ R and observes new state s t +1 ∈ S . The return at decision stage t , denoted by G t , is the discounted sum of future rewards, G t = ∑ ∞ k =0 γ k r t + k +1 . A policy π : S × A → [0 , 1] specifies the probability of selecting action a ∈ A in state s ∈ S . The objective is to learn a policy that maximises the expected return.

The state transition graph of an MDP is a weighted, directed graph whose nodes correspond to the states of the MDP and whose edges correspond to possible transitions between states. Specifically, an edge ( u, v ) exists on the graph if it is possible to transition from state u ∈ S to state v ∈ S by taking some action a ∈ A ( u ) . In this paper, we use uniform edge weights of 1 .

The actions of an MDP take exactly one decision stage to execute. We refer to them as primitive actions . Using primitive actions, it is possible to define abstract actions , also known as skills , whose execution can take a variable number of decision stages. Furthermore, primitive and abstract actions can be combined to form complex action hierarchies. In this work, we represent abstract actions using the options framework [1, 2]. An option o is a three-tuple ( I o , π o , β o ) , where I o ⊂ S is the initiation set, specifying the set of states in which the option can start execution, π o : S × A → [0 , 1] is the option policy, and β o : S → [0 , 1] is the termination condition, specifying the probability of option termination in a given state. An option policy is ultimately defined in terms of primitive actions-because primitive actions are the fundamental units of interaction between the agent and its environment-but this can be done indirectly by allowing options to call other options, making it possible for agents to act, learn, and plan with hierarchies of primitive and abstract actions.

3 Proposed Approach

To define a skill hierarchy, we use modularity maximisation as a central organising principle, applied at multiple time scales. Specifically, we represent the possibilities of interaction between the agent and its environment as a graph and identify partitions of this graph that maximise modularity [3-5].

A partition of a graph is a division of its nodes into mutually exclusive groups, called clusters . The modularity of a partition composed of a set of clusters C = { c 1 , c 2 , . . . , c k } is

where e ii denotes the proportion of total edge weight in the graph that connects two nodes in cluster c i , and a i denotes the proportion of total edge weight in the graph with at least one end connected to a node in cluster c i . A resolution parameter ρ > 0 controls the relative importance of e ii and a i . Intra-cluster edges contribute to both e ii and a i while inter-cluster edges contribute only to a i . A partition that maximises modularity will have relatively dense connections within its clusters and relatively sparse connections between its clusters.

Finding a partition that maximises modularity for a given graph is NP-complete [6]. Therefore, when working with large graphs, approximation algorithms are needed. The most widely used approximation algorithm is the Louvain algorithm [7], which is an agglomerative hierarchical graph clustering approach. While no formal analysis exists, the runtime of the Louvain algorithm has been observed empirically to be linear in the number of graph edges [8]. It has been successfully applied to graphs with millions of nodes and billions of edges [7].

An important feature of the Louvain algorithm is that, as a hierarchical graph clustering method, it exposes the structure of a graph at multiple levels of granularity. Specifically, the output of the Louvain algorithm is a sequence of partitions of the input graph. This sequence has a useful structure: multiple clusters found in one partition in the sequence are merged into a single cluster in the next partition in the sequence. In other words, the output is a hierarchy of clusters, with earlier partitions containing many smaller clusters that are merged into fewer larger clusters in later partitions. This hierarchical structure forms the basis of our characterisation of a useful multi-level skill hierarchy.

The Louvain algorithm starts by placing each node of the graph in its own cluster. Nodes are then iteratively moved locally, from their current cluster to a neighbouring cluster, until no gain in modularity is possible. This results in a revised partition corresponding to a local maximum of modularity with respect to local node movement. The revised partition is used to define an aggregate graph as follows: each cluster in the partition is represented as a single node in the aggregate graph, and a directed edge is added to the aggregate graph if there is at least one edge that connects neighbouring clusters in the corresponding direction. This process is then repeated on the aggregate graph, and then on the next aggregate graph, and so on, until an iteration is reached with no modularity gain. Pseudocode for the Louvain algorithm is presented in Section H of the supplementary material.

Let h denote the number of partitions returned by the Louvain algorithm when applied to the state transition graph. We use each of the h partitions to define a single layer of skills, resulting in an action hierarchy with h levels of abstract actions (skills) above primitive actions. Each level of the hierarchy contains one or more skills for efficiently navigating between neighbouring clusters of the state transition graph. Specifically, we define an option for navigating from a cluster c i to a neighbouring cluster c j as follows: the initiation set consists of all states in c i ; the option policy efficiently takes the agent from a given state in c i to a state in c j ; the option terminates with probability 1 in states in c j , with probability 0 otherwise.

Taking advantage of the natural hierarchical structure of the partitions produced by the Louvain algorithm, we compose the skills at one level of the hierarchy to define the skills at the next level up. That is, at each level of the hierarchy, option policies call actions (options or primitive actions) from the level below, with primitive actions being called directly by option policies from only the first level of the hierarchy. We call the resulting set of skills the Louvain skill hierarchy .

4 Related Work

There have been earlier approaches to skill discovery using the state transition graph. The approach we propose here differs from them in two fundamental ways. First, it is novel in its use of modularity maximisation as a central organising principle for skill discovery. Secondly, it produces a multi-level hierarchy, whereas existing graph-based approaches produce hierarchies with only a single level of skills above primitive actions.

Many existing approaches to skill discovery use the state transition graph to identify useful subgoal states and define skills that efficiently take the agent to these subgoals. Suggestions for useful subgoals have often been inspired by the concept of a 'bottleneck'. They include states that are on the border of strongly-connected regions of the state space [9], states that allow transitions between different regions of the state space [10], and states that lie on the shortest path between many pairs of states [11]. To identify such states, several approaches use graph centrality measures [11-15]. Others use graph

partitioning algorithms [9, 16-22]. Alternatively, it has been proposed that 'landmark' states found at the centre of strongly-connected regions of the state-space can be used as subgoals [23].

The proposed approach is most directly related to skill discovery methods that make use of graph partitioning to identify meaningful regions of the state transition graph and define skills for navigating between them [16, 24-28]. Three of these methods use the concept of modularity. One such approach is to generate a series of possible partitions by successively removing the edge with the highest edge betweenness from a graph, then selecting the partition with the highest modularity [25]. A second approach is to generate a partition using the label propagation algorithm and then to merge neighbouring clusters until no gain in modularity is possible [28]. In these two approaches, modularity maximisation is applied after first producing candidate partitions using different criteria. Consequently, the final partition does not maximise modularity overall. The time complexity of the label propagation method is near-linear in the number of graph edges, whereas the edge betweenness method has a time complexity of O ( m 2 n ) on a graph with m edges and n nodes. A third approach is by Xu et al. [27], who use the Louvain algorithm to find a partition that maximises modularity but, unlike our approach, define skills only for moving between clusters in the highest-level partition, discarding all lower-level partitions. All three methods produce a single level of skills above primitive actions.

Several approaches have used the graph Laplacian [29, 30] to identify skills that are specifically useful for efficiently exploring the state space. It is unclear how to arrange such skills to form multi-level skill hierarchies. In contrast, the proposed approach produces a set of skills that are naturally arranged into a multi-level hierarchy.

While existing graph-based methods do not learn multi-level hierarchies, policy-gradient methods have made some progress towards this goal. Bacon et al. [31] extended policy-gradient theorems [32] to allow the learning of option policies and termination conditions in a hierarchy with a single level of skills above primitive actions. Riemer et al. [33] further generalised these theorems to support multi-level hierarchies. Fox et al. [34] propose an imitation learning method that finds the multi-level skill hierarchy most likely to generate a given set of example trajectories. Levy et al. [35] propose a method for learning multi-level hierarchies of goal-directed policies, with each level of the hierarchy producing a subgoal for the lower levels to navigate towards. However, these methods are not without their limitations. Unlike the approach proposed here, they all require the number of hierarchy levels to be pre-defined instead of finding a suitable number automatically. They do not judiciously define initiation sets, instead making all skills available in all states. They also target different types of problems than we do, such as imitation-learning or goal-directed problems.

5 Empirical Analysis

We analyse the Louvain skill hierarchy in the six environments depicted in Figure 1: Rooms, Grid, Maze [23], Office, Taxi [36], and Towers of Hanoi. In all environments, the reward is -0 . 001 for each action and an additional +1 . 0 for reaching a goal state. In Rooms, Grid, Maze, and Office, there are four primitive actions: north, south, east, and west. In Taxi, there are two additional primitive actions: pick-up-passenger and put-down-passenger. Some decisions in Taxi are irreversible. For instance, after picking up the passenger, the agent cannot return to a state where the passenger is still waiting to be picked up. We describe each of these environments fully in Section A of the supplementary material.

When generating partitions of the state transition graph using the Louvain algorithm, we used a resolution parameter of ρ = 0 . 05 , unless stated otherwise. When converting the output of the Louvain

Rooms

Grid

Maze

Office

R

Y

G

B

Taxi

Towers of Hanoi

algorithm into a concrete skill hierarchy, we discarded all levels of the cluster hierarchy where the mean number of nodes per cluster was less than 4 . Our reasoning is that skills that navigate between such small clusters execute for only a very small number of decision stages (often only one or two) and are not meaningfully more abstract than primitive actions. For all methods used in our comparisons, we generated options using the complete state transition graph and learned their policies offline using macro-Q learning [37]. We trained all hierarchical agents using macro-Q learning and intra-option learning [38]. Although these algorithms have not previously been applied to multi-level hierarchies, they both extend naturally to this case. The primitive agent was trained using Q-Learning [39]. The shaded regions on the learning curves represent the standard error in 40 independent runs. All experiments are fully described in Section B of the supplementary material.

Our analysis is directed by the following questions: What is the Louvain skill hierarchy generated in each environment? How does this skill hierarchy impact the learning performance of the agent? How do the results change as the number of states increases? Does arranging skills into a multi-level hierarchy provide benefits over a flat arrangement of the same skills? What is the impact of varying the value of the resolution parameter ρ ?

Louvain Skill Hierarchy. We first examine the cluster hierarchies generated by applying the Louvain algorithm to the state transition graphs of various environments. Figure 2 shows the results in Rooms, Office, Taxi, and Towers of Hanoi. Section D of the supplementary material shows additional results in Grid and Maze.

In Rooms, the hierarchy has four levels. At level three, we see that each room has been placed in its own cluster. Moving up the hierarchy, at level four, two of these rooms have been joined together into a single cluster. Moving down the hierarchy, each room has been divided into smaller clusters at level two, and then into even smaller clusters at level one. The figure illustrates how the corresponding skill hierarchy would enable efficient navigation between and within rooms.

In Office, at the top level, we see six large clusters connected by corridors. As we move lower down the hierarchy, we see that these large clusters have been divided into increasingly smaller regions. At level three, there are many rooms that form their own cluster. At level two, most rooms have been divided into multiple clusters. The figure reveals how the corresponding skill hierarchy would enable efficient navigation of the environment at multiple time scales.

In Taxi, the state transition graph has four disconnected components, each corresponding to one particular passenger destination: R, G, B, or Y. In Figure 2, we show only one of these components, the one where the passenger destination is B. The results for the other components are similar. The Louvain cluster hierarchy has four levels. At the top level, we see four clusters. In three of these clusters, the passenger is waiting at their starting location (R, G, or Y). In the fourth cluster, the passenger is either in-taxi or has been delivered to their destination (B). Navigation between these clusters is unidirectional, with only three possibilities. The three corresponding skills navigate the taxi to the passenger location and pick up the passenger. Moving one level down the hierarchy, the clusters produce skills that move the taxi between the left and the right-hand side of the grid, which are connected by the bottleneck state at the centre of the taxi navigation grid.

In Towers of Hanoi, the hierarchy has three levels. At level three, moving between the three clusters corresponds to moving the largest disk to a different pole. Each of these clusters has been divided into three smaller clusters at level two and then into three even smaller clusters at level one. A similar structure exists at levels two and one. The smaller clusters at level two correspond to moving the second-largest disc between different poles; the smallest clusters at level one correspond to moving the third-largest disc between different poles.

In all of these domains, the Louvain skill hierarchy closely matches human intuition. In addition, it is clear how skills at one level of the hierarchy can be composed to produce the skills at the next level.

Learning Performance. We compare learning performance with the Louvain skill hierarchy to approaches based on Edge Betweenness [25] and Label Propagation [28], as well as to the method proposed by Xu et al. [27]. These methods are the most directly related to the proposed approach, as discussed in Section 4. In addition, we compare to options that navigate to local maxima of Node Betweenness [11], a subgoal-based approach that captures the many different notions of the bottleneck concept in the literature. Similarly to the proposed approach, it is also one of the few approaches in the literature to explicitly characterise a target skill hierarchy for an agent to learn. We also compare to Eigenoptions [29], which are derived from the Laplacian of the state transition

Rooms

Level 4

Level 3

Level 2

Level 1

Office

Level 5

Level 4

Level 3

Level 2

Taxi

Level 4

Level 3

Level 2

Level 1

Towers of Hanoi

Level 3

Level 2

Level 1

CCI1

yi

1TTE

graph and encourage efficient exploration. Finally, we include a Primitive agent that uses only primitive actions. Primitive actions are available to all agents.

We present learning curves in Figure 3. The Louvain agent has a clear and substantial advantage over other approaches in all domains except for Towers of Hanoi, where its performance is much closer to that of the other hierarchical agents, with none of them performing much better than the primitive agent. This is consistent with existing results reported in this domain (e.g., by Jinnai et al. [30]).

Section C of the supplementary material contains further analysis comparing Louvain skills and skills based on node betweenness.

Scaling to larger domains. We experimented with a multi-floor version of the Office environment, with floors connected by a central elevator, where two primitive actions move the agent up and down between two adjacent floors. The number of states in this domain can be varied by adjusting parameters such as the number of office floors. We use this environment to explore how the performance of the agent and the Louvain skill hierarchy change as the number of states in the environment increases.

10

20

30

40

50

60

0

1

2

3

4

5

6

Rooms

Epoch

Reward

0

20

40

60

80

100

0

4

8

12

Maze

Epoch

Reward

0

10

20

30

40

50

0

1

2

3

4

5

6

Towers of Hanoi

Epoch

Reward

0

10

20

30

40

50

0

5

10

15

20

Taxi

Epoch

Reward

0

50

100

150

200

0

5

10

15

Office

Epoch

Reward

20

40

60

80

100

0

5

10

15

Grid

Epoch

Reward

Louvain

Eigenoptions

Edge Bet.

Primitive

Figure 4a presents results in an Office environment with 2537 states. It shows that the Louvain agent learns much more quickly than all other agents. Some alternatives, including the Eigenoptions agent, fail to achieve any learning.

Figure 4b shows how the Louvain skill hierarchy changes with the size of the state space in the Office environment. The figure shows the depth of the skill hierarchy in a series of fifteen Office environments of increasing size, ranging from a single floor ( ∼ 10 3 states) to one thousand floors ( ∼ 10 6 states). The depth of the hierarchy increased very gradually with the number of states in the environment. The maximum hierarchy depth reached was eight in the largest environment tested.

Hierarchical versus flat arrangement of skills. An alternative to the Louvain skill hierarchy is a flat arrangement of the skills, where each skill calls primitive actions directly rather than indirectly through other (lower-level) skills. We expected the hierarchical arrangement to lead to faster learning than the flat arrangement due to the additional learning updates enabled by the hierarchical relationship between the skills. Figure 5 shows that this is indeed the case. In the figure, Louvain shows an agent that uses the Louvain skill hierarchy while Louvain flat shows an agent that uses the Louvain skills but where the skill policies call primitive actions directly rather than through other skills. In addition, the figure shows a number of agents that use only a single level of the Louvain hierarchy

100

200

300

0

5

10

15

Epoch

Reward

Louvain

Edge Bet.

Eigenoptions

Primitive

(a)

10 3

10 4

10 5

10 6

5

6

7

8

Number of States

Hierarchy Depth

(b)

15

30

45

60

0

1

2

3

4

5

6

Rooms

Epoch

Reward

Louvain

Louvain flat

Level 1

Level 2

Level 3

Level 4

Primitive

0

10

20

30

40

50

0

5

10

15

20

Taxi

Epoch

Reward

0

20

40

60

80

100

0

4

8

12

Maze

Epoch

Reward

( Level 1 , Level 2 , Level 3 , or Level 4 ), with option policies that call primitive actions directly. Primitive actions were available to all agents. The figure shows that the hierarchical agent learns more quickly than the flat agent. Furthermore, the agents using individual levels of the Louvain hierarchy learn more quickly than the primitive agent but not as quickly as the agent using the full Louvain hierarchy.

Impact of the resolution parameter. When using very high values of the resolution parameter ρ , the Louvain algorithm terminates without producing any partitions. On the other hand, when using very low values of ρ , it runs until a partition is produced where all nodes are merged into a single cluster.

Figure 6 shows how changing the resolution parameter ρ impacts the Louvain cluster hierarchy in Towers of Hanoi and Rooms. In Towers of Hanoi, at ρ = 10 , the output is a single level containing many small clusters comprised of three nodes. As ρ gets smaller, the cluster hierarchy remains the same until ρ = 3 . 3 , at which point a second level is added. The first level remains identical to the single level produced at ρ = 10 . The second level contains larger clusters, each formed by merging three of the clusters from the first level. As ρ is reduced further, additional levels are added to the hierarchy. When a new level is added to the hierarchy, the existing levels generally remain the same.

A sensitivity analysis on the value of ρ showed that a wide range of ρ values led to useful skills, and that performance gradually decreased to no worse than that of a primitive agent at higher values of ρ . Section E of the supplementary material contains full details of this sensitivity analysis and a more detailed discussion on the choice of ρ .

6 Discussion and Future Work

An important research direction for future work is incremental learning of Louvain skill hierachies as the agent interacts with its environment-because the state transition graph will not always be available in advance. We explored the feasibility of incremental learning in the Rooms environment.

10 -5

10 -4

10 -3

10 -2

10 -1

10 0

10 1

3

9

27

81

Towers of Hanoi

Resolution Parameter

Nodes Per Cluster

Level 5

Level 4

Level 3

Level 2

Level 1

10 -5

10 -4

10 -3

10 -2

10 -1

10 0

10 1

2

8

22

60

Rooms

Resolution Parameter

Nodes Per Cluster

10

20

30

40

50

60

0

1

2

3

4

5

6

Epoch

Reward

Ceiling

Replace

Update

Primitive

(a) Incremental Performance

(b) 30 States

(c) 60 States

(d) All States

The results are shown in Figure 7. The agent started with an empty state transition graph and no skills. Every m decision stages, it updated its state transition graph with new nodes and edges, and it revised its skill hierarchy using one of two approaches: Replace or Update .

In the first approach ( Replace ), the agent applied the Louvain algorithm to create a new skill hierarchy from scratch. In the second approach ( Update ), the agent incorporated the new information into its existing skill hierarchy, using an approach similar to the Louvain algorithm. This second approach starts by assigning each new node to its own cluster; the new nodes are then iteratively moved locally, between neighbouring clusters (both new and existing) until no modularity gain is possible. This revised partition is used to define an aggregate graph and the entire process is repeated on the aggregate graph, and the next aggregate graph, and so on, until an iteration is reached with no modularity gain. The cluster membership of existing nodes stays fixed; only new nodes have their cluster membership updated. The result of both approaches is a revised set of partitions, from which a revised hierarchy of Louvain skills is derived. Section I of the supplementary material presents pseudocode for the two incremental approaches.

Figure 7a shows the performance of the incremental agents. The agents started with only primitive actions; after decision stages 100 , 500 , 1000 , 3000 , and 5000 , the state transition graph was updated and the skill hierarchy was revised. The figure compares performance to a Primitive agent and an agent using the full Louvain skill hierarchy, whose performance acts as a Ceiling for the incremental agents. Both incremental agents learned much faster than the primitive agent and only marginally slower than the fully-informed Louvain agent. The two incremental agents had similar performance throughout training but Replace reached a higher level of asymptotic performance than Update , as expected. The reason is that partitions produced early in training are based on incomplete information; Replace discards these early imperfections while Update carries them forward. But there is a tradeoff: building a new skill hierarchy from scratch has a higher computational cost than updating an existing one.

Figures 7b-7d show the evolution of the partitions as the Update agent performed a random walk in Rooms. The partitions were updated after observing 30 states, 60 states, and all possible transitions. The figure shows that, as more nodes were added, increasingly higher-level skills were produced.

These results demonstrate the feasibility of learning Louvain skills incrementally. A full incremental method for learning Louvain skills may take many forms, and different approaches may be useful under different circumstances, with each having its own strengths and weaknesses. We leave the full development of such algorithms to future work.

Another direction for future work is extending Louvain skills to environments with continuous state spaces such as robotic control tasks. Such domains present a difficulty to all skill discovery methods that use the state transition graph due to the inherently discrete nature of the graph. If the critical step of constructing an appropriate graphical representation of a continuous state space can be achieved, all graph-based methods would benefit. Some approaches have already been proposed in the literature. We use one such approach [40, 20, 26] to examine the Louvain hierarchy in a variant of the Pinball domain [41], which involves manoeuvring a ball around a series of obstacles to reach a goal, as

Pinball Layout

Level 3

Level 2

Level 1

shown in the leftmost panel of Figure 8. The state is represented by two continuous values: the ball's position in the horizontal and vertical directions. At each decision stage, the agent can choose to apply a small force to the ball in each direction. The amount of force applied is stochastic and causes the ball to roll until friction causes it to come to a rest. Collisions between the ball and the obstacles are elastic.

We sampled 4000 states, added them to the state transition graph, then added an edge between each node and its k -nearest neighbours according to Euclidean distance, assigning an edge weight of e -4 d 2 to an edge that connects two locations u and v with Euclidean distance d between them. We then applied the Louvain algorithm to the resulting graph. The result was the cluster hierarchy shown in Figure 8. It had three levels. Moving between clusters in the top level corresponds to skills that enable high-level navigation of the state space, and take into account features such as the natural bottlenecks caused by the obstacles, allowing the agent to efficiently change its position. Moving between clusters in the lower-level partitions enable more local navigation.

Currently, Louvain skills at one level of the hierarchy are composed of skills from only the previous level. Such skills may not provide the most efficient navigation between clusters. Future work could consider composing skills from all lower levels, including primitive actions.

Because Louvain skills are derived from the connectivity of the state transition graph, we expect them to be suitable for solving a range of problems in a given environment. Examining their use for transfer learning is a useful direction for future work.

A possible difficulty with building multi-level skill hierarchies is that having a large number of skills can end up hurting performance by increasing the branching factor. One solution that has been explored in the context of two-level hierarchies is pruning less useful skills from the hierarchy [21, 42]. Further research is needed to address this potential difficulty.

A key open problem for graph-based skill discovery methods generally is how to best make use of state abstraction. We suggest that the most natural place to introduce state abstraction is when constructing the state transition graph itself. Instead of a concrete state, each node could represent an abstract state based on some learned representation of the environment. The proposed method-and any other existing graph-based method-could then directly use this abstract state transition graph to define a skill hierarchy.

Lastly, we note that the various characterisations of useful skills proposed in the literature, including the one proposed here, are not necessarily competitors. To solve complex tasks, it is likely that an agent will need to use many different types of skills. An important avenue for future work is exploring how different ideas on skills discovery can be used together to enable agents that can autonomously develop complex and varied skill hierarchies.

Acknowledgments and Disclosure of Funding

We would like to thank the members of the Bath Reinforcement Learning Laboratory for their constructive feedback. This research was supported by the Engineering and Physical Sciences Research Council [EP/R513155/1] and the University of Bath. This research made use of Hex, the GPU Cloud in the Department of Computer Science at the University of Bath.

References

R. S. Sutton, D. Precup, and S. Singh. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence , 112(1-2):181-211, 1999.

D. Precup. Temporal abstraction in reinforcement learning . PhD thesis, University of Massachusetts Amherst, 2000.

M. E. Newman and M. Girvan. Finding and evaluating community structure in networks. Physical Review E , 69(2):026113, 2004.

E. A. Leicht and M. E. Newman. Community structure in directed networks. Physical Review Letters , 100(11):118703, 2008.

A. Arenas, J. Duch, A. Fernández, and S. Gómez. Size reduction of complex networks preserving modularity. New Journal of Physics , 9(6):176, 2007.

U. Brandes, D. Delling, M. Gaertler, R. Görke, M. Hoefer, Z. Nikoloski, and D. Wagner. Maximizing modularity is hard. arXiv preprint:physics/0608255 , 2006.

V. D. Blondel, J.-L. Guillaume, R. Lambiotte, and E. Lefebvre. Fast unfolding of communities in large networks. Journal of Statistical Mechanics: Theory and Experiment , 2008(10):10008, 2008.

A. Lancichinetti and S. Fortunato. Community detection algorithms: a comparative analysis. Physical Review E , 80(5):056117, 2009.

I. Menache, S. Mannor, and N. Shimkin. Q-Cut - Dynamic discovery of sub-goals in reinforcement learning. In Proceedings of the 13th European Conference on Machine Learning , ECML '02, pages 295-306. Springer-Verlag, 2002.

Ö. ¸ Sim¸ sek and A. G. Barto. Using relative novelty to identify useful temporal abstractions in reinforcement learning. In Proceedings of the 21st International Conference on Machine Learning , ICML '04, pages 95-102. ACM, 2004.

Ö. ¸ Sim¸ sek and A. G. Barto. Skill characterization based on betweenness. In Advances in Neural Information Processing Systems 21 , NeurIPS '09, pages 1497-1504. Curran Associates, Inc., 2009.

P. Moradi, M. E. Shiri, and N. Entezari. Automatic skill acquisition in reinforcement learning agents using connection bridge centrality. In International Conference on Future Generation Communication and Networking , pages 51-62. Springer, 2010.

A. A. Rad, M. Hasler, and P. Moradi. Automatic skill acquisition in reinforcement learning using connection graph stability centrality. In Proceedings of 2010 IEEE International Symposium on Circuits and Systems , pages 697-700. IEEE, 2010.

M. A. Imanian and P. Moradi. Autonomous subgoal discovery in reinforcement learning agents using bridgeness centrality measure. International Journal of Electrical and Computer Sciences , 11(5):54-62, 2011.

P. Moradi, M. E. Shiri, A. A. Rad, A. Khadivi, and M. Hasler. Automatic skill acquisition in reinforcement learning using graph centrality measures. Intelligent Data Analysis , 16(1): 113-135, 2012.

J. H. Metzen. Online skill discovery using graph-based clustering. In Proceedings of the Tenth European Workshop on Reinforcement Learning , EWRL '13, pages 77-88. PMLR, 2013.

Ö. ¸ Sim¸ sek, A. P. Wolfe, and A. G. Barto. Identifying useful subgoals in reinforcement learning by local graph partitioning. In Proceedings of the 22nd International Conference on Machine learning , ICML '05, pages 816-823. ACM, 2005.

S. J. Kazemitabar and H. Beigy. Using strongly connected components as a basis for autonomous skill acquisition in reinforcement learning. In 6th International Symposium on Neural Networks , ISSN '09, pages 794-803. Springer, 2009.

N. Entezari, M. E. Shiri, and P. Moradi. A local graph clustering algorithm for discovering subgoals in reinforcement learning. In International Conference on Future Generation Communication and Networking , pages 41-50. Springer, 2010.

P.-L. Bacon and D. Precup. Using label propagation for learning temporally abstract actions in reinforcement learning. In Proceedings of the Workshop on Multiagent Interaction Networks , MAIN '13, 2013.

N. Taghizadeh and H. Beigy. A novel graphical approach to automatic abstraction in reinforcement learning. Robotics and Autonomous Systems , 61(8):821-835, 2013.

S. J. Kazemitabar, N. Taghizadeh, and H. Beigy. A graph-theoretic approach toward autonomous skill acquisition in reinforcement learning. Evolving Systems , 9(3):227-244, 2018.

R. Ramesh, M. Tomar, and B. Ravindran. Successor options: an option discovery framework for reinforcement learning. In Proceedings of the 28th International Joint Conference on Artificial Intelligence , IJCAI '19, pages 3304-3310. AAAI Press, 2019.

S. Mannor, I. Menache, A. Hoze, and U. Klein. Dynamic abstraction in reinforcement learning via clustering. In Proceedings of the 21st International Conference on Machine Learning , ICML '04, pages 71-78. ACM, 2004.

M. Davoodabadi and H. Beigy. A new method for discovering subgoals and constructing options in reinforcement learning. In Proceedings of the 5th Indian International Conference on Artificial Intelligence , pages 441-450, 2011.

F. Shoeleh and M. Asadpour. Graph based skill acquisition and transfer learning for continuous reinforcement learning domains. Pattern Recognition Letters , 87:104-116, 2017.

X. Xu, M. Yang, and G. Li. Constructing temporally extended actions through incremental community detection. Computational Intelligence and Neuroscience , 2018.

M. D. Farahani and N. Mozayani. Automatic construction and evaluation of macro-actions in reinforcement learning. Applied Soft Computing , 82:105574, 2019.

M. C. Machado, M. G. Bellemare, and M. Bowling. A Laplacian framework for option discovery in reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning , ICML '17, pages 2295-2304. PMLR, 2017.

Y. Jinnai, J. W. Park, D. Abel, and G. Konidaris. Discovering options for exploration by minimizing cover time. In Proceedings of the 36th International Conference on Machine Learning , ICML '19, pages 3130-3139. PMLR, 2019.

P.-L. Bacon, J. Harb, and D. Precup. The option-critic architecture. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 31, 2017.

R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems 12 , NeurIPS '99. MIT Press, 1999.

M. Riemer, M. Liu, and G. Tesauro. Learning abstract options. In Advances in Neural Information Processing Systems 31 , NeurIPS '18. Curran Associates, Inc., 2018.

R. Fox, S. Krishnan, I. Stoica, and K. Goldberg. Multi-level discovery of deep options. arXiv preprint:1703.08294 , 2017.

A. Levy, G. Konidaris, R. Platt, and K. Saenko. Learning multi-level hierarchies with hindsight. In Proceedings of the 7th International Conference on Learning Representations , ICLR '19, 2019.

T. G. Dietterich. Hierarchical reinforcement learning with the MAXQ value function decomposition. Journal of Artificial Intelligence Resesearch , 13(1):227-303, 2000.

A. McGovern, R. S. Sutton, and A. H. Fagg. Roles of macro-actions in accelerating reinforcement learning. In Grace Hopper Celebration of Women in Computing , volume 1, pages 13-18, 1997.

R. S. Sutton and D. Precup. Intra-option learning about temporally abstract actions. In Proceedings of the 15th International Conference on Machine Learning , pages 556-564. Morgan Kaufman, 1998.

C. J. C. H. Watkins. Learning from delayed rewards . PhD thesis, King's College, Cambridge United Kingdom, 1989.

S. Mahadevan and M. Maggioni. Proto-value functions: A Laplacian framework for learning representation and control in Markov decision processes. Journal of Machine Learning Research , 8(10), 2007.

G. Konidaris and A. Barto. Skill discovery in continuous reinforcement learning domains using skill chaining. In Advances in Neural Information Processing Systems 22 , NeurIPS '09, pages 1015-1023. Curran Associates, Inc., 2009.

D. M. Farahani and N. Mozayani. Evaluating skills in hierarchical reinforcement learning. International Journal of Machine Learning and Cybernetics , 11(10):2407-2420, 2020.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method to discover skill hierarchy by applying hierarchical graph clustering methods to expose the structure. The proposed method introduces Louvain method for skill hierarchy revealing and produces skill hierarchy with multiple-grained of action abstractions. The proposed method is tested on six different environments and compared with other methods.\n\nSTRENGTHS:\nThis paper introduces the concept of graph clustering to discover action hierarchies, which is an interesting idea. The references are described in details, which makes the intuition and technique of the proposed method more reasonable. It is appreciated also the experiments in six different problems and the evaluation with other methods, as well as the ablation study.\n\nWEAKNESSES:\nThere are several weaknesses. First, this paper is poorly written. There are a lot of grammatical errors and unprofessional writing styles, which makes it difficult to understand. Technical writing in Proposed Approach part in this paper is also limited, causing a great deal of confusion about its proposed method.\n\nThis paper attempts to solve the problem of action hierarchy discovery of agents. This problem is extremely difficult and could not be well solved by introducing Louvain-based graph clustering methods as described in the paper. More details will be described in the limitations. Moreover, the proposed approach section in the paper spends plenty of words to describe the process of the Louvain algorithm. However, the Louvain algorithm has been proposed in 2008 and there have been a lot of improved works during 2010-2015, which should not be the focus of the paper.\n\nIn addition, although there are many tasks in the experiment section, these problems are all simple and some of them can be solved well even using simple planning methods. It is unnecessary to discover action hierarchies in these scenarios. Agents’ skills should be learned in more complex scenarios with more constraints, such as StarCraft II and the Minecraft (the recently popular environment), but these complex scenarios are not considered in this paper.\n\nQUESTIONS:\nHow about the learning efficiency of the proposed algorithm? Does the size of the graph limit the learning efficiency of the proposed method compared to the original reinforcement learning algorithm?\n\nI am confused that there is no diversity in the skills formed by combining only 4 actions with same type in these simple scenarios including Rooms. Poor diversity limited the performance of policies learned from the reinforcement learning framework.\n\nThere are too many confusing descriptions, including:\n\n•\t\"an autonomous agent\" in Line1\n\n•\t\"graphical structure\" in Line2\n\n•\t\"its environment\" in Line2\n\n•\t\"multiple levels of abstraction\" in Line4\n\n•\t\"regions of the state\" in Line5\n\n•\t\"connected within themselves\" in Line6\n\n•\t\"various levels of granularity\" in Line16\n\n•\t\"outcome\" in Line17\n\n•\t\"characterisation\" in Line19\n\n•\t\"existing approaches to skill discovery\" in Line55\n\n•\t\"are not without their limitations\" in Line72\n\nLIMITATIONS:\nExisting reinforcement learning algorithms based on the state-action space have achieved great results, especially in simple scenarios, which can achieve efficient exploration in the policy space. The reason for introducing skills to split the primitive action space into multiple parts for separate exploration is that the state action search space is so large in facing with more complex scenarios that the primitive reinforcement learning algorithm is inefficient for exploration. However, it is difficult for the skill-based approach to converge to the global optimal solution. The possibility of convergence to a local optimal solution in each subspace is high, which is more likely to occur in simple scenarios. This paper does not propose a theoretical guarantee on the learning efficiency of the algorithm and lacks the optimization analysis of the algorithm. In addition, it does not conduct experimental evaluation on complex scenarios to answer aforementioned questions.\n\nLouvain is a commonly used in community discovery to mine graph clusters. However, the learning process of Louvain algorithm and the learning process of reinforcement learning framework are independent. How can the clustering information of state action space mined by Louvain algorithm actively guide the reinforcement learning? How can the policy search process in the state-action space be fed back to the mining process of Louvain? I think it is difficult. In addition, the modularity seems to be independent of the reward in reinforcement learning, which may lead to the inability of combining the Louvain algorithm with the reinforcement learning algorithm, no matter what approach is taken.\n\nI wonder if graph partitioning can really guide skill learning. First concerning about the model-based reinforcement learning, which generates a complete state action transition graph. However, once the transition graph is completely knowable, traditional reinforcement learning methods can achieve great performance. There is no need to develop reinforcement learning method based on graph partitioning. In case of model-free reinforcement learning, where the policy learning of traditional reinforcement learning methods is difficult, graph partitioning will play a positive role. However, if the transition graph is unknown, it is impossible to reveal expective clusters of the transition graph. Therefore, the proposed method in this paper would be ineffective. But I think that combining graph partitioning method with model-free reinforcement learning with incomplete transition graph is a promising direction.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a graph-partitioning-based method to automatically learn multi-level hierarchies of skills at varying timescales in reinforcement learning. The method assumes existence of a complete state transition graph and applies Louvian algorithm to create a hierarchy of state clusters. By merging states, the algorithm creates state partitions that maximize modularity, which measures the quality of partitions based on strong connections within clusters and weak connections between them.\n\nThe work considers an important unsolved problem of automatically creating multi-level skill hierarchies. The paper is written well, however, it needs more clarity. Some details regarding the assumptions made and the scope of problems need to be clarified right from the beginning to improve quality. The empirical evaluation could be improved by diversifying the choice of the domains because four out of six domains are extremely similar navigation domains, and clearly justifying the choice of baselines and their input requirements.\n\nSTRENGTHS:\nThe number of levels in a hierarchy do not need to be predefined, unlike existing methods. The proposed method automatically finds the levels in the hierarchy using no gain in modularity as a stopping condition.\n\nWEAKNESSES:\n(I) The proposed method relies on concrete state transition graph and then uses hierarchal clustering based algorithm to iteratively merge states and produce a hierarchy of clusters. Hence, the method might not scale well to problems with large state space as the memory requirement would increase exponentially and learning a complete state transition graph would require exploration of the complete state space/high number of samples. \n\n(II) The assumption regarding the availability of a complete state space graph and the scope of the problems (discrete state space, low-dimensional, etc.) the method can handle need to be stated clearly.  \n\n(III) The empirical evaluation could be improved by diversifying the selection of domains. It is unclear how the approach would perform in more complex decision-making tasks. \n\n(IV) An analysis of comparison of the number of samples required by each baseline to learn a hierarchy and option policies is needed.\n\n\nMinor:\n\n(I) line 20: has -> have\n\n(II) line 112: the partition -> the partition c_i is\n\nQUESTIONS:\n(I) Can you explain \"we use each of the h partitions to define a single layer of skills, resulting in a hierarchy of h levels..\"? It is not clear how total number of partitions is directly affecting the number of the layers.\n\n(II) How can state abstraction be employed using the proposed method?\n\n(III) Can the current method handle continuous state space problems?\n\n(IV) How does the proposed work relate to [1]?\n\n(V) In empirical evaluation, how is the office world (without any objects) different from the maze world domain? \n\n(VI) Can you comment about scalability of the approach to handle domains with large state spaces?\n\n(VII) Do the baselines assume a state transition graph or start learning from scratch? Does the learning performance in Fig 3 include the samples required to learn the hierarchies? If not, how many samples were required to learn the hierarchies by different methods along with the samples required to learn the option policies?\n\n\nReferences:\n\n[1] Fox, R., Krishnan, S., Stoica, I. and Goldberg, K., 2017. Multi-level discovery of deep options. arXiv preprint arXiv:1703.08294.\n\nLIMITATIONS:\n(Included in summary, weaknesses)", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
Given a graph of connectivity of the state space of an environment, this paper proposes to use a hierarchical clustering technique, the Louvain algorithm, to provide reinforcement learning with a multi-level skill representation.  This allows policies to take actions at variable time scales, and experimental results show that the identified clusters allow faster learning in the domains studied.

STRENGTHS:
- Clarity of presentation.  The graph clustering algorithm is motivated and described very clearly.  The experiments are also well described.  The visualizations of the hierarchical results are informative and easy to interpret.

- Comparison with baselines.  The paper describes five alternative graph-based approaches, and shows superior performance of the Louvain-based method.  

- Initial explorations into some limitations.  The paper points out that the method requires full knowledge of the state space and it's graph representation.  Initial results explore what happens when the clustering is learned online with interaction in the environment.  The paper also explores extending the method to continuous state spaces.  

- Thorough discussion section.

WEAKNESSES:
- Although the description of the Louvain algorithm and most of the rest of the paper is quite thorough and well presented, the method for training a policy using the hierarchical clustering output is left comparatively sparse.  Especially since the cited method may not be readily familiar to a modern audience.

QUESTIONS:
The paper is very clear.  Other than the weakness mentioned above, I do not feel I have pressing questions.

LIMITATIONS:
Limitations are more than adequately addressed, as mentioned in the strengths section above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of discovering a hierarchy of skills based on the state-transition graph. The proposed approach builds on an existing method called the Louvain algorithm, which creates a hierarchy of state clusters; i.e., the lowest level will place nearby states in the same (small) cluster, the second level will place those smaller clusters in bigger clusters, and so on. The clusters are defined using a measure of modularity (nodes within a cluster have dense connections, but there are sparse connections between different clusters). The skills are then defined using the options framework, and their behavior is to take the agent from any state within the cluster to the adjacent clusters. The empirical evaluation presented indicates that the method is capable of finding such hierarchies of skills and that, in some cases, using these skills yields a better performance when learning to solve a task, when compared with existing methods. Additionally, the empirical evaluation demonstrates the method's efficacy under scenarios with continuous state spaces (assuming a discretization method) and under scenarios with many states. Finally, the authors present preliminary results for a possible extension of the method to an interactive case where the transition graph is also learned dynamically instead of being assumed available originally.\n\nSTRENGTHS:\n- The paper addresses a very interesting, novel, and important problem: to autonomously identify a hierarchy of skills with multiple levels (not only two as in most previous work), without specifically setting the number of levels it should have.\n- The proposed method is intuitive and has a different/creative look at a problem that has not been solved yet. \nThe paper is very clearly written, with a very good display of the empirical results, analyzing both qualitative and quantitative aspects of the learned skills.\n- The experiments conducted indicate that the hierarchies learned are presenting the desired behavior (i.e., clusters well connected within themselves but not well connected between each other).\n- The domains used in the empirical evaluation cover a wide range of scenarios: smaller, simpler MDPs, but also MDPs with many states and even an MDP with discretized continuous states. This helps demonstrate the effectiveness of the method at different levels of complexity.\n\nWEAKNESSES:\n- The method requires complete knowledge of the state-transition graph, which is often not available in real-life applications. However, I believe the paper still presents an important contribution and a first step towards solving the problem of finding multi-level hierarchies of skills. Additionally, the authors present preliminary results for an incremental version of the algorithm, where the skills are adjusted as the estimated model is learned.\n- Without limiting the number of clusters/levels in some way, the algorithm could end up returning a lot of skills, which could make the learning process later on more difficult instead of facilitating it (given the increased action space). The authors mention this in the paper and propose one possible improvement (to ignore lower-level clusters that contain a small number of states).\n- The method finds skills using only the transition graph, not the reward function. Thus, it would not be as useful in tasks with a large state space but where only a small subset of it is used for solving the task.\n\nQUESTIONS:\n- Why did the authors select Q-learning as the baseline for primitive actions instead of a more recent, and better-performing, algorithm?\n- It seems like the first two plots of Figure 4 do not present the results for Level 4. Could the authors please clarify this?\n- Currently, as per my understanding, all edges in the graph have the same weight. This would imply that when finding the skills in non-deterministic MDPs, a transition that happens with a 10% probability would have the same importance as one that always happens. Do the authors have an insight into how the algorithm could be changed to consider the transition probabilities?\n\nLIMITATIONS:\nThe authors discuss the limitations of the work. No major concerns.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}