SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 4b shows how the Louvain skill hierarchy changes with the size of the state space in the Office environment. The figure shows the depth of the skill hierarchy in a series of fifteen Office environments of increasing size, ranging from a single floor ( $\sim 10^3$  states) to one thousand floors ( $\sim 10^6$  states). The depth of the hierarchy increased very gradually with the number of states in the environment. The maximum hierarchy depth reached was eight in the largest environment tested.
Hierarchical versus flat arrangement of skills. An alternative to the Louvain skill hierarchy is a flat arrangement of the skills, where each skill calls primitive actions directly rather than indirectly through other (lower-level) skills. We expected the hierarchical arrangement to lead to faster learning than the flat arrangement due to the additional learning updates enabled by the hierarchical relationship between the skills. Figure 5 shows that this is indeed the case. In the figure, Louvain shows an agent that uses the Louvain skill hierarchy while Louvain flat shows an agent that uses the Louvain skills but where the skill policies call primitive actions directly rather than through other skills. In addition, the figure shows a number of agents that use only a single level of the Louvain hierarchy
Figure 4: (a) Learning performance in a two-floor Office containing 2537 states. (b) How the depth of the skill hierarchy scales with the size of the state space in multi-floor Office.
Figure 5: Learning curves comparing various different Louvain agents. An epoch corresponds to 100 decision stages in Rooms, 300 in Taxi, and 750 in Maze. The skill hierarchy contained three levels in Rooms and Taxi, and four levels in Maze.
(Level 1, Level 2, Level 3, or Level 4), with option policies that call primitive actions directly. Primitive actions were available to all agents. The figure shows that the hierarchical agent learns more quickly than the flat agent. Furthermore, the agents using individual levels of the Louvain hierarchy learn more quickly than the primitive agent but not as quickly as the agent using the full Louvain hierarchy.
Impact of the resolution parameter. When using very high values of the resolution parameter  $\rho$ , the Louvain algorithm terminates without producing any partitions. On the other hand, when using very low values of  $\rho$ , it runs until a partition is produced where all nodes are merged into a single cluster.
Figure 6 shows how changing the resolution parameter  $\rho$  impacts the Louvain cluster hierarchy in Towers of Hanoi and Rooms. In Towers of Hanoi, at  $\rho = 10$ , the output is a single level containing many small clusters comprised of three nodes. As  $\rho$  gets smaller, the cluster hierarchy remains the same until  $\rho = 3.3$ , at which point a second level is added. The first level remains identical to the single level produced at  $\rho = 10$ . The second level contains larger clusters, each formed by merging three of the clusters from the first level. As  $\rho$  is reduced further, additional levels are added to the hierarchy. When a new level is added to the hierarchy, the existing levels generally remain the same.
A sensitivity analysis on the value of  $\rho$  showed that a wide range of  $\rho$  values led to useful skills, and that performance gradually decreased to no worse than that of a primitive agent at higher values of  $\rho$ . Section E of the supplementary material contains full details of this sensitivity analysis and a more detailed discussion on the choice of  $\rho$ .
An important research direction for future work is incremental learning of Louvain skill hierarchies as the agent interacts with its environment—because the state transition graph will not always be available in advance. We explored the feasibility of incremental learning in the Rooms environment.
Figure 6: How the Louvain algorithm's output when applied to Towers of Hanoi and Rooms changes with the resolution parameter. The cluster hierarchy had a maximum of four levels in Towers of Hanoi and five levels in Rooms.
(a) Incremental Performance
(b) 30 States
(c) 60 States
Figure 7: (a) Performance when incrementally discovering Louvain options in Rooms. An epoch corresponds to 100 decision stages. (b-d) How the state transition graph and top-level partitions produced by the Update approach evolved as an agent explored Rooms. The hierarchy contained 2 levels after visiting 30 states, 3 levels after visiting 60 states, and 5 levels after observing all possible transitions.
(d) All States
The results are shown in Figure 7. The agent started with an empty state transition graph and no skills. Every  $m$  decision stages, it updated its state transition graph with new nodes and edges, and it revised its skill hierarchy using one of two approaches: Replace or Update.
In the first approach (Replace), the agent applied the Louvain algorithm to create a new skill hierarchy from scratch. In the second approach (Update), the agent incorporated the new information into its existing skill hierarchy, using an approach similar to the Louvain algorithm. This second approach starts by assigning each new node to its own cluster; the new nodes are then iteratively moved locally, between neighbouring clusters (both new and existing) until no modularity gain is possible. This revised partition is used to define an aggregate graph and the entire process is repeated on the aggregate graph, and the next aggregate graph, and so on, until an iteration is reached with no modularity gain. The cluster membership of existing nodes stays fixed; only new nodes have their cluster membership updated. The result of both approaches is a revised set of partitions, from which a revised hierarchy of Louvain skills is derived. Section I of the supplementary material presents pseudocode for the two incremental approaches.
Figure 7a shows the performance of the incremental agents. The agents started with only primitive actions; after decision stages 100, 500, 1000, 3000, and 5000, the state transition graph was updated and the skill hierarchy was revised. The figure compares performance to a Primitive agent and an agent using the full Louvain skill hierarchy, whose performance acts as a Ceiling for the incremental agents. Both incremental agents learned much faster than the primitive agent and only marginally slower than the fully-informed Louvain agent. The two incremental agents had similar performance throughout training but Replace reached a higher level of asymptotic performance than Update, as expected. The reason is that partitions produced early in training are based on incomplete information; Replace discards these early imperfections while Update carries them forward. But there is a trade-off: building a new skill hierarchy from scratch has a higher computational cost than updating an existing one.
Figures 7b-7d show the evolution of the partitions as the Update agent performed a random walk in Rooms. The partitions were updated after observing 30 states, 60 states, and all possible transitions. The figure shows that, as more nodes were added, increasingly higher-level skills were produced.
These results demonstrate the feasibility of learning Louvain skills incrementally. A full incremental method for learning Louvain skills may take many forms, and different approaches may be useful under different circumstances, with each having its own strengths and weaknesses. We leave the full development of such algorithms to future work.
Another direction for future work is extending Louvain skills to environments with continuous state spaces such as robotic control tasks. Such domains present a difficulty to all skill discovery methods that use the state transition graph due to the inherently discrete nature of the graph. If the critical step of constructing an appropriate graphical representation of a continuous state space can be achieved, all graph-based methods would benefit. Some approaches have already been proposed in the literature. We use one such approach [40, 20, 26] to examine the Louvain hierarchy in a variant of the Pinball domain [41], which involves manoeuvring a ball around a series of obstacles to reach a goal, as
Pinball Layout
Level 2
Figure 8: The layout of the Pinball environment—with the green ball in its initial position, the red goal, and several obstacles—and the Louvain cluster hierarchy produced by the Louvain algorithm.
shown in the leftmost panel of Figure 8. The state is represented by two continuous values: the ball's position in the horizontal and vertical directions. At each decision stage, the agent can choose to apply a small force to the ball in each direction. The amount of force applied is stochastic and causes the ball to roll until friction causes it to come to a rest. Collisions between the ball and the obstacles are elastic.
We sampled 4000 states, added them to the state transition graph, then added an edge between each node and its  $k$ -nearest neighbours according to Euclidean distance, assigning an edge weight of  $e^{-4d^2}$  to an edge that connects two locations  $u$  and  $v$  with Euclidean distance  $d$  between them. We then applied the Louvain algorithm to the resulting graph. The result was the cluster hierarchy shown in Figure 8. It had three levels. Moving between clusters in the top level corresponds to skills that enable high-level navigation of the state space, and take into account features such as the natural bottlenecks caused by the obstacles, allowing the agent to efficiently change its position. Moving between clusters in the lower-level partitions enable more local navigation.
Currently, Louvain skills at one level of the hierarchy are composed of skills from only the previous level. Such skills may not provide the most efficient navigation between clusters. Future work could consider composing skills from all lower levels, including primitive actions.
Because Louvain skills are derived from the connectivity of the state transition graph, we expect them to be suitable for solving a range of problems in a given environment. Examining their use for transfer learning is a useful direction for future work.
A possible difficulty with building multi-level skill hierarchies is that having a large number of skills can end up hurting performance by increasing the branching factor. One solution that has been explored in the context of two-level hierarchies is pruning less useful skills from the hierarchy [21, 42]. Further research is needed to address this potential difficulty.
A key open problem for graph-based skill discovery methods generally is how to best make use of state abstraction. We suggest that the most natural place to introduce state abstraction is when constructing the state transition graph itself. Instead of a concrete state, each node could represent an abstract state based on some learned representation of the environment. The proposed method—and any other existing graph-based method—could then directly use this abstract state transition graph to define a skill hierarchy.
Lastly, we note that the various characterisations of useful skills proposed in the literature, including the one proposed here, are not necessarily competitors. To solve complex tasks, it is likely that an agent will need to use many different types of skills. An important avenue for future work is exploring how different ideas on skills discovery can be used together to enable agents that can autonomously develop complex and varied skill hierarchies.
We would like to thank the members of the Bath Reinforcement Learning Laboratory for their constructive feedback. This research was supported by the Engineering and Physical Sciences Research Council [EP/R513155/1] and the University of Bath. This research made use of Hex, the GPU Cloud in the Department of Computer Science at the University of Bath.
[37] A. McGovern, R. S. Sutton, and A. H. Fagg. Roles of macro-actions in accelerating reinforcement learning. In Grace Hopper Celebration of Women in Computing, volume 1, pages 13-18, 1997.
[38] R. S. Sutton and D. Precup. Intra-option learning about temporally abstract actions. In Proceedings of the 15th International Conference on Machine Learning, pages 556-564. Morgan Kaufman, 1998.
[39] C. J. C. H. Watkins. Learning from delayed rewards. PhD thesis, King's College, Cambridge, United Kingdom, 1989.
[40] S. Mahadevan and M. Maggioni. Proto-value functions: A Laplacian framework for learning representation and control in Markov decision processes. Journal of Machine Learning Research, 8(10), 2007.
[41] G. Konidaris and A. Barto. Skill discovery in continuous reinforcement learning domains using skill chaining. In Advances in Neural Information Processing Systems 22, NeurIPS '09, pages 1015-1023. Curran Associates, Inc., 2009.
[42] D. M. Farahani and N. Mozayani. Evaluating skills in hierarchical reinforcement learning. International Journal of Machine Learning and Cybernetics, 11(10):2407-2420, 2020.