Accelerating Task Generalisation with Multi-Level Skill Hierarchies
Authors: Thomas P Cannon
Özgür Şimşek
Venue: ICLR 2025 Poster (2025)
Keywords: Reinforcement Learning
Generalisation
Hierarchical Reinforcement Learning
Primary area: reinforcement learning
TL;DR: This work presents Fracture Cluster Options (FraCOs), a novel multi-level hierarchical reinforcement learning framework that improves task generalisation.

Abstract: Developing reinforcement learning agents that can generalise effectively to new tasks is one of the main challenges in AI research. This paper introduces Fracture Cluster Options (FraCOs), a multi-level hierarchical reinforcement learning method designed to improve generalisation performance. FraCOs identifies patterns in agent behaviour and forms temporally-extended actions (options) based on the expected future usefulness of those patterns, enabling rapid adaptation to new tasks. In tabular settings, FraCOs demonstrates effective transfer and improves performance as the depth of the hierarchy increases. In several complex procedurally-generated environments, FraCOs consistently outperforms state-of-the-art deep reinforcement learning algorithms, achieving superior results in both in-distribution and out-of-distribution scenarios.

Published as a conference paper at ICLR 2025
ACCELERATING TASK GENERALISATION WITH MULTI-LEVEL SKILL HIERARCHIES
Thomas P. Cannon
Özgür Simşek
Department of Computer Science University of Bath Bath, United Kingdom
Department of Computer Science University of Bath Bath, United Kingdom
tc2034 (e03340bthhh... aapathh.c.ll uk
o.
ac uk
ABSTRACT
Developing reinforcement learning agents that can generalise effectively to new tasks is one of the main challenges in Al research. 'This paper introduces Frac- ture Cluster Options (FraCOs), multi-level hierarchical reinforcement learning method designed to improve generalisation performance. FraCOs identifies pat- terns agent behaviour and forms lemporally-extendde actions (options) based the expected future usefulness of those patterss, enabling rapid adaptation new tasks. In tabular settings, FraCOs demonstrates effective transfer and im- proves performance the depth of the hierarchy increases. In several complex sereduurally-eeeeratee environments, FraCOs consistently outperforms state-of- the-art deep reinforcement learning algorithms, achieving superior results in both in-distribution and out-of-distributin scenarios.
INTRODUCTION
fundamental goal of AI research is to develop agents that can leverage structured prior knowledge, either provided or learned, to act competently in unfamiliar domains (Pateria et al., [022]. This is common behaviour in animals. For example, many newborn mammals, such as foals, can walk shortly after birth due to innate motor patterns; and human infants display instinctive stepping mo- tions when supported (Adolph & Robinson 2013) Dominici et al.) 2011). These innate behaviours, shaped by evolution, act priors that guide goal-directed actions and enable rapid adaptation.
Humans are believed to organise behaviours into a hicrarchy of lemporally-cxtender actions, which helps break complex tasks into simpler, manageable steps (Rosenbloom & Newell 1986: Laird et al., 1987 Brunskill Li 2014). For instance, decision making in humans often involves planning with high-level actions such "pick up glass" "drive college," each of which comprises subtasks such as "reach for glass" or "pull door handle." These subtasks eventually decompose into basic motor movements. Notably, parts of this hierarchy shared between tasks; for instance, both "pick up glass" and "pull door handle" involve similar gripping movements. Such shared temporally- extended actions. often referred as skills, are reusable behaviours that provide sigmificant benefit when applied across tasks. Specifically, skills enable rapid learning of new tasks beyond those previously experienced by leveraging prior knowledge.
Inspired by this human capability, generating such hierarchical organisation in algorithms could allow artificial agents to adapt quickly to new tasks (Heess et al., 2016). By decomposing complex tasks into reusable, enabling faster learning and better generalisation.
actions, agents can leverage shared behaviours across tasks,
Despite advances made in representing, learning, and using behaviour hierarchies, generalising these behaviours across diverse tasks remains a significant challenge for artificial agents (Cobbe et al.) 2.19) Existing approaches struggle with effectively transferring skills to environments, limit- ing their ability to adapt to real-world scenarios (Pateria et al.) 2021).
Here introduce Fracture Cluster Options (FraCOs), framework for defining, generating and using multi-level hierarchical skills that are designed to maximise an estimate of their future use- fulness. We present rigorous empirical evaluation that shows that FraCOs substantially enhances

---

Published as a conference paper at ICLR 2025
out-of-distribution learning. It outperforms three ofate-of-the-arr baselines Proximal Policy Opti- mization (PPO) (Schulman et al. (017). Option Critic with PPO (OC-PPO) (Klissarov et al. 2017)
and Phasic Policy Gradient (Cobbe et al.) 2021) in both in-distribution and out-of-distribution learning across several environments from the Procgen benchmark (Cobbe et al. 2020).
2 BACKGROUND
Reinforcement learning aims to identify how agents should act in their environment to achieve their (Suutinn & Barto 2018). The state of the environment and the action choices of the agent in a given state may be discrete, continuous, or multidimensional. Most reinforcement learning problems are framed as Markov Decision Process (MDP), defined tuple (S, A, P. R. 7). where S is the set of possible states, A is the set of possible actions, is the transition probability function with P(s, a, s') indicating the probability of transitioning from state s to state s' after taking action a, R is the reward function with R(ss a, s') indicating the expected reward when transitioning from state 8 to state s' via action a, and 7 [0, 1] is the discount factor. A policy л maps states probability distribution over actions, guiding the agent's 's behaviour.
At each time step t 2 O, the agent observes the current state St and chooses action at based on its policy ग (st). The environment then transitions into state 8141 and the agent receives reward r=+1- A trajectory is the sequence of observations Su, ag 71, $1, 022 I's, S3, that reflects the agent's 's interacts with its environment. The agent's 's objective is to learn policy that maximises some well defined function of the reward it receives. In this paper, we aim to maximise the cumulative discounted return, G, = -i= *
We define an environment as the external system with which the agent interacts, characterized by (S, P). A task in given environment defines, in addition, reward function R and discount factor 7.
Hierarchical Reinforcement Learning (HRL) organises the decision making of reinforcement learning agents into multiple levels of abstraction. A widely used approach is the options frame- work (Sutton et al. 1999). An option 2 is tuple (4g, T2. B2), where I, is the option's 's initiation set, describing the set of states where the option can be initiated, T= is the option policy that governs action selection while the option is active, and B : S [0. 1] is the option's 's termination condition, expressing the probability of termination at a given state. Options provide a structured way to rep- resent skills, which are temporally-abstract actions that are expected to provide benefit when reused across tasks.
Generalisation in reinforcement learning refers the ability of an agent accumulate rewards in environments, or parts of environments, that it has not been explicitly trained on. Generalisation challenges can be categorised based on the relationship between training and testing distributions, either falling within independent and identically distributed (IID) scenarios or extending to out-of- distribution (OOD) contexts. Additionally, generalisation challenges can be classified by the features of the environment that change, including the state space, observation space, dynamics, and rewards. This classification leads to eight distinct generalisation challenges (Kirk et al. 202I).
In this work, we focus on generalisation performance in OOD tasks where the state space S and reward function vary, while the action space A and the underlying transition dynamics remain constant. This setup reflects real-world scenarios where agent, such as a robot, operates under fixed physical principles but encounters diverse tasks.
3 RELATED WORK
Policy transfer methods, such as Finn et al. (2017), Grant et al. 2018)) Frans al. (0118), Cobbe et al. (202T), and |Mazorre et al. (2022). have shown some success in improving generalisation. However, they often struggle in ofi-t---istributtin (OOD) settings, as they typically rely on task- specific adaptation. In contrast, skill transfer methods learn reusable sub-policies, such as options, that can be flexibly composed, offering a more modular and adaptable framework for transfer learn- ing. By decomposing complex tasks into smaller, reusable components, skill transfer preserves and reuses prior knowledge in novel situations, potentially addressing challenges in OOD generalisation (Konidaris & Barto! 2007 Barreto al. 2019 Tessler et al. 2017| Mann & Choe 2013). However,
2

---

Published as a conference paper at ICLR 2025
existing approaches are limited to learning set of skills that choose directly among primitive tions, lacking the ability to operate in a truly hierarchical manner, being able to choose among both primitive actions and other skills.
A small number of methods have been proposed in the literature for learning multi-level hicrar- chies (Riemer et al. 2018 Evans & Şimsek 2023 Levy et al. 2019 Fox et al. 2017) but these methods have not addressed skill transfer as a mechanism for accelerating task adaptation and gen- eralisation. Additionally, some of these methods, for instance, those by Levy al. (2019) and Evans & Şümsek 2023), face difficulties in large, complex problems, for example, when using pixel pixel-based representations. Furthermore, state-based sub-goal methods, including Levy & Simsek (2023), struggle to transfer learned sub- uubgoolss to different state spaces and fail to account for variability in action sequences required to reach these sub- sub-goals; for example, "booking a holi- day" could involve "calling travel agent" or "using the internet," each demanding different skills. In contrast, FraCOs avoids creating state-based sub-goals, providing more flexible framework for transfer across state spaces.
Levy et al.) (2019) and Evans
Our work is closely related to Discovery Deep Options (DDO) by Fox et al. (0117). DDO employs an expectation gradient method construct hierarchy top-down of options from expert demonstra- tions. However, DDO does not optimise for generality and it remains unclear how the discovered options perform in unseen tasks. Moreover, the reliance demonstrations limits the development of increasingly more complex abstractions. In comparison, FraCOs builds bottom-up, forming pro- gressively more complex abstractions and selecting options based their expected usefulness in future tasks, directly addressing generalisation challenges.
Also related our work is Hierarchical Option Critic (HOC) by Riemer al. (2018) HOC gen- eralises the option-critic framework introduced by Bacon et al. (2177) to a multi-level hierarchy. HOC learns all options simultaneously during training. Both option-critic and HOC suffer from option collapse, where either all options converge to the same behaviour or one particular option is chosen consistently (Harutyunyan et al. 2019). Additionally, the added complexity of option-critic methods can slow learning compared to non-hierarchical approaches such as PPO (Schulman et al.) 2017 Zhang & Whiteson 2019). FraCOs addresses these limitations by naturally preventing option collapse through its option selection process.
4 FRACTURE CLUSTER OPTIONS
We hypothesise that identifying reoccurring patterns in an agent's behaviour across successfully completed tasks will improve performance on future tasks that the agent has not yet experienced. Correspondingly, we propose an approach to multi-level skill discovery that consists of three stages: (1) identifying similar patterns in an agent's behaviour across multiple tasks, (2) selecting the most useful of these patterns those considered to be the most likely to appear in trajectories of all pos- sible tasks, and (3) using these identified patterns basis for generating options for future use. In this section, we discuss each stage in turn.
4.1 IDENTIFYING PATTERNS IN AGENT BELIVIIOUR
Our objective is to identify and cluster the most useful patterns in agent behaviour that lead to successful outcomes. To achieve this, we require a method for capturing and analysing patterns produced during task execution. To this end, we introduce the concept of a fracture, defined as a sequence of actions that start at a specific state. A fracture is a tuple (s, aq 02 ...,... oz), where is the start state and 01, a2,..., .... is sequence of i actions that can be initiated at state s.
Given a set of trajectories T from tasks that an agent has experienced, we can generate a set of candidate fractures. For a single trajectory 7 E T of length n, let F. denote the set of all fractures that can be derived from this trajectory:
F = {(1,, 41 a++l ((4b011)) |0 < L b))
(1)
where the parameter b controls the temporal length of the fracture. We can repeat this for all 7 E T such that + (Fi, F2, F2,..... ..... FiTT } is the complete set of fractures derived from all trajectories in T.
3

---

Published as a conference paper at ICLR 2025
Can fractures capture the underlying structure of the environment? We explore this question in Four Rooms, a classic grid- based reinforcement learning environment, consisting of four rooms, connected to each other with narrow doorways. The environment has single goal state; the agent receives positive reward upon reaching this state, which terminates the episode. This environment is depicted in the top left corner in Figure T Further details can be found in Appendix [A.7] In all of our grid-worid implementations, the agent can observe only a 7x7 area centred on itself and scalar indicating the direction of the reward. This is similar to MiniGrid (Chevalier- Bisiierr-Boivertt al. 2024). except that our observations are ego-centric.
We train tabular Q-learning agents multi- ple versions of the environment with different goal states, generating trajectories for both the trained agents and an agent selecting actions randomly. We then create fractures following Equation D with b = 2. To reveal the struc- tural differences between the fractures derived from the random agent and the trained agent, we use UMAP (Meinnes et al.] 2018)) di- mensionality reduction technique that projects the high-dimensional fracture data into two di- mensions. UMAP is particularly useful for this task because it preserves local similarities within the data. We show the resulting two- dimensional visualisation in Figure ! The fig- ure reveals near- uniform distribution for the agent acting randomly, while the fractures from trained agents form distinet clusters, reflecting the underlying structures in successful trajecto- ries. This observation is consistent across other environments, shown in Appendix A.7.4
Fracture Visualisations in 2D Latent Space
40
Random Trained
30
20
10-
-2
-30
-30
-20
-10
10
20
30
40
Figure 1: A two-dimensional representation of the fractures (b = 2) derived from agents acting for 10,000 time-steps in Four Rooms.
To identify clusters of fractures, we employ unsupervised clustering techniques. Specifically, we HDBSCAN (Campello et al.) 2013) for all tabular methods in this work. Figure 2 shows four clusters randomly selected for visualisation, after fractures with chain length of b = are grouped into clusters using HDBSCAN. For each of the four clusters, the visualisation shows all fractures within that cluster, demonstrating that, despite differences in starting states, action sequences, and final states, the fractures within each cluster share similar semantic meanings.
Discovered Fracture Clusters
Figure 2: Four examples of discovered fracture clusters (b = 4) from agents trained in Four Rooms. Each cluster represented by colour. In the four examples, the green circles represent possible starting states, blue arrows indicate actions, the width of the arrows shows the frequency of the state- action pair within the cluster, and the red circles indicate the final states of the fracture.
4

---

Published as a conference paper at ICLR 2025
4.2 SELECTING USEFUL FRACTURE CLUSTERS
Fracture clusters formed in Section 4.1 identify behaviours that are similar to each other; however, the number of clusters identified can be very high, with some being highly task specifie. Developing all clusters into options may burden the agent with action choices that are not particularly useful. therefore essential to identify the clusters that are most likely to be useful in future tasks.
Consider the hypothetical scenario in which we can observe all possible trajectories across all possi- ble tasks. We will consider trajectory to be successful if its cumulative return exceeds predefined threshold, similar to the criterion proposed by Chollet (2019) Table IO shows all minimum re- turns used in our work. We use Xw to denote the set of all tasks with successful outcomes. this ideal setting, the set of all successful fractures, denoted by Φw. can be derived from the set of all successful trajectories, denoted by Tw.
To sensibly select fracture clusters, we must evaluate their potential for reuse in future tasks. We do this by defining the usefulness U of fracture cluster é based on its likelihood of contributing to success across tasks. Specifically, usefulness is determined by three factors, described below.
Appearance probability, (0). For given fracture cluster o, this is the probability that a randomly selected successful trajectory E Tw. from randomly selected successful task Iw E Nw. will contain at least one fracture o E Φ.
Relative frequency, Py(ő). For a given fracture cluster o, this is the proportion of times that some fracture e E Φ appears among all successful fractures ow.
Entropy of usage, (Ø). For given fracture cluster ã. this is the entropy of at least one @ E & appearing across all successful trajectories Tw
The usefulness of fracture cluster Ø is defined as the mean of these three factors:
Ug
(T) + ( + H())
Appendix A.17 includes an ablation study that explores the impact of each term.
In an ideal scenario, we could observe all possible tasks and trajectories and directly calculate the usefulness Uş of each fracture cluster. This is generally not possible. Instead, we must rely on available data, using the tasks and trajectories encountered during training. Let index individual tasks and N be the total number of experienced tasks. We form a from the N experienced tasks.
Estimating appearance probability. We approximate Pa (Φ) by using a Bayesian approach, mod- elling the probability that at least one fracture likelihood with Beta conjugate prior. The prior parameters and B, both set to 1, reflect an unin- formative prior. We define the appearance indicator Sn. to be equal to if any fracture & E Ø appears in trajectory Tre and 0 otherwise.
appears in successful trajectory binomial
Estimating relative frequency. estimate P)(Ø) by counting the occurrences of E Ø in P... which we denote as count(ő, +.,, ).). and dividing this count by the total number of fractures in T.-
Estimating entropy. The entropy H()) is approximated using Shamnon's entropy formulation Shannon 1948). Specifically, the term count)
represents the empirical probability of observing
[Tol
some fracture & E a trajectory Tw. This probability is used to compute the entropy. The nor- malisation factor N₂ ensures that the entropy is scaled appropriately relative the total number of unique fracture clusters Ø.
We derive the full approximation in Appendix A.2 shown below:
The result is expressed as the expected usefulness
Σ N
n= Cn ta +
count( (
P.))
count (,, Tw
count(O, Tw)
E|U3] =
logN;
(3)
P
[Ta]
Tw E Tw
Aspearance Probubillty
Relative Froquency
Estimated Eneropy
5

---

Published as a conference paper at ICLR 2025
Figure 3: The eight fracture clusters with the highest expected usefulness in Nine Rooms. Expected usefulness decreases from left to right in the top row, then from left to right in the bottom row. Green points represent possible starting states, blue arrows indicate actions, the width of the arrows shows the frequency of the state-action pair within the cluster, and the red points indicate the final states of the corresponding fracture.
Fracture clusters that yield the highest expected usefulness will be selected be developed into op- tions. To evaluate the effectiveness of Equation 3 we train an agent to reach 100 randomly selected goal states in the Nine Rooms environment, shown in Figure 3 We then collect evaluation trajecto- ries, form fractures with chain length of 4, cluster the fractures, and compute their expected usefulness. In Figure we plot the eight fracture clusters with the highest expected usefulness. The fracture cluster with the highest expected usefulness moves the agent from the starting state in all sensible directions, without repetitions movements. The majority the other fracture clusters transverse bottlenecks.
Forming multiple levels the skill hierarchy. Once the most useful fracture clusters have been identified, they can be converted into options, extending the agent's action space; this is detailed in Section 4.3 When learning a new task, the agent can now choose from both primitive actions and these higher-level behaviours. To build additional levels of the hierarchy, the process of iden- tifying and clustering fractures is repeated. In subsequent iterations, trajectories- and consequen- tially, fractures-mat consist of mix of primitive actions and higher-level options. This iterative approach naturally leads to the creation of a multi-level hicrarchical structure, where each level behaviours.
captures increasingly complex temporally-exee
4.3 USING FRACTURE CLUSTERS
To leverage the most useful fracture clusters, transform them into options call Fracture Clus- ter Options, or FraCOs. A FraCO z is characterised by an initiation set Ig, a termination condition 3g, and policy Tg.
Initiation set. The initiation set 12 defines the states in which the FraCO z can be initiated. To construct I,, we first consider all possible action sequences of length b (the chain length) denoted as (01, 42,.... as). where E A, = 1....bb. b. We can now define the set of all possible fractures
state s as follows:
F, {s, a) A*),
where à represents the set of all action sequences of length b.
For each fracture E Fg. we can estimate the probability that it belongs to a FraCO z. The set of fractures assigned cluster z in state can be defined as:
G t6 F, P(o E 2) 0),

---

Published as a conference paper at ICLR 2025
where P(ò 2) denotes the estimated probability that fracture & belongs to cluster 2, and 0 is threshold parameter. The method for estimating P(ø E 2) can vary depending on the implementa- tion. In our tabular implementation. In our deep implementation, a neural network is trained to predict this probability. FraCO z can be initiated in state if Gz,s s is not empty:
directly use the prediction function provided by HDBSCAN.
I. { *0}
(5)
Policy. When FraCO 2 is initiated in state 8, follows the policy TT. as described in Algo- rithm [1 The policy selects the fracture oz (s, a) from Gz., that has the highest probabil- ity p(o. 2) and then executes the sequence of actions a = (a1, dg......p.. If one of the selected actions is another FraCO z', the agent must compute Gz,, and recursively call the policy until the option terminates.
Algorithm 1: Option policy ny <2(Gz,, 2,9: 2,
Input: Ga,,, set options Z, state 8 Initialize: Select the fracture Φ. = (s, a) from Cz,. with the highest probability P(ø. E), where is sequence of actions (a1, a2,.....5) for each action a, in
if a, is a primitive action then Execute aj, resulting in new state s; else
Termination condition. The FraCO termi- nates under conditions: when all actions in the selected fracture Φz have been executed or when the agent attempts to execute nested FraCO z' but cannot find a matching fracture (Gz', = 0). The termination condition (s) end is therefore defined as follows:
a, is another option z': Compute G.'...
Recursively call TT (G=', 7,8);
end
|
all actions in Φz have been executed if the next action is z' and Gz',= otherwise
(s)s
(6)
Learning with FraCOs. FraCOs are fixed once created. The agent learns to choose between primitive actions and available FraCOs using standard reinforcement learning algorithms (e.g.. Q- learning for tabular settings, PPO for deep learning implementations).
Example FraCO usage. An agent in state s considers all possible fractures in that state, Fs. For each fracture Fs, the agent then estimates the probability P(ś 2;) that the fracture á belongs to 2,, for each FraCO E Z. If a fracture has a probability above the threshold 0, then 2; becomes available as an option.
The agent's policy π selects from among the set of available FraCOs and primitive actions. Suppose it selects FraCO 21. The agent picks the fracture oz, with the highest probability P(e,, E 2 and executes its action sequence.
As the agent executes these actions, if one of these actions is another FraCO, say 22, then the agent forms new F3 for the new current state s', estimates P( E 22), and selects the most probable fracture for 22, which is above the threshold 0. If the agent cannot find a fracture such that P( E 22) > 0, then the option terminates.
This process repeats until the termination condition Bs, is met, completing the execution of FraCO 21.
EXPERIMENTAL RESULTS
We evaluate FraCOs three different experiments. The first experiment focuses on OOD reward generalisation tasks using tabular FraCOs agent in the Four Rooms, Nine Rooms, and Ramesh Maze (Ramesh et al. 2019) grid-world environments. The second experiment examines OOD state generalisation tasks within novel environment called MetaGrid, explained in detail in Ap- pendix A.7 The final experiment evaluates a deep FraCOs agent, implemented with a three-level hierarchy using PPO in the procgen suite of environments (Cobbe et al. 2020

---

Published as a conference paper at ICLR 2025
Interquartile Mean Episodic Returns of Different Depths of FraCOs
Evaluation Episode Retur
Four Rooms
Nine Rooms
Ramesh Maze
1.0
0.5
Depth
Depth
Depth
Depth
0 Learning
8
40 60
80
100
50
100
150
200
250
Global step (x10?)
Figure 4: Episodic returns with a tabular FraCOs agent trained in the Four Rooms, Nine Rooms, and Ramesh Maze environments. Results show interquartile means of 10 independently seeded experiments, with shaded areas indicating the standard error.
We compare FraCOs' performance with CleanRL's Procgen PPO and PPG implementations (Huang al. 2022) and Option Critic with PPO (OC-PPO) (Klissarov et al. 2017). Further details on baseline implementations are provided in appendices A. a.13
In the grid-world environments, the agent receives reward of +1 for reaching the goal and an additional -0.001 at each time step; episodes have maximum length of 500 time steps; the fracture chain length is set at = In Procgen environments, we set to Further details on chain-length selection provided in Appendix A.8
5.1 EXPERIMENT 1: TABULAR REWARD GENERALISATION
In this experiment, we evaluate FraCOs in the Four Rooms, Nine Rooms, and Ramesh Maze en- viromments, using fixed state space while varying the reward function R. During the FraCOs discovery phase, a tabular Q-learning agent is trained on 50 distinct tasks, each with unique re- ward location. After completing training for each task, the top 20 FraCOs are extracted from the agent's 'ss final trajectories, as described in Section 4 and incorporated into the action space. This pro- cess is repeated iteratively, corresponding to each level of the hierarchy, with FraCOs being added incrementally at each stage.
In the evaluation phase, we create four new agents, each with access to progressively deeper hierarchy of FraCOs. The agents trained on 10 unseen test musks-each with a unique reward location and evaluated periodically. As shown Figure 4 results demonstrate that learning is progressively accelerated unseen tasks as the depth of the hierarchy increases.
5.2 EXPERIMENT 2: TABULAR STATE GENERALISATION
In this experiment, we evaluate FraCOs in a novel environment called MetaGrid, designed to test state generalisation. MetaGrid is navigational grid-world constructed from structured 7x7 building blocks, which can be combined in various ways to create novel state spaces while preserving areas of local structure. The agent observes environment through a 7 x window, consistent with our other grid-world environments. For more details on MetaGrid, refer to Appendix[A.7
During the FraCOs discovery phase, tabular Q-learning agent trained at each hierarchy level on 100 randomly generated 14 14 MetaGrid tasks. A task in this experiment corresponds unique state space S and reward location R. After training, 20 FraCOs are extracted from final trajectories and incorporated into the action space.
In the evaluation phase, for each hierarchy level, a separate agent created. These agents are then trained in two settings: (1) previously unseen 14 14 domains and (2) larger 21 x 21 domains. Peri- odic evaluation episodes are conducted during training to track performance. As shown in Figure these results also demonstrate that the rate of learning increases with deeper hierarchies.
8

---

Published as a conference paper at ICLR 2025
Interquartile Mean Episodic Returns of Different Depths of FraCOs
Evaluation Episode Return
Unseen 14x14
Unseen 21x21
0.5
Depth
0.0
Depth
Depth
Depth
O Learning
6
20
30
40
50
60
Global step (x10*)
Figure 5: Episodic returns for tabular FraCOs in unseen MetaGrid domains of varying sizes. Results are the interquartile means of 10 independently seeded experiments, with shaded areas indicating the standard error.
5.3 EXPERIMENT DEEP STATE AND REWARD GENERALISATION
In this experiment, test FraCOs in OOD tasks from the Procgen benchmark (Cobbe al. 2020 which is a suite of procedurally generated arcade-style environments designed to assess generalisa- tion across diverse tasks; further details are provided in Appendix A.7 We compare FraCOs with three methods, Option Critic with PPO (OC-PPO) (Klissarov et al. 2017), PPO (Schulman et al.) 2017). and Phasic Policy Gradient (PPG) (Cobbe al. 2021 across eight Procgen environments, where each task has unique unseen state space S and reward function R.
FraCOs modifications. To handle the challenges of apply- ing traditional clustering to high-din -ith-emimniionl pixel data, we simplify the approach by grouping fractures with the same action sequences, regardless of state differences. Addition- ally, a neural network used to estimate initiation states and policies, which reduces the computational burden of performing discrete search over the complex 64 x x state space and managing 15 possible actions during mil- lions of training steps. These modifications do not change the fundamental approach of FraCOs, only its implementa- tion. Full details of these modifications are provided in Ap- pendix 10 with further information the experiments, baselines, and hyperparameters in Appendix A.1I
Average IQM in environments
Average IOM evaluation return
OOD
0.8
HID
0.6
FraCOs SSR
FraCOs
PPG
OC-PPO
PPO
Figure 7: Mean min-mac normalised IQM returns with standard erross across Procgen environments.
FraCOs and OC-PPO both learn options during 20-million time-step warm-up phase, with tasks drawn from the first 100 levels of each Proegen environment. FraCOs learns two sets of 25 options, corresponding to different hierarchy levels, while OC-PPO learns a total of 25 options. After the warm-up, the policy over options is reset, and training continues for an
Normalised IQM Episodic Returns
Climber
Plunder
Ninja
Normalised Return
1.0
PraCOs SSR IID
RraCOs
0.8
PPO
PPG ID
0.6
GC-IFQ ID
0.4
PraCOs 00D
0.2
PPO 000
PPG 000
Global step (x 10*)
Figure 6: A comparison of learning curves of sample of three Procgen environments.

---

Published as a conference paper at ICLR 2025
Environment specific IID and OOD IQM
1.0
0.8
FraCOs SSR
1QM evaluation
0.6
FraCOs
PPG
0.4
OC-PPO
PPO
0.2
OOD
0.0
Coinrun
Ninja
Plunder
Bigfish
Fruithot
Leaper
Climber
Dodgeball
Figure 8: Min-max normalised IQM returns with standard errors in individual Procgen environ- ments.
additional 5 million time steps. During this phase, we periodically conduct evaluation episodes on both IID and OOD tasks, with OOD tasks drawn from Procgen levels beyond 100.
We test two versions of FraCOs: one where the policy over options is completely reinitialised after the warm-up phase, and another that transfers a Shared State Representation (SSR), referred to as FraCOs-SSR. In the SSR, a shared convolutional layer encodes the state, followed by distinct linear layers for the critic, policy over options, and option policies. These convolutional layers are not reset after warm-up, enabling shared feature representation across tasks. Since OC-PPO inherently relies on a shared state representation to define its options and meta-policy, comparing with FraCOs-SSR offers fair evaluation. Without this shared representation, OC-PPO struggles to maintain stable and meaningful options across tasks. Despite this adjustment for fairness, we find that FraCOs, even without SSR, consistently outperforms all baselines. For further implementation details of FraCOs-SSR, refer to Appendix A..1
Figure provides the mean min-max normalised interquartile mean (IQM) across all Procgen envi- ronments. Figure 8 provides the results each environment. We also provide three sample learning curves in Figure Each experiment was repeated with eight seeds. On average, observe that FraCOs and FraCOs-SSR are able to improve both IID and OOD returns over all baselines.
DISCUSSION AND LIMITATIONS
We introduced Fracture Cluster Options (FraCOs) as novel framework for multi-level hierarchical reinforcement learning. In tabular settings, FraCOs demonstrated accelerated learning on unseen tasks, with performance improving as the depth of the skill hierarchy increased. deep reinforce- ment learning experiments, FraCOs outperformed state-of-the-a algorithms OC-PPO, PPO, and PPG on both in-distribution (IID) and out-of-distribution (OOD) tasks, showcasing its potential for robust generalisation.
In current form, FraCOs has a number of limitations. Clustering methods, such as HDBSCAN, struggle to accurately predict cluster assignments for new data points environments grow more complex. Simplified techniques and neural networks were introduced here to address this difficulty but further research into scalable clustering solutions is needed. Additionally, this work focused on discrete action spaces; extending FraCOs to continuous action spaces remains as future work. De- spite these limitations, FraCOs provides strong foundation for advancing hierarchical reinforce- ment learning and improving generalisation across tasks.
ACKNOWLEDGEMENTS
This work was supported by the UKRI Centre for Doctoral Training in Accountable, Responsible and Transparent AI (ART-AD) [EP/S023437/1] and the University of Bath. We would like to thank the members of the Bath Reinforcement Learning Laboratory for their constructive feedback. This research made use of Hex, the GPU Cloud in the Department of Computer Science at the University of Bath.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Karen E Adolph and Scott R Robinson. 15 the road to walking: What learning to walk tells us about development. The Oxford Handbook of Developmental Psychology, Vol. 1: Body and Mind, 1: 403, 2013.
Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
André Barreto, Diana Borsa, Shaobo Hou, Gheorghe Comanici, Eser Aygün, Philippe Hamel, Daniel Toyama, Shibl Mourad, David Silver, Doina Precup, et The option keyboard: Combining skills in reinforcement learning. Advances in Neural Information Processing Systems, 32, 2019.
Emma Brunskill and Lihong Li. Pac-inspired option discovery in lifelong reinforcement learning. In International conference on machine learning, pp. 116-324. PMLR, 2014.
Ricardo JGB Campello, Davoud Moulavi, and Jörg Sander. Density-based clustering based on hier- archical density estimates. In Pacific-Asia conference on knowledge discovery and data mining, pp. 160-172. Springer, 2013.
Maxime Chevalier-Boiswe Bolun Dai, Mark Towers, Rodrigo Perez-Vicente Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Mod- ular & customizable reinforcement learning environments for goal-oriented tasks. Advances in Neural Information Processing Systems, 36, 2024.
François Chollet. On the measure of intelligence. arXiv preprint drXiv:1911.01547, 2019.,
Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman. Quantifying generaliza- tion in reinforcement learning. International conference on machine learning, pp. 1282-1289, PMLR, 2019,
Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning, pp. 2048- 2056. PMLR, 2020.
Karl Cobbe, Jacob Hilton, Oleg Klimov, and John Schulman. Phasic policy gradient. In Interna- tional Conference on Machine Learning, pp. 2020-2027. PMLR, 2021.
Nadia Dominici, Yuri Ivanenko, Germana Cappellini, Andrea d' Avella, Vito Mondi, Marika Ci- cehese, Adele Fabiano, Tiziana Silei, Ambrogio Di Paolo, Carlo Giannini, et al. Locomotor primitives in newborn babies and their development. Science, 334(66558 2011.
Joshua B Evans and Özgür Şimşek. Creating multi-level skill hierarchies in reinforcement learning. Advances in Neural Information Processing Systems, 26:48472--48884 2023.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. International Conference on Learning Representa- tions, 2019. URL https: http::::///peeeevviiiee
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp. 1 1126-11333 PMLR, 2017.
Roy Fox, Sanjay Krishnan, Ion Stoica, and Ken Goldberg. Multi-level discovery of deep options. 2017.
arXiv preprint arXiv:
Kevin Frans, Jonathan Ho, Xi Chen, Pieter Abbeel, and John Schulman. Meta learning shared hierarchies. In International Conference on Learning Representations, 2018.
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient- based meta-learning as hierarchical bayes. In International Conference on Learning Representa- tions, 2018. URL htt.t::tt///tt..t. /ttp::/::////eee/eeeeeeee net. woi.ffommmmin_ou000 -
Anna Harutyunyan, Will Dabney, Diana Borsa, Nicolas Heess, Remi Munos, and Doina Precup. The termination critie. arXiv preprint arXiv: 1902.099900999 2019.

---

Published as a conference paper at ICLR 2025
Nicolas Heess, Greg Wayne, Yuval Tassa, Timothy Lillicrap, Martin Riedmiller, and David Silver. Learning and transfer of modulated locomotor controllers. arXiv preprint arXiv:661000512,, 2016.
Shengyi Huang, Rousslan Fermand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Ki- nal Mchta, and JoAGo GM AraAšjo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1-18, 2022.
Diederik P Kingma and Max Welling. arXiv: 2013.
Auto-encoding variational
bayes.
arXiv preprint
Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. A survey of generalisation in deep reinforcement learning. arXiv preprint urXiv:2111.0977 2021.
Martin Klissarov, Pierre-Lue Bacon, Jean Harb, and Doina Precup. Learnings options end-to-end for continuous action tasks. arXiv preprint arXiv:1712.00004, 2017.
George Dimitri Konidaris and Andrew G Barto. Building portable options: Skill transfer in rein-
forcement learning. In International Joint Conference on Artificial Intelligence, volume 7, pp. 895-900, 2007.
John E Laird, Allen Newell, and Paul S Rosenbloom. Soar: An architecture for general intelligence. Artificial intelligence, 33(1):1-64, 1987.
Andrew Levy, George Konidaris, Robert Platt, and Kate Saenko. Learning multi-level hierarchies with hindsight. Proceedings of International Conference on Learning Representations, 2019.
Timothy A Mann and Yoonsuck Choe. Directed exploration in reinforcement learning with trans- ferred knowledge. European Workshop on Reinforcement Learning, pp. 59-76. PMLR, 2013.
Bogdan Mazoure, Ilya Kostrikov, Ofir Nachum, and Jonathan J Tomppon. Improving zero-shot generalization in offline reinforcement learning using generalized similarity functions. Advances in Neural Information Processing Systems, 35:25088-255 2018888-111,,,111111111111111111 2022.
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv: 2018.
arXiv preprint Volodymyr Mnih. Asynchronous methods for deep reinforcement learning. 2016.
arXiv:1602.01788, 2016.
Shubham Pateria, Budhitama Subagdja, Ah-hwee Tan, and Chai Quek. Hierarchical reinforcement learning: A comprehensive survey, ACM Computing Surveys (CSUR), 5-(5):1-35, 2021.
Rahul Ramesh, Manan Tomar, and Balaraman Ravindran. Successor options: An option discovery framework for reinforcement learning. arXiv preprint arXiv:1905.0573/ 2019.
Matthew Riemer, Miao Liu, and Gerald Tesauro. Learning abstract options. Advances in neural information processing systems, 31, 2018
Paul S Rosenbloom and Allen Newell. The chunking of goal hierarchies: A generalized model of practice. Machine learning-an artificial intelligence approach, 2:247, 1986.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1xX.:01707.0634777 2017.
Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379-422 1948.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A frame- work for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181- 211, 1999.
12

---

Published as a conference paper at ICLR 2025
Chen Tessler, Shahar Givony, Tom Zahavy, Daniel Mankowitz, and Shie Mannor. A deep hierarchi- cal approach to lifelong learning in minecraft. Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
Mark Towers, Jordan K. Terry, Ariel Kwiatkowski, John U. Balis, Gianluca de Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Arjun KG, Markus Krimmel, Rodrigo Vicent,icente, An- drea Pierré, Sander Schulhoff, Jun Jet Tai, Andrew Tan Jin Shen, and Omar G. Younis. Gymma- sium, March 2023. URL.ttps:://enoee zenodo wttp://oo///////.00000000
Shangtong Zhang and Shimon Whiteson. Dac: The double actor-critic architecture for learning options. Advances Neural Information Processing Systems, 32, 2019.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 GLOSSARY OF TERMS AND DERIVATIONS
Term/Symbol
Definition/Derivation
MDP (Markov Deci-
A mathematical framework for modelling decision-making defined by the tuple (S, A, R, 7). where:
sion Process)
Set of possible states.
A: Set of possible actions.
P: Transition probability function, P(s, s').
R: Reward function, R(s, s's))
Discount factor, 7 E [0, 1].
5
Set of possible states in an MDP. Set of possible actions in an MDP.
P
Transition probability function; P(s, s, gives the probability of tran- sitioning from state s to s' after action a.
11
Reward function; R[(s, a, 8') is the reward received when transitioning from to s' via action
Discount factor in an MDP, 7 E [0, 1]. representing the importance of
future rewards.
State of the agent at time step t.
02
Action taken by the agent at time step L.
Next state after taking action o, from state St.
(s;))
Policy of the agent, mapping state s to a probability distribution over actions.
G₁
G2 sequence of states, actions, and rewards experienced by the agent: (su, "', 81. (2.'2....)
Cumulative discounted return from time defined
7"
Trajectory (T E
Task
task defined as a unique MDP.
Reward generalisa-
MDPs with R values outside of the training distribution, while S, A, and P. remain within distribution.
tion tasks
State generalisation
MDPs with S and R values outside of the training distribution, while A, and P, remain within distribution.
tasks
Option
In HRL, temporally extended action, defined by:
I: Initiation set.
T Intra-option policy.
(s)): Termination condition.
I
Initiation set of an option; the set of states where the option can be initi- ated.
Intra-option policy; the policy followed while the option is active. Termination condition an option; gives the probability of terminating the option in state
((8)
FraCOs (Fracture Cluster Options)
The proposed method for defining, forming, and utilizing multi-level hi- erarchical options based on expected future usefulness.
A fracture; a state paired with sequence of actions:
(St, Og, Ot-1, 01+b-1)
Continued on next page...
14

---

Published as a conference paper at ICLR 2025
Term/Symbol
Prfinition/Derivaaie
b
Chain length of a fracture; specifies the number of actions following the state &t.
F
Set of fractures derived from a single trajectory:
F {(Et, at, 0t+l.-.. ((+b-1) 10 S
b}
Complete set fractures from all trajectories: (Fi, F2...., FiTi}
fracture cluster; group fractures with similar behaviours. A measure evaluate fracture clusters based on their potential for reuse in future tasks:
Usefulness
Metric
(U())
1 (Plō To [w] + Ple Pw] + H(Ø Xw)) =
Expected Usefulness terms
S Appearance indicator; Gn = if any fracture & E ē appears in trajectory Tri clse O.
8: Parameters of the Beta prior distribution, typically set to
1.
N: Total number of experienced tasks.
D Set of all successful fractures.
Te: Set of successful trajectories.
Ng: Normalization constant for entropy calculation.
Derivation of Ap- pearance Probabil-
Using Bayesian inference, the appearance probability is estimated as:
Σn Sn +
ity
Ple E Tw xwl
N + N+++3
Where Gn observations modeled Bernoulli random variables with a Beta prior.
Derivation of Rela- tive Frequency
Calculated as:
count (ount(ō,c 2w))
P 2..]
(8)
+ an
Where count(o, ..:) is the number of times fractures in Φ appear among all successful fractures &.. The entropy of a fracture cluster's 's usage is:
Derivation of En-
tropy of Usage
count( (annt( Tw)
count O, Tw) [Tue
H(6 X.,) = Σ
logN;
TT0l
TWETT
This measures the diversity of the fracture cluster's 's usage across tasks. Parameters of the Beta distribution used in Bayesian estimation; set to
a, |
- 1,9 for uninformative prior.
Sn
Appearance indicator for task n; S. if any fracture in appears trajectory else O).
Set of all Fracture Cluster Options (FraCOs).
2
single FraCO: option derived from fracture cluster.
Continued next page...
15

---

Published as a conference paper at ICLR 2025
Term/Symbol
Definition/Derivation
I
Initiation set of FraCO 2; states where can be initiated:
1 {8 E S G2,s * 0)
B2
Termination condition of FraCO 2; 2 terminates when all actions in the selected fracture have been executed or when no matching fracture is found:
if all actions in dg have been executed if next action is z and G otherwise
B(x)
0
T=
Policy of FraCO defines the sequence of actions when the option is active (see Algorithm in the paper).
G2,8
Set of fractures assigned to cluster in state st
G2,- {6 e F,
Threshold hyperparameter for cluster membership: determines a frac- ture belongs to a cluster based probability.
N
Total number of experienced tasks or samples.
Set of all trajectories.
An individual trajectory from the set T.
Ta
successful trajectory; meets predefined success criterion.
T
Set of successful trajectories.
Xn
Set of tasks corresponding successful trajectories.
Action set of the environment; may include primitive actions and options.
N.
Number of cluster-options (options derived from fracture clusters).
A.2 DERIVATION OF THE USEFULNESS METRIC
In this appendix, derive the usefulness (U) metric for fracture clusters. This metric is used to identify which fracture clusters have the greatest potential for reuse across different tasks. Useful- ness is function of the following three factors:
1. The probability that fracture cluster appears in any given successful task:
P(A(Ø) (w)
2. The probability that a fracture cluster is selected from the set of successful fracture clusters:
P(A(0) [.]]
3. The entropy of the fracture cluster's usage across all successful tasks:
H(A(²) T
where A(o) represents any & ē. However, for sake of notation clarity, we drop the for this derivation.
Usefulness (U) is then defined as the normalized sum of these three factors:
U =
Ple E Ta|x.] P( + H( [Tx)
The objective is to select fracture clusters that maximize the usefulness, i.c., arg max (ō)
16

---

Published as a conference paper at ICLR 2025
A.3 DERIVING P(ð Tw wwww USING BAYESIAN INFERENCE
We want model the probability that fracture cluster á appears any given successful task, Ple E Tw] For each successful task Iw from the set of successful tasks X the presence of fracture cluster Φ the corresponding trajectory Tw represented by binary random variable Cu.. where:
Sw {o
Ø E Tw (fracture cluster appears in the trajectory),
otherwise
The variable Swr is modeled as Bernoulli random variable:
Sa Bernoull(p))
where p is the probability that fracture cluster e appears in the trajectory T, of task Tw- Since we are uncertain about the true value of p. we place Beta distribution prior on pt
p Beta( (3)
where a and | are hyperparameters representing our prior belief about the likelihood of & appearing trajectory. Given total of N tasks, the likelihood for each observation Sw is: P(G.lp) pfn pp) 1-
where Sn is क appears in trajectory Tw for task nn, and otherwise.
Using Bayes' theorem, the posterior distribution of after observing data is: P(plCI......CM x
Substituting the likelihood and the Beta prior, we get:
P(plc ..... (N) II p1 p)1-Ga S.h.p-a1p pa-1(1 p))-1
This simplifies to:
P(pl......, .N) x pe G+a-1 (1-p) N --
Thus, the posterior distribution for follows a Beta distribution:
plc1 CN Beta(an,,3)) Bx)
where:
ON Σ Sn + a, 3N = Σ So B
In our experiments, we set a = and = 1, representing an uninformative prior.
A.4 DERIVING P[
The second component of the usefulness metric, P( o w is the probability that fracture cluster is selected from the of successful fracture clusters. This can be computed as the relative frequency of à in the & w of successful clusters:
count (6 P.))
Pl
where count (0, ) the number of times & appears the set of successful clusters, and [ is the total number of successful clusters.
17

---

Published as a conference paper at ICLR 2025
A.5 DERIVING H() Tw)
The entropy term H Tw) measures the unpredictability or diversity the usage of fracture cluster & across successful tasks. Entropy is defined as:
H Tw)
Σ pló Twl logN (plØ) Tw]l)
where p(0 Tw] the proportion of times that fracture cluster appears in trajectory Tw:
count ( Tw)
p Tw)
Two
Here, |Te represents the length of the trajectory Tio and N3 is the total number of fracture clusters considered. The choice of logarithm base, Ng. reflects the fact that normalize entropy relative to the number of fracture clusters.
A.6 EXPECTED USEFULNESS
Having derived the empirical estimations of the three components of usefulness now combine these elements to calculate the expected usefulness each fracture cluster. The expected usefulness incorporates the posterior distribution from Bayesian inference for Plg E Tw| as well as the empirical counts for the other components.
Thus, the expected usefulness for each fracture cluster is calculated as:
ΣN Cn +a
count(0, Fwr)
count (0, Tw))
count Tw))
E[U(A)] = N
Σ
logN
[
Twr
Two
Tu ET
where and A are the parameters of the beta distribution, which set to and 3 = 1 in our experiments.
By calculating this expected usefulness, we can rank the fracture clusters according their potential for reuse in future tasks. The ranking helps focus on fracture clusters that are more likely to appear in successful outcomes and contribute to the agent's performance across diverse scenarios.
A.7 ENVIRONMENTS
This section provides details the environments used in our experiments, including standard grid- world domains (Four Rooms, Grid, Ramesh Maze), MetaGrid, and the Proegen suite. Each environ- ment has been designed to evaluate different aspects of the agent's behaviour, such navigation, exploration, and task performance.
A.7.1 WERD-WORLD ENVIRONMENTS
We use three standard grid environments: Four Rooms, Grid, and Ramesh Maze. Figure illustrates these environments.
In these grid-world environments, the action space is discrete, with four possible (primitive) actions:
{0, 1,
These actions correspond to moving Up, Down, Left, and Right, respectively. The agent's observa- tion space is 7x7 grid centered on itself, meaning it only observes portion of the environment at any given time. This localized view allows the agent learn how to navigate based on nearby features. This design is similar MiniGrid (Chevalier-Boisse Boisvert al. 2024 but in our case, the observations are ego-centric, always centered on the agent.
In tabular learning, the agent this 7x7 observation space predict initiation states, though the f-uunctoon still based on absolute coordinates. We not employ state-tree transitinn graphs in any of our experiments.
18

---

Published as a conference paper at ICLR 2025
(a) Four rooms
(b) Grid
(c) Ramesh maze
Figure 9: Examples of the Four Rooms (a), Grid (Nine Rooms) (b), and Ramesh Maze (c) environ- ments. In each, black represents walls, blue represents the agent, yellow represents the goal, and white represents empty space. The agent upon reaching the goal. The goal's location in these figures an example one many possible positions. These versions of Four Rooms, Grid and Ramesh Maze part of the MetaGrid suite and thus have a 7x7 observation space centred on the agent.
move up, down, left, and right, receiving reward
e)
10
10
15
Figure 10: a) c) demonstrate the building blocks which MetaGrid domains can be created from. f) and g) demonstrate 21x21 configurations using these building blocks.
A.7.2 METAGRID ENVIRONMENT
The MetaGrid environment extends the standard grid-world setup by allowing for procedurally gen- erated maps of varying sizes. MetaGrid introduces randomness in the layout of walls and goal locations, ensuring that the agent encounters diverse set of environments during training and eval- uation. Figure deemonstrates the building blocks which all environments in MetaGrid are formed from, and Figure shows examples of MetaGrid environments in two different sizes: 14x14 and 21x21.
The action space in MetaGrid is identical to the one used in the standard grid worlds, with four discrete actions: Up. Down, Left, and Right. Similarly, the agent observes a 7x7 grid centered on itself, allowing it make decisions based local information. The procedural generation in MetaGrid provides varied environments for the agent to adapt to, making a more challenging and dynamic environment compared to static grid worlds.
19

---

Published as a conference paper at ICLR 2025
(a)
(b)
(c)
(d)
Figure 11: Examples of randomly generated MetaGrid environments. The blue square represents the agent, yellow represents the reward, white represents empty space, and black represents walls. Subfigures (a) and (b) show 14x grids, while (c) and (d) show 21x21 grids.
A.7.3 PROCGEN ENVIRONMENTS
Procgen is a suite of procedurally generated environments designed to test generalisation and per- formance across diverse tasks such as navigation, exploration, combat, and puzzle-solving, Each environment provides different variation every reset, preventing the agent from memorizing specific layouts or solutions. Please see Cobbe et 2020) for the full details of these environ- ments.
Action Space Procgen environments feature discrete actions like movement (up, down, left, right) and interactions je.g.jump, shoot). Depending on the task, the action space can range from 5 to 15 actions, covering basic navigation and task specific interactions.
Observation Space Unlike grid-baud grid-based environments, Procgen uses 64x64 RGB pixel observations, providing rich visual input. The agent must interpret features such as walls, enemies, and obstacles to navigate and interact effectively.
Rewards Rewards Proegen are sparse, given completing tasks like reaching goals or defeating enemies. The agent must learn to explore efficiently and develop strategies for long-tem success.
Optional Parameters To simplify learning and reduce computational cost, we activated the follow- ing Procgen parameters:
Distribution mode "easy": Provides easier levels.
Use backgrounds "False": Backgrounds are black to avoid additional noise.
Restrict themes "True": Limits visual variation single theme, such as consistent wall styles in environments like CoinRun.
Overall, these environments--ranging from grid-world environments with discrete action spaces and ego-centric observations the more complex Procgen environments with pixel-based observa- cons--offer diverse set of challenges for our agents. The combination of procedurally generated environments in MetaGrid and Procgen ensures that the agents are tested on both fixed and highly variable environments, making them suitable for evaluating the robustness of the learning algorithms used our experiments.
A.7.4 UMAP VISUALISATIONS OF OTHER ENVIRONMENTS
The structure observed in the latent projections of clustered Fracos seen in Section 4.1 is also demonstrated in trained agents other simple environments: Figure 12 visualises fracture structures in Grid (Nine rooms), CartPole, and LunarLander. The Grid environment is shown in Figure 9 CartPole and LunarLander are standard environments from the Farama Foundation Gymnasium suite Towers al. (2023).
20

---

Published as a conference paper at ICLR 2025
Fracture Visualisation in 2D latent Space
CartPole
Grid
LunarLander
-10
Figure 12: Two dimensional visualisations of the fractures formed for trained agents in CartPole, Grid (Nine Rooms) and LunarLander.
A.8 FRACOS CIAIN LENGTIIS AND DEPTH LIMITS
In our FraCOs method, the process of matching clusters involves conducting discrete search potential action permutations. This process requires forming all possible permutations of actions and passing them through the saved clusterer to determine which permutation is currently viable for the meta-policy to choose. The complexity of this search is captured by the permutation formula:
N
PN. (N. - B)!
where B is the length of the action chain, and N₂ is the total number of cluster-options. As result, the time complexity for performing this search grows factorially, O (N B)1 as more cluster-
N-
options introduced when the chain length is increased. This rapidly becomes computationally expensive as these numbers increase.
A.8.1 CHAIN LENGTH AND DEPTH IN TABULAR EXPERIMENTS
Due to this factorial growth, crucial to limit the number of cluster-options and the chain length in experiments that involve discrete cluster search (Experiments 1, 2 and 3.1). ). For all experiments using cluster search, we chose chain length of keep the computational complexity manageable. Additionally, restricted the depth of the FraCOs hierarchy to 3 or 4 levels, depending the complexity of the environment.
This limitation depth and chain length helps maintain balance between the richness of the learned options and the feasibility performing the cluster search in reasonable amount of time. The factorial growth of the search process becomes prohibitive more cluster-options deeper chains are introduced, making this constraint necessary for efficient execution of our experiments.
A.8.2 WHY THIS NOT A PROBLEM FOR NEURAL NETWORK CLUSTER PREDICTIONS
In contrast, when we extend the FraCOs initiation and action estimation be used with neural networks, the limitation of conducting computationally expensive permutation searches is alleviated. Neural networks can learn initiation sets and make predictions in continuous manner, bypassing the need for discrete cluster searches. This removes the necessity of factorially growing search complexity, allowing for more flexibility chain lengths and depth.
However, the challenge in neural network experiments lies in the sheer number of timesteps required for training and evaluation. For example, in our deep experiments (at a depth of two), training involved many millions of timesteps across nine different environments, totaling 180 million steps for pre-training, repeated for three seeds. Testing required an additional 120 million steps, also repeated for three seeds. This was only for FraCOs. When include experiments using OC (Option Critic), PPO, and PPO25, the total number of timesteps across experiments becomes 3.06 billion.
To manage these computational demands, we used a chain length of 3. This allowed us to con- duct only two warm-up phases, while ensuring that options could still be executed for reasonable
21

---

Published as a conference paper at ICLR 2025
duration (a maximum of nine steps, i... 3x3). This setup enabled complete the necessary pre-training and testing without sacrificing the quality of the learned options while maintaining computational feasibility.
A.9 FRACOS EXPERIMENT PARAMETERS FOR TABULAR METHODS
Tabular Q-Learning is reinforcement learning algorithm where the agent learns the optimal action- value function (s,, a), which estimates the expected cumulative reward for taking action in state and following the optimal policy thereafter. The agent interacts with the environment, updates the Q-values for each state-action pair based on rewards, and converges to the optimal policy over time Sutton & Barto (2018). We implement vectorized learning method.
The key hyperparameters used in all Tabular Q-Learning experiments are listed in Table 2
Table 2: Hyperparameters for Tabular Q-Learnin Experiment
Hyperparameter
Value
Description
eps
0.1
Exploration rate for e-greedy policy. Determines the probability of taking random action instead of the tion with the highest Q-value.
alpha
0.1
Learning rate for Q-value updates. Controls how much the Q-value is updated in each iteration.
gamma
0.99
Discount factor for future rewards.
num_steps
64
Number of steps per episode before environment reset. Maximum timesteps allowed in an episode.
max.ep_length
1000
anneal.lr
True
Whether to anneal the learning rate as training pro- gresses.
batch-size
64
Number of sate-action-reward tuples processed in batch.
Number of Envs 64
Number of vectorized environments
In Tabular Q-Learning, the agent repeatedly updates its Q-values for each state-action pair, gradually converging to the optimal policy. By balancing exploration and exploitation, adjusting the learning rate, and prioritizing long-term rewards, the agent learns to optimize its decision decision-making in the given environment.
FraCOs (Fracture Cluster Options) for Tabular Methods: For reproducibility, the key hyperpa- rameters used in the clustering process and other implementation details are outlined below.
In tabular experiments, we use HDBSCAN as the clustering method (Campello al. 2013). The clustering hyperparameters are:
Table 3: Clustering Hyperparameters for FraCOs
Hyperparameter
Value
Chain length (b) Minimum cluster size
2
15
Metric
Euclidean
Minimum samples
Generate minimum spanning tree
True
The following table lists the minimum success rewards required for each environment:
FraCOs-Specifie Hyperparameters:
The generalisation strength represents the threshold fracture cluster must pass to be considered for initiation. The threshold is defined as HDBSCAN, predict st sength > generalisation strength.
The FraCOs bias factor determines how much initial Q-values should be scaled to encourage transfer, similar optimistic initial values. For FraCOs bias depth annealing, the bias depth increases with
22

---

Published as a conference paper at ICLR 2025
Table 4: Minimum Success Reward per Environment
Environment
Minimum Success Reward
Four Rooms
0.97
Grid
0.60
Ramesh Maze
0.70
MetaGrid 14x14 14
0.95
Table 5: FraCOs Hyperparameters
Hyperparameter
Value
Generalisation strength
0.01
FraCOs bias factor FraCOs bias depth anneal
100
True
deeper FraCOs. For example, with bias factor of 100 and depth of 3, the first bias is scaled by the cube root of 100, the second by the square root, and the final by 100.
A.10 FRACOS MODIFICATIONS FOR DEEP LEARNING
In our experiments with deep learning, particularly in environments with large state spaces such the 64x64x3-dimension. Proegen environments, found that HDBSCAN failed to accurately capture meaningful clusters or predict clusters effectively. Initially, we attempted to integrate a Variational Autoencoder (VAE) to its latent space representation in the fracture formation pro- cess Kingma & Welling 2013). However, clustering methods still struggled to deliver satisfactory results.
Consequently, we adopted a simpler clustering approach and shifted to using neural networks to predict both the initiation states and the policy.
Simpler Clustering. To simplify the clustering process, we based clusters solely on sequences of actions. For instance, with a chain length of three, any fracture formed by the action sequence "up". "up", "right" was clustered together, independent of the state. While this approach overlooks some intricacies captured by state-based fracture formations, was necessary handle the increased complexity of environments like Procgen.
Cluster Selection. The cluster selection process remained unchanged. We continued use the usefulness metric, as defined in Equation 3 to select clusters.
Initiation Prediction. Instead of relying on clustering methods to predict initiation states, we trained a neural network to predict the states corresponding to each fracture cluster. This process involved two steps:
First, we trained Generative Adversarial Network (GAN) to augment the states in each fracture cluster (since neural networks typically require large datasets).
2. Using both the real and generated states, we trained a neural network as a classifier to predict which fracture cluster given state belonged to. One neural network was trained to predict all initiations at each hierarchical level. This method significantly improved efficiency, redueing the need for permutation-basee discrete searches single forward pass.
Policy Prediction. For policy prediction, utilized shortcut. Since all FraCOs are derived from trajectories generated by pre-trained agent, the policy of this trained agent already serves as an approximation for the FraCO policy. We saved the agent's policy at the end of trajectory generation and used as the policy for all FraCOs. This reuse of the agent's 's policy minimized additional computation without sacrificing accuracy.
Termination Condition. The termination condition for each FraCO was determined solely by the chain length, maintaining fixed execution limit per option.
23

---

Published as a conference paper at ICLR 2025
A.11 EXPERIMENT PARAMETERS FOR DEEP METHODS
All deep methods in our experiments were based on CleanRL's implementation of PPO for the Procgen environments, and we used the same hyperparameters from this implementation. This ensured that the PPO baseline was syperparameter-tuneee providing a strong and well-optimized baseline for comparison. However, FraCOs and OC-PPO were not specifically hyperparameter- tuned for these environments. Despite this, we consider it reasonable to assume that FraCOs and OC- PPO would perform near their best, as they share similar PPO update mechanisms and underlying structures with the baseline PPO.
While there exists an implementation OC-PPO by Klissarov et al. (017). we opted to implement version to maintain consistency with CleanRL's PPO implementation. This approach was necessary to ensure that any observed differences in performance were due to algorithmic design rather than implementation differences.
We chose OC-PPO over the standard Option Critic for two primary reasons. First, in our experiments with the standard Option Critic, we observed that the options collapsed quickly, converging to the same behaviour. By integrating the entropy bonus provided by the PPO update, we were able to alleviate this collapse and maintain more diverse option behaviours. Second, PPO has demonstrated significantly better performance than Advantage Actor Critic (A2C) (Mnih) 2016) in the Procgen environments. Given that the original Option Critic framework is based on A2C, using it as baseline would have led to an unfair comparison, as A2C has been shown to be less effective in these environments. Therefore, incorporating PPO in both FraCOs and OC-PPO allowed for a fairer and more balanced comparison.
We outline the hyperparameters which we used the implementation below. All code will be provided from the authors github upon publication.
A.11.1 PPO
We use CleanRL's Procgen implementation our baseline Huang et al. (2022). The only adaption we make is that we implement entropy annealing, we also have some wrappers which mean Procgen can be used with the Gymnasium API. We have another wrapper which is used for handling multi- level FraCOs in vectorized environments. These wrappers are also applied to the baseline PPO.
Hyperparameters Table provide hyperparameters for PPO.
Parameter
Value
Explanation
easy
activates "easy" Procgen setting
gamma
0.999
The discount factor.
vf_coef
0.5
Coefficient for the value function loss.
ent_coef
0.01
Entropy coefficient.
norm adv
true
Whether to normalize advantages.
num-envs
64
Number of parallel environments.
anneal_Ir
true
Whether to linearly anneal the learning rate.
clip_coee
0.1
Clipping coefficient for the policy objective in PPO.
num steps
256
Number of decisions per environment per update.
anneal.ent
false
Whether to anneal the entropy coefficient over time,
clip_vloss
true
Whether to clip the value loss in PPO.
gae lambla
0.95
The lambda parameter for Generalized Advantage Estimation (GAE).
proc_start
Indicates the starting level for Procgen environments.
learning_rate
0.0005
The learning rate for the PPO optimizer.
grad norm
0.5
Maximum norm for gradient clipping.
poddate_epochi
2
Number of epochs per update.
num. minibatches
Number of minibatches.
max clusters per clusterer
25
The maximum number FraCOs per level.
Table 6: Selected parameters for the FraCOs implementation with PPO
24

---

Published as a conference paper at ICLR 2025
A.11.2 FRACOS
Neural Network architectures.
Input: с X h X w (image observation).
Convolutional Layer 1: 16 filters, kernel size x 3, stride 1, padding 1, followed by ReLU activation.
Convolutional Layer 2: 32 filters, kernel size 3, stride padding 1, followed by ReLU
activation.
Convolutional Layer 3: 32 filters, kernel size x 3, stride padding 1, followed by ReLU activation.
Flatten Layer: Converts the output of the last convolutional layer into ID tensor.
Fully Connected Layer: 256 units, followed by ReLU activation.
Actor Head: A linear layer with 256 input units and total action dims output units, initialized with standard deviation of 0.01.
Critic Head: A linear layer with 256 input units and output unit (for the value function), initialized with a standard deviation of
The model consists of two heads:
Actor Head: Outputs probability distribution over the action space for the agent to select actions.
Critic Head:
Outputs the value function, which estimates the expected return for the cur-
rent state.
The network uses ReLU activations after each convolutional and fully connected layer, and the actor and critic heads share the same convolutional layers but have distinct fully connected output layers. In the FraCOs-SSR implementation, the convolutional layers are not reset after the warm-up phase.
Shared State Representation (SSR) details.
In the above architecture, both the actor and critic heads share a common set of convolutional layers. These shared layers process the image observations from the environment and extract useful spatial features that are fed into both the actor and critic branches. The use of shared convolutional layers allows the model to leverage the same learned feature representations for both policy and value estimation, promoting efficiency and consistency in learning.
In the FraCOs-SSR implementation, the shared convolutional layers are trained during the initial warm-up phase, but they are not reset afterward. This allows the network to retain its learned feature representations across multiple tasks and reuse them for both policy and value estimation during subsequent training phases. By freezing these convolutional layers after the warm-up phase, the network preserves its ability to generalize, while the distinct fully connected layers the actor and critic heads continue to adapt to new tasks.
Hyperparameters
Table 7 provides the full list of FraCOs hyperparameters in experimentation.
25

---

Published as a conference paper at ICLR 2025
Paramefer
Value
Explanation
easy
activates "easy" Procgen setting
gamma
0.999
The discount factor.
vf coef
0.5
Coefficient for the value function loss.
coef
0.01
Entropy coefficient.
norm_adv
true
Whether to normalize advantages.
num_envs
64
Number of parallel environments.
anneal Ir
true
Whether to linearly anneal the learning rate.
clip-coef
0.1
Clipping coefficient for the policy objective in PPO.
num_steps
128
Number of decisions per environment per update.
anneal ent
true
Whether to anneal the entropy coefficient over time.
clip_ clip_vloss
true
Whether to clip the value loss in PPO
gae lambda
0.95
The lambda parameter for Generalized Advantage Estimation
(GAE).
proc start
Indicates the starting level for Proegen environments.
learning rate
0.0005
The learning rate for the PPO optimizer.
maxx-gra_noom:
0.5
Maximum norm for gradient clipping
aydate_epochis
2
Number of epochs per update.
num_minibatches
Number of minibatches
max.clluteers.per.cuustteerr
25
The maximum number FraCOs per level
Table 7: Selected parameters for the FraCOs implementation with PPO
A.11.3 OC-PPO
Architectures
Input:
x h w (image observation).
Convolutional Layer 32 filters, kernel size 3x3, stride 2, followed by ReLU activation.
Convolutional Layer 64 filters, kernel size 3x3, stride 2, followed by ReLU activation.
Convolutional Layer 64 filters, kernel size 3x3, stride 2, followed by ReLU activation.
Flatten Layer: Converts the output of the last convolutional layer into ID tensor.
Fully Connected Layer: 512 units, followed by ReLU activation.
The model consists several heads:
Option Selection Head: A linear layer with 512 input units and num cottoontooosss output units (for selecting options), initialized with orthogonal weight initialization and a bias of 0.0.
Intra-Option Action Head: linear layer with 512 input units and num-action output units (for selecting actions within an option), initialized with orthogonal weight initializa- tion and a bias of 0.0.
Critic Head: A linear layer with 512 input units and output unit (for the value function), initialized with orthogonal weight initialization and a standard deviation of 1.
Termination Head: A linear layer with 512 input units and output unit (for predicting termination probabilities), followed by a sigmoid activation to output probability between 0 and 1.
The architecture designed share a common state representation across different heads (option selection, intra-option action selection, value estimation, and option termination). Each head uses the shared state representation for their specific outputs:
Option Selection Head: Outputs a probability distribution over available options.
Intra-Option Action Head: Outputs probability distribution over the primitive actions available within the current option.
26

---

Published as a conference paper at ICLR 2025
Critic Head:
Outputs the value function, estimating the expected return for the current
state.
Termination Head: Outputs termination probability for each option, determining whether the agent should terminate the option at the current state.
Hyperparameters. Table provide the hyperparameters used in OC-PPO.
Parameter
Value
Explanation
casy
Activates the "easy" Procgen setting.
gamma
0.999
The discount factor for future rewards.
vf vfef
0.5
Coefficient for the value function loss in PPO.
norm
true
Whether to normalize advantages before policy update.
num_envs
32
Number of parallel environments for training.
anneal_Ir
true
Linearly anneals the learning rate throughout training.
clip.coef
0.1
Clipping coefficient for the PPO policy objective.
num_steps
256
Number of steps per environment before update is performed.
anneal ent
true
Whether to anneal the entropy coefficient over time.
clip vloss
false
Whether to clip the value loss in PPO updates.
gae_lamdda
0.95
Lambda for Generalized Advantage Estimation (GAE).
proc_start
Indicates the starting level for the Procgen environment.
num_options
25
Number of options learned by the agent.
ent_coeffaacton
0.01
Coefficient for the entropy of the action policy.
ent_coef_option
0.01
Coefficient for the entropy of the option policy.
learning rate
0.0005
Learning rate for the PPO optimizer.
max www.ggrdd_oooe
0.1
Maximum norm for gradient clipping.
update_epochs
2
Number of epochs per PPO update.
num minibatches
Number minibatches per PPO update.
Table 8: Selected hyperparameters for OC-PPO implementation
A.12 OC-PPO UPDATE MECHANISM
The Option-Critic with PPO (OC-PPO) extends the standard Proximal Policy Optimization (PPO) algorithm by incorporating hierarchical options through the Option-Critic (OC) framework. The following key components distinguish OC-PPO from the standalone PPO and OC implementations:
Separate Action and Option Policy Updates: OC-PPO, two sets policy updates are performed: one for the action policy within an option and one for the option selection policy. Both policies are optimized using the clipped PPO objective, but they operate at different levels of the hierarchy:
Action Policy: The action policy selects the primitive actions based on the current option. For each option, the log-pro log-probabilities of actions are calculated, and the advantage function is used to update the action policy.
Option Policy: The option policy determines which option should be selected at each state. This option selection also updated using the PPO objective, with its own probabbiisss and advantage terms.
Both the action and option policies are clipped to prevent overly large updates, following the stan- dard PPO procedure:
Losspalicy = max
(((T):
Tnew
A(x))
clip
Tnew
1 11,,,,1 (l-,,11e))
Told
Told
Here, Teow and wold represent the new and old policies for both actions and options, and A(T) is the advantage function. This clipping is applied separately for both action and option updates, providing stability in training.
27

---

Published as a conference paper at ICLR 2025
2. Shared State Representation for Action and Option Policies: The OC-PPO architecture shares the state representation between the action and option policies but maintains distinct linear layers for each policy. The state representation is learned via shared convolutional network. This shared representation ensures that both action and option policies are informed by the same state encoding, allowing for consistent hierarchical decision-making.
Action Policy Head: Receives the state representation and outputs the action logits for the current option.
Option Policy Head: Receives the state representation and outputs the option logits for option selection.
3. Termination Loss for Options: A unique component of OC-PPO the termination loss, which encourages the agent to decide when to terminate an option and select a new one. The termina- tion function outputs the probability that the current option should terminate. This probability is combined with the advantage function to compute the termination loss:
L SeSeenmiiato
cetemminationprrbbbbli (return value)]
The termination loss is minimized when the agent terminates the option appropriately, i.e., when the return associated with continuing the current option is less than the estimated value of switching to new option. The termination probability is computed by separate network head from the shared state representation.
4. Hierarchical Advantage Calculation: OC-PPO calculates separate advantage terms for actions and options:
Action Advantage: Based on the immediate rewards from the environment while following the current option's policy.
Option Advantage: Based on the value of switching to a new option versus continuing with the current option.
Each advantage is normalized independently, and separate PPO updates are applied to both the action and option policies based their respective advantage functions.
Regularization via Entropy for Both Action and Option Policies: As in standard PPO, en- tropy regularization is applied to encourage exploration. However, in OC-PPO, this regularization applied both at the action level (to encourage diverse action selection within an option) and at the option level (to encourage exploration of different options). The overall loss function includes separate entropy terms for actions and options:
LoSSentropy = Olaction H (Tation)
+ Otoption H (Topton)
6. Clipping for Value Function: Like in PPO, OC-PPO also employs clipping for the value function updates to prevent large changes the value estimate between consecutive updates. This applies to the shared value function, which evaluates the expected returns from both primitive actions and options.
LosSvalue 0.5 max (VView - R2), (Vola + clip(Vnew - Vold, -e, e))
A.13 HYPERPARAMETER SWEEPS
In this appendix provide evidence of hyperparameter sweeps for both OC-PPO and HOC. We use the hyperparameters found for OC-PPÓ in our final results for Procgen. We decided not to use HOC after having difficulty with option collapse.
We conduct all hyperparameter sweeps for 2 million time-steps on a the Proegen environments of Fruitbot, Starpilot and Bigfish. OC-PPO had 105 total experiments and HOC had 170.
28

---

Published as a conference paper at ICLR 2025
A.13.1 OC-PPO
In Figure 13 we visualise all experiments conducted over all seeds for learning rates [le-3, le-4, le-5] and entropy coefficients of [le-1, le-2, le-3]. In Table 2 we state the averaged results. We decided on 0.01 for the learning rate and 0.001 for the entropy coefficient.
Learning Rate
Entropy Coeff
30
5e-4
5e-2
4e-2
3e-4
3e-2
2e-2
20-
1e.4
5e-5
5e-3
4e-5
40:3
3e-3
20-5
2e:3
1e:5
Figure 13: Parallel coordinates plot demonstrating effects of hyperparameters, This is for OC-PPO
Learning Rate
Entropy Coefficient
Final Returns
0.00001
0.83
0.001
0.79
0.1
0.86
0.0001
3.2
0.001
5.67
0.01
1.92
0.1
0.83
0.001
12.67
0.001
15.83
0.01
12.22
0.1
11.75
Table 9: Results for different learning rates and entropy coefficients for OC-PPO.
29

---

Published as a conference paper at ICLR 2025
A.14
FULL PROCGEN RESULTS
Learning curves across all tested Procgen environments and shown in Figure 14
Normalized IQM Episodic Retums
Dodgeball
Leaper
Frui:bot
1.0
1.0
0.8
Normalizz lized Return
0.8
Normalized lizdd Return
0.8
Normalized Return
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0
Global step
1e6
Global step
1e6
Global step
1e6
Ciimber
Bigflsh
Plunder
1.0
1.0
Normalized Return
0.8
Normalized Return
0.8
Normalized Return
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
Global step
1e6
Global step
1e6
Global step
1e6
Ninja
Coinrun
1.0
Normalizeed Return
0.8
Mormalized Return
0.8
OC PPO COD
0.6
0.6
PPG DOD
PPO COD
FračOs DOD
0.2
FraCOs 5SR 00D
0.0
0.0
2
Global step
1e6
Global step
Figure 14: Learning curves for all methods on all Procgen environments
A.15
SUCCESS CRITERIA HYPERPARAMETERS
Table states all success criteria used in our work.
30

---

Published as a conference paper at ICLR 2025
Environment
Minimum Success Refurns
Four Rooms
0.97
Nine Rooms
0.60
Ramesh Maze
0.70
MetaGrid 14x14
0.95
BigFish
5
Climber
7
CoinRun
7.5
Dodgeball
5
FruitBot
7.5
Leaper
7
Ninja
7.5
Plunder
10
Table 10: Minimum Success Returns for Various Environments
A.16 CLUSTERING ANALYSIS
We compared different clustering methods and hyperparameters to ensure we were using a sensible combination. We desired our clustering method have prediction functionality for new data and didn't want to specify the number of clusters. The only sensible clustering method which remained was HDBSCAN. Regardless we analysed others ensure HDBSCAN was not significantly worse.
We first generated trajectories in the Nine Rooms environment, created embeddings using UMAP and then conducted clustering with various methods and parameters. Figures 15| proovvide visual- isations of the results. We decided on HDBSCAN with minimum size of 15 and Fucledian distance metries.
31

---

Published as a conference paper at ICLR 2025
HDBSCAN Clustering
Stams, Mutricmuu idean
Suapm Matki Plstin
SizemS, Nentic-raaccuttss
Sizsm15,
Sizem15, Metricemanhallan
Size=15, baaycibaa
Sieem25, Metricmeac idean
Si Metricmaenhallnn
Si Muntimbraaee
Figure 15: HDBSCAN clustering comparison
32

---

Published as a conference paper at ICLR 2025
DBSCAN Clustering Results
Eps
WnntmeleesS: Eps-1.0
MinSapgles-5
Figure 16: DBSCAN clustering comparison
Mean Shift Clustering Results
Bandwidh-e.00
Bundkldhhe3
Figure 17: Mean Shift clustering comparison
33

---

Published as a conference paper at ICLR 2025
OPTICS Clustering Results
MinSamples=
MinSamples-10 10
Figure 18: Optics clustering comparison
KMcans Clustering Results
Custers-10
Oustas=50
Clrterse100
Figure 19: Kmeans clustering comparison
Gaussian Mixture Model Clustering Results
Components=10 10
Components::0 50
Components-100
Figure 20: GMM clustering comparison
34

---

A.17 USEFULNESS WEIGHTING QUALITATIVE ANALYSIS
Published as conference paper at ICLR 2025
The usefulness equation assumes equal weighting of appearance probability, relative frequency, and estimated entropy. Here, qualitatively demonstrate the effects of altering these weights. We rewrite the equation below and, in Tables and [2] conduct ablation studies and visualize the top eight selected Fracture Clusters in the nine rooms environment. For this experiment, set the success threshold to 0.97 and use chain length of four.
The usefulness equation assumes equal weighting of appearance probability, relative frequency, and estimated entropy. Here, we qualitatively demonstrate the effects of altering these weights. We rewrite the equation below and, in Tables [1 and [2] conduct ablation studies and visualize the top eight selected Fracture Clusters in the nine rooms environment. For this experiment, we set the success threshold to 0.97 and use a chain length of four.
The tables demonstrate that each metric selects reasonable fracture clusters. Comparing the top and bottom eight fracture clusters of each metric, we intuitively observe that the top-ranked clusters are more likely to be useful for future tasks. However, the rankings exhibit distinct differences depending on the ablations. For instance, when = 0, = was deemed successful. This is evident in the first-ranked fracture cluster for this configuration. With B 1, the top-ranked cluster represents trajectories that frequently move away from the initial state common occurrence. Conversely, with 1, the selected cluster traverses a bottleneck, reflecting the prioritization of appearance in the successful trajectory. This outcome aligns with intuition since most failed trajectories also originate from the initial state, causing such clusters to rank lower when 1.
O, the ranking prioritizes the relative frequeney of selecting fracture cluster, regardless of whether the corresponding trajectory
The effects of the entropy term (C) are harder to interpret intuitively without detailed understanding of the full training set.
35
While tuning A, B, and C Equation 10 could offer benefits during the research phase, we adopted the equal-weight assumption. This approach simplifies the model's 's hyper-parameters and provides a baseline for future work. Adopting equal weights is not uncommon; for instance, EEsenbach et al. (2019) use equal weighting their skill discovery objective.
A - t
count(o, Pw))
count (Φ. Tw)
[ount(() Tw)) [Tw]
B
c
Σ
logN,
(10)
N N+a+8 [D
Tor
You ET
(1) Appearance Probability
(2) Relative Froquency
(3) Entropy

---

Table 11: Ranking of fracture clusters based different weight configurations. Each cell contains an image visualisation of that fracture cluster.
Published as conference paper at ICLR 2025
Rank
A=1, B=1, C=1
A=1, B=0, C=0
A=0, B=1, C=0
A=0, B=0, C=1
Label: 17
Label: 45
Label: 17
Label: 17
Label: 63
Label: 21
Label: 63
Label: 63
2
Continued page...

---

Rank
A=1, B=1, C=1
A=1, B=0, C=0
A=0, B=1, C=0
A=0, B=0, C=1
Published as conference paper at ICLR 2025
Label: 16
Label: 59
Label: 116
Label: 42
Label: 42
Label: 73
Label: 42
Label: 16
Continued on next page...

---

Rank
A=1, B=1, C=1
A=1, B=0, C=0
A=0, B=l, C=0
A=0, B=0, C=1
Published as conference paper at ICLR 2025
Label: 116
Label: 15
Label: 16
Label: 133
5
Label: 29
Label: 57
Label: 29
Label: 116
Continued on next page...

---

Rank
A=1, B=1, C=1
A=1, B=0, C=0
A=0, B=1, C=0
A=0, B=0, C=1
Published as conference paper at ICLR 2025
Label: 35
Label: 22
Label: 58
Label: 35
Label: 133
Label: 53
Label: 133
Label: 29

---

Table 12: Ranking of fracture clusters based on different weight configurations. Each cell contains an image visualisation of that fracture cluster. Here negative indicates worst rank. -1 is the lowest rank cluster for instance.
Published as conference paper at ICLR 2025
Rank
A=1, B=1, C=1
A=1, B=0, C=0
A=0, B=1, C=0
A=0, B=0, C=1
Label:
Label: 13
Label: 109
Label:
OF
Label:
Label: 98
Label: 85
Label:
-2
Continued on next page...

---

Rank
A=1, B=1, C=1
A=1, B=0, C=0
A=0, B=1, C=0
A=0, B=0, C=1
Published as conference paper at ICLR 2025
Label:
Label:
Label: 31
Label: 87
-3
Label: 87
Label: 87
Label: 120
Label: 136
Continued on next page...

---

Rank
A=1, B=1, C=1
A=1, B=0, C=0
A=0, B=1, C=0
A=0, B=0, C=1
Published as conference paper at ICLR 2025
Label: 136
Label: 136
Label: 68
Label:
-5
Label: 46
Label: 93
Label: 78
Label: 76
-6
Continued on next page...

---

Rank
A=1, B=1, C=1
A=1, B=0, C=0
A=0, B=1, C=0
A=0, B=0, C=1
Published as conference paper at ICLR 2025
Label: 76
Label: 83
Label: 93
Label: 46
-7
Label:
Label: 31
Label: 83
Label:
-8


Reviews:
[{"review_id": "bcTxsxw9QQ", "reviewer": "Reviewer_11zq", "summary": "This work introduces fractures for learning bottom-up abstractions by clustering agent behavior across tasks.", "questions": "- HOC is not used as a baseline; why not? \n- FraCO is not compared against OC-PPO (or any other hierarchical approaches) in the tabular settings; why not?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Paper is generally well-written.\n* Related work appears exhaustive.\n* Limitations are adequately addressed. \n* Results are generally compelling.\n* Findings (with potential caveats; see below) are likely to be useful to the broader field.", "weaknesses": "- The only domain where appropriate baselines are compared against is ProcGen. The results are still compelling, but another domain where baselines are also evaluated would have been useful to benchmark FraCO's relative utility. Additionally, some didactic experiments in the tabular settings with more appropriate baselines would be useful. \n- Despite discussing more advanced baselines in the related work (most notably HOC, which can support multiple levels of hierarchy), OC is the only one used.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7DjMYlUJjp", "reviewer": "Reviewer_R928", "summary": "This paper focuses on the problem of reinforcement learning policies having a limited ability to generalize or adapt quickly to new environments. In order to overcome this limitation, the authors propose Fracture Cluster Options (FraCO). FraCOs leverage the history of trajectories on a set of training tasks, to create a set of options (alongside the primitive actions of the environment) for the agent to use during the testing phase where the agent is trained on the test tasks. The fracture is defined as the induced trajectory from state $s_t$, using actions $a_t, a_{t+1}, a_{t+2}, ..., a_{t+b-1}$ up to some length $b$. The motivation for doing this is given through a visualization of the latent space of the four rooms environment in which clusters of fractures show similar behaviours. In order to determine what options should be learned in this post-training phase, the authors create an Expected Usefulness metric based on the (1) Appearance Probability of a fracture in a trajectory, (2) Relative Frequency that some fracture appears in a successful trajectory, and (3) Entropy of Usage of that fracture. \n\nThe authors then demonstrate the performance of their FraCO method on 4 standard grid-world environments, Metagrid environments, and Procgen environments.", "questions": "1. In the Procgen experiments the authors disregarded the state information and only clustered based on the action sequences. Did you experiment with the addition of state information and find that it was causing issues (i.e. it was too noisy to use) ?\n2. Could FraCO be used to facilitate better exploration of the state space? Or any other goals of learning options?\n3. The use of a GAN to augment the dataset of Fracture Clusters was really interesting. Could this also be used in other regimes (i.e. a fixed computational budget regime) to learn FraCOs (or options in general) ? \n4. What kinds of skills/actions do the FraCOs learn?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Novel approach for discovering re-usable options to accelerate generalization abilities\n2. Thorough empirical evaluation of the FraCO method across several benchmarks\n3. Clear & honest evaluation of how well the FraCO method may work going forward to additional environments/settings", "weaknesses": "1. The paper is written in a manner to accelerate the generalization abilities of RL agents. However, there isn't a single mention of sample complexity in the paper. All of the plots and figures are concerned with the overall success rate of the agent. \n2. The given results across the 3 benchmarks are difficult to understand given that they are all success rate plots. In reinforcement learning, it is much better practice to use IQM plots [1] in order to compare the performance of multiple algorithms/configurations.\n3. I strongly disagree with the decision to not tune the hyperparameters of the PPO algorithm when adding the additional option methods. It is very difficult to argue that PPO + options with the same hyperparameters as PPO alone. The hyperparameters of each method should have been tuned to ensure that a fair comparison has been made. \n4. The organization of the paper could be vastly improved. It was very difficult to find relevant formulas/definitions/etc in the current version because they are mixed in with the motivating example(s). I think it would benefit the overall quality of the writing to separate these two components. \n5. The final experiments in the Procgen environments should be run for more random seeds. I would say that 5 is a minimum, 10 is ideal. \n6. The weights of the Expected Usefulness should be empirically chosen. In addition, each component should be ablated (i.e. only use component 1) in order to determine what each of the 3 components of the metric add to the framework. \n\n\n\n[1] https://arxiv.org/abs/2108.13264", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r5o43IQNNj", "reviewer": "Reviewer_Cp7c", "summary": "This paper introduces a method for learning options which can be used to generalize across tasks.\nFraCOs are learned based on patterns of a state and multiple subsequent actions observed in trajectories which reach a desired return threshold and are thus deemed successful.\nThe authors present results from multiple environments which demonstrate how the use of FraCOs improves learning across tasks which may vary in state space or reward function.", "questions": "Is there a reason that no continuous state spaces are used?\n\nWhile the components of the usefulness metric all seem reasonable to include, is there any justification (possibly experimental) for why they are weighted the same?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The problem of learning options (or other forms of sub-tasks) is an important issue in hierarchical RL.\n\nThe paper is well-written and generally easy to follow.\n\nThe motivation for the method provided in Section 4 is strong.\n\nThe experimental results effectively demonstrate the benefit of FraCOs in terms of generalizing between tasks and accelerating learning in new tasks in discrete state spaces.", "weaknesses": "There is a lack of discussion of continuous state spaces and no experiment(s) involving them.\n\nIt seems that a lot of data is required to learn the FraCOs before learning the actual policy, e.g., allowed to discover FraCOs in 50 of the 60 tasks used in Experiment 1. It would be nice to see a comparison of how the amount of FraCO pre-training affects the learning in later tasks and how that relates to, e.g., OC-PPO.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tL8yMcDa0k", "reviewer": "Reviewer_6Ht3", "summary": "The paper introduces Fracture Cluster Options (FraCOs), a multi-level hierarchical reinforcement learning method designed to improve generalization in new tasks. By identifying patterns in agent behavior and forming options based on expected usefulness, FraCOs achieves outperformance in both in-distribution and out-of-distribution settings. The approach demonstrates generalization and performance over existing deep reinforcement learning algorithms across complex procedurally generated environments.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "•\tOriginality: The concept of fractures to find recurring patterns is innovative for hierarchical reinforcement learning. This paper proposes a method to cluster and select fractures, as well as a framework to utilize fractures in hierarchical reinforcement learning.\n\n•\tQuality: The paper effectively uses descriptive language and figures to illustrate core concepts.\n\n•\tSignificance: The proposed method is shown to outperform several baselines.", "weaknesses": "1.\tThe Grid-world environments used in the experiments are limited in variety where all tasks are navigation across rooms. While Procgen environments were also tested, some details are lacking, such as the result of each environment, and the result of FraCOs in Figure 6 (also see question 5). In addition, I suggest that more baselines work on Procgen should also be compared (e.g. PPG[1]).\n\n2.\tIn Section 4.1 and Figure 2, fractures are visualized with distinct clusters in 2D. I wonder if clusters in other tasks are as clearly separated as in this example, especially since the clustering approach is unsupervised and does not consider semantics.\n\n3.\tBased on the concern in question 2, I wonder about the reliability of using UMAP and HDBSCAN for clustering fractures. In Section 4.2, you state that fracture clusters are based on behavioral similarity, and you use an unsupervised approach to identify them. However,  Section 4.1 has provided a mathematical definition of fractures. It is unclear whether the identified clusters consistently align with this definition. Do you compare it with other clustering methods? Please clarify the methods further and provide additional analysis of the clustering results.\n\n4.\tThe definition of usefulness (normalized sum) lacks explanation, with no supporting experiments or rationale for this decision. Would the proportion of each factor affect the result? Is there any ablation study to demonstrate the impact of each component?\n\n5.\tThe FraCOs results without SSR are unconvincing, and Figure 6 does not even include FraCOs without SSR. SSR assumes shared convolutional layers encode the state, which is a strong assumption. The results would be more convincing with a clearer explanation of why SSR are applied to FraCOs in this part, including assumption and an analysis of its impact on the results. Providing the complete result on Procgen would also be helpful.\n\n[1] Cobbe, Karl W., Jacob Hilton, Oleg Klimov, and John Schulman. \"Phasic policy gradient.\" In International Conference on Machine Learning, pp. 2020-2027. PMLR, 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)