Accelerating Task Generalisation with Multi-Level Skill Hierarchies

Authors: Thomas P Cannon, Özgür Şimşek
Primary area: reinforcement learning
Keywords: Reinforcement Learning, Generalisation, Hierarchical Reinforcement Learning

Abstract:
Developing reinforcement learning agents that can generalise effectively to new tasks is one of the main challenges in AI research. This paper introduces Fracture Cluster Options (FraCOs), a multi-level hierarchical reinforcement learning method designed to improve generalisation performance. FraCOs identifies patterns in agent behaviour and forms temporally-extended actions (options) based on the expected future usefulness of those patterns, enabling rapid adaptation to new tasks. In tabular settings, FraCOs demonstrates effective transfer and improves performance as the depth of the hierarchy increases. In several complex procedurally-generated environments, FraCOs consistently outperforms state-of-the-art deep reinforcement learning algorithms, achieving superior results in both in-distribution and out-of-distribution scenarios.

Decision: accept (poster)
Reviewer scores: 6, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1730225191510

thread_type:
official_review

<thread_object>
creator: reviewer_6ht3

<content>
summary:
The paper introduces Fracture Cluster Options (FraCOs), a multi-level hierarchical reinforcement learning method designed to improve generalization in new tasks. By identifying patterns in agent behavior and forming options based on expected usefulness, FraCOs achieves outperformance in both in-distribution and out-of-distribution settings. The approach demonstrates generalization and performance over existing deep reinforcement learning algorithms across complex procedurally generated environments.

strengths:
•	Originality: The concept of fractures to find recurring patterns is innovative for hierarchical reinforcement learning. This paper proposes a method to cluster and select fractures, as well as a framework to utilize fractures in hierarchical reinforcement learning.

•	Quality: The paper effectively uses descriptive language and figures to illustrate core concepts.

•	Significance: The proposed method is shown to outperform several baselines.

weaknesses:
1.	The Grid-world environments used in the experiments are limited in variety where all tasks are navigation across rooms. While Procgen environments were also tested, some details are lacking, such as the result of each environment, and the result of FraCOs in Figure 6 (also see question 5). In addition, I suggest that more baselines work on Procgen should also be compared (e.g. PPG[1]).

2.	In Section 4.1 and Figure 2, fractures are visualized with distinct clusters in 2D. I wonder if clusters in other tasks are as clearly separated as in this example, especially since the clustering approach is unsupervised and does not consider semantics.

3.	Based on the concern in question 2, I wonder about the reliability of using UMAP and HDBSCAN for clustering fractures. In Section 4.2, you state that fracture clusters are based on behavioral similarity, and you use an unsupervised approach to identify them. However,  Section 4.1 has provided a mathematical definition of fractures. It is unclear whether the identified clusters consistently align with this definition. Do you compare it with other clustering methods? Please clarify the methods further and provide additional analysis of the clustering results.

4.	The definition of usefulness (normalized sum) lacks explanation, with no supporting experiments or rationale for this decision. Would the proportion of each factor affect the result? Is there any ablation study to demonstrate the impact of each component?

5.	The FraCOs results without SSR are unconvincing, and Figure 6 does not even include FraCOs without SSR. SSR assumes shared convolutional layers encode the state, which is a strong assumption. The results would be more convincing with a clearer explanation of why SSR are applied to FraCOs in this part, including assumption and an analysis of its impact on the results. Providing the complete result on Procgen would also be helpful.

[1] Cobbe, Karl W., Jacob Hilton, Oleg Klimov, and John Schulman. "Phasic policy gradient." In International Conference on Machine Learning, pp. 2020-2027. PMLR, 2021.

questions:
See above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730687911335

thread_type:
official_review

<thread_object>
creator: reviewer_cp7c

<content>
summary:
This paper introduces a method for learning options which can be used to generalize across tasks.
FraCOs are learned based on patterns of a state and multiple subsequent actions observed in trajectories which reach a desired return threshold and are thus deemed successful.
The authors present results from multiple environments which demonstrate how the use of FraCOs improves learning across tasks which may vary in state space or reward function.

strengths:
The problem of learning options (or other forms of sub-tasks) is an important issue in hierarchical RL.

The paper is well-written and generally easy to follow.

The motivation for the method provided in Section 4 is strong.

The experimental results effectively demonstrate the benefit of FraCOs in terms of generalizing between tasks and accelerating learning in new tasks in discrete state spaces.

weaknesses:
There is a lack of discussion of continuous state spaces and no experiment(s) involving them.

It seems that a lot of data is required to learn the FraCOs before learning the actual policy, e.g., allowed to discover FraCOs in 50 of the 60 tasks used in Experiment 1. It would be nice to see a comparison of how the amount of FraCO pre-training affects the learning in later tasks and how that relates to, e.g., OC-PPO.

questions:
Is there a reason that no continuous state spaces are used?

While the components of the usefulness metric all seem reasonable to include, is there any justification (possibly experimental) for why they are weighted the same?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730694220433

thread_type:
official_review

<thread_object>
creator: reviewer_r928

<content>
summary:
This paper focuses on the problem of reinforcement learning policies having a limited ability to generalize or adapt quickly to new environments. In order to overcome this limitation, the authors propose Fracture Cluster Options (FraCO). FraCOs leverage the history of trajectories on a set of training tasks, to create a set of options (alongside the primitive actions of the environment) for the agent to use during the testing phase where the agent is trained on the test tasks. The fracture is defined as the induced trajectory from state $s_t$, using actions $a_t, a_{t+1}, a_{t+2}, ..., a_{t+b-1}$ up to some length $b$. The motivation for doing this is given through a visualization of the latent space of the four rooms environment in which clusters of fractures show similar behaviours. In order to determine what options should be learned in this post-training phase, the authors create an Expected Usefulness metric based on the (1) Appearance Probability of a fracture in a trajectory, (2) Relative Frequency that some fracture appears in a successful trajectory, and (3) Entropy of Usage of that fracture. 

The authors then demonstrate the performance of their FraCO method on 4 standard grid-world environments, Metagrid environments, and Procgen environments.

strengths:
1. Novel approach for discovering re-usable options to accelerate generalization abilities
2. Thorough empirical evaluation of the FraCO method across several benchmarks
3. Clear & honest evaluation of how well the FraCO method may work going forward to additional environments/settings

weaknesses:
1. The paper is written in a manner to accelerate the generalization abilities of RL agents. However, there isn't a single mention of sample complexity in the paper. All of the plots and figures are concerned with the overall success rate of the agent. 
2. The given results across the 3 benchmarks are difficult to understand given that they are all success rate plots. In reinforcement learning, it is much better practice to use IQM plots [1] in order to compare the performance of multiple algorithms/configurations.
3. I strongly disagree with the decision to not tune the hyperparameters of the PPO algorithm when adding the additional option methods. It is very difficult to argue that PPO + options with the same hyperparameters as PPO alone. The hyperparameters of each method should have been tuned to ensure that a fair comparison has been made. 
4. The organization of the paper could be vastly improved. It was very difficult to find relevant formulas/definitions/etc in the current version because they are mixed in with the motivating example(s). I think it would benefit the overall quality of the writing to separate these two components. 
5. The final experiments in the Procgen environments should be run for more random seeds. I would say that 5 is a minimum, 10 is ideal. 
6. The weights of the Expected Usefulness should be empirically chosen. In addition, each component should be ablated (i.e. only use component 1) in order to determine what each of the 3 components of the metric add to the framework. 



[1] https://arxiv.org/abs/2108.13264

questions:
1. In the Procgen experiments the authors disregarded the state information and only clustered based on the action sequences. Did you experiment with the addition of state information and find that it was causing issues (i.e. it was too noisy to use) ?
2. Could FraCO be used to facilitate better exploration of the state space? Or any other goals of learning options?
3. The use of a GAN to augment the dataset of Fracture Clusters was really interesting. Could this also be used in other regimes (i.e. a fixed computational budget regime) to learn FraCOs (or options in general) ? 
4. What kinds of skills/actions do the FraCOs learn?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730696478192

thread_type:
official_review

<thread_object>
creator: reviewer_11zq

<content>
summary:
This work introduces fractures for learning bottom-up abstractions by clustering agent behavior across tasks.

strengths:
* Paper is generally well-written.
* Related work appears exhaustive.
* Limitations are adequately addressed. 
* Results are generally compelling.
* Findings (with potential caveats; see below) are likely to be useful to the broader field.

weaknesses:
- The only domain where appropriate baselines are compared against is ProcGen. The results are still compelling, but another domain where baselines are also evaluated would have been useful to benchmark FraCO's relative utility. Additionally, some didactic experiments in the tabular settings with more appropriate baselines would be useful. 
- Despite discussing more advanced baselines in the related work (most notably HOC, which can support multiple levels of hierarchy), OC is the only one used.

questions:
- HOC is not used as a baseline; why not? 
- FraCO is not compared against OC-PPO (or any other hierarchical approaches) in the tabular settings; why not?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732035227106

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of changes.

comment:
We sincerely thank all reviewers for their detailed and constructive feedback. Your insights have significantly enhanced the overall quality of our work. Below, we summarise the key changes made in response to your comments:

- **Statistical reliability**: Procgen experiments have been re-run with **eight seeds** to improve robustness.
- **Fairer comparisons**: All plots now use the **Interquartile Mean (IQM)** as recommended.
- **Hyperparameter search**: A thorough search was conducted for OC-PPO, and are included in Appendix A.13.
- **Ablation studies**:
    - The **Expected Usefulness weightings** have been analysed, with results added to Appendix A.17.
    - Additional insights into clustering approaches, including HDBSCAN hyperparameter sweeps, are now provided in Appendix A.16.
- **Expanded results**:
    - Full results across **all tested Procgen environments** are now presented in the main text, with learning curves detailed in Appendix A.14.
    - Results for **PPG** have been added for completeness.
    - Full FraCOs and FraCOs-SSR results are included in the main body.
- **Improved clarity**:
    - A **glossary of terms** has been added at the start of the appendix to aid navigation.

We hope these changes address the reviewers' concerns and improve the clarity and rigour of our work. We sincerely appreciate your thoughtful feedback and the time you’ve dedicated to reviewing our submission. Your insights have been invaluable in refining this research, and we look forward to any further discussions or suggestions you may have. Thank you for considering our work.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents a method for constructing options via clustering "fractures", which are defined as a state and a finite-length sequence of actions. The authors present a method for measuring the expected usefulness of the discovered fractures and select the ones with highest expected usefulness.

The paper is clear and well-written, the method novel, and empirical evaluations demonstrate its strength.

While there were some concerns on the baselines used, in particular with the tabular experiments, I do agree with the authors that the tabular settings are mostly meant as a validation of the method, rather than as a proper empirical comparison against other methods (which were developed for deep RL environments).

There were also some concerns raised as to whether the claims of "accelerating generalization" are justified, especially given the strong title. It seems this claim is mostly evaluated by running on the ProcGen suite, rather than through a more systematic study (beyond the tabular visualizations). I tend to agree with this sentiment and would suggest the authors temper the claims of "increased generalization".

Despite these drawbacks, I do feel the paper presents an interesting and novel algorithm that may be useful for others in the community. I would strongly suggest the authors incorporate the feedback provided by the reviewers, in particular with respect to the strenghts of the claims.