Incorporating Spatial Information into Goal-Conditioned Hierarchical Reinforcement Learning via Graph Representations
Authors: Shuyuan Zhang
Zihan Wang
Xiao-Wen Chang
Doina Precup
Venue: Accepted by TMLR ()

Abstract: The integration of graphs with Goal-conditioned Hierarchical Reinforcement Learning (GCHRL) has recently gained attention, as intermediate goals (subgoals) can be effectively sampled from graphs that naturally represent the overall task structure in most RL tasks. However, existing approaches typically rely on domain-specific knowledge to construct these graphs, limiting their applicability to new tasks. Other graph-based approaches create graphs dynamically during exploration but struggle to fully utilize them, because they have problems passing the information in the graphs to newly visited states. Additionally, current GCHRL methods face challenges such as sample inefficiency and poor subgoal representation. This paper proposes a solution to these issues by developing a graph encoder-decoder to evaluate unseen states. Our proposed method, Graph-Guided sub-Goal representation Generation RL (G4RL), can be incorporated into any existing GCHRL method when operating in environments with primarily symmetric and reversible transitions to enhance performance across this class of problems. We show that the graph encoder-decoder can be effectively implemented using a network trained on the state graph generated during exploration. Empirical results indicate that leveraging high and low-level intrinsic rewards from the graph encoder-decoder significantly enhances the performance of state-of-the-art GCHRL approaches with an extra small computational cost in dense and sparse reward environments.

Published in Transactions on Machine Learning Research (09/2025)
Incorporating Spatial Information into Goal-Conditioned Hi- erarchical Reinforcement Learning via Graph Representa- tions
Shuyuan Zhang
shanpan..hanggBmuill.gggg mogil
School of Computer Science McGill University, Mila
Zihan Wang
zihan, wwwgg megill.
School of Computer Science
McGill Universily, Mila
Xiao-Wer Chang
Chang@cs.megille
School of Computer Science
McGill University
Doina Precup
dprecup@es.moglllccm
School of Computer Science
McGall University,
Google Deepmind,
CIPAR Fellow
Reviewed on OpenReview: https: // openreview. net/ net/ forum? id- a 37Bo4s5ghk
Abstract
The integration of graphs with Goal-conditioned Hierarchical Reinforcement Learning (GCHRL) has recently gained attention, as intermediate goals (subgoals) can be effectively sampled from graphs that naturally represent the overall task structure most RL tasks. However, existing approaches typically rely on domain-specific knowledge to construct these graphs, limiting their applicability
tasks. Other graph-based approaches create graphs dynamically during exploration but struggle to fully utilize them, because they have problems passing the information in the graphs to newly visited states. Additionally, current GCHRL methods face challenges such as sample inefficiency and poor subgoal representa- tion. This paper proposes solution to these issues by developing a graph encoder-decoder to evaluate unseen states. Our proposed method, Graph-Guided sub-Goal representation Generation RL (G4RL), can be incorporated into any existing GCHRL method when op- erating in environments with primarily symmetric and reversible transitions to enhance performance across this class of problems. We show that the graph encoder-decoder can be effectively implemented using network trained the state graph generated during exploration. Empirical results indicate that leveraging high and low-level intrinsic rewards from the graph encoder-decoder significantly enhances the performance of statc-of-tho-art GCHRL approaches with an extra small computational cost dense and sparse reward environments.
1 Introduction
Traditional reinforcement learning methods face great challenges when learning policies in environments with long time horizons and sparse rewards. To address these challenges, Hierarchical Reinforcement Learning (HRL) methods have been proposed to break problems into smaller, manageable subproblems conducive

---

Published in Transactions on Machine Learning Research (09/2025)
1999 Kulkarni et al. 2016 Vezhnevets et al, 2017 Levy et al. 2017) have demonstrated HRT'ss capability in handling large, reward environments. Among HRL methods, Goal-Conditioned Hierarchical Reinforcement Learning (GCHRL) has attracted much attention duc to its well-defined paradigm (Nachum ct al., 2018b) and its resemblance to the human thinking process (Zhao el al. 2022 Hsu et al. 2024). Although GCHRL has shown superior performance compared to non-hierarchical methods in some scenarios, questions such as how to learn better subgoal representations and explore the state space more efficiently (Nachum et al., 2018a; Guo et al. 2021 remain unanswered.
to more efficient learning. Previous works (Sutton et al.
GCHRL methods typically utilize two levels of agents as described in Nachum et al. 2018b). The high-level agent chooses the next target state based on its current state, while the low-level agent decides how to reach this larget. Both agents face their own challenges. The high-level agent suffers from sample inefficiency in environments with a large state-action space, a problem also encountered by non-hierarchical methods, while the low-level agent is trained solely by the reward signals derived from the distances between the current state and subgoals in the representation space, making the reward signals highly susceptible to poor subgoal representations.
Previous works have tried to either enhance the efficiency of high-level exploration (Huang et al. 2019
2022) or find a good subgoal space to boost GCHRL's performance in complex environments Zhang et al.
(Wang et al. 2024]. To the best. of our knowledge, while these methods have made progress in their respective aspects, no prior work has attempted to address the efficiency of high-level exploration and the accuracy of low-level learning signals in a unified framework. Yet, such integration is crucial, as increasing the effectiveness of the high-level agent and the accuracy of the low-level agent within a cohesive framework can yield performance improvements that surpass simple additive effects.
Recently, combining graph theory with RL has become trend in the community (Lee et al. 2022 Gieselmann & Pokorny 2021 as graphs are inherently well-suited for representing the environment and task structure,
Previous work has focused on arcas such as docision-making through graph scarch or traversal Wan et al. 2021
Shang el al.
Eysenbach el al. 2019), and as well as using graphs as world models (Zhang el al. 2019 Yet, many previous works rely on pre-crafted graphs, limiting their generalizability,
2019
2021
Huang et al.
Additionally, most existing works (Zhu et al.) 2022 Hong et al.. 2022) constructed graphs directly from the original state space, which cannot provide meaningful guidance when the agent encounters state that is not represented in those graphs.
To utilize graph representations when a new state (node) is encountered, transitioning to these representa- Khoshraftar & An 2024) through graph learning is a viable option. Recent studios have shown that employing graph representations for learning can improve underlying RT.
tions (Hamilton 2020 Chen et al. 2020
performance (Klissarov & Precup
2020
Klissarov & Machado 2023).
In this paper, we propose novel approach that simultaneously addresses all of the aforementioned problems. Specifically, we construct state graph based on visited states until the number of nodes reaches a threshold and after that we update the graph by adding and dropping nodes as new states are visited. Using this graph, we build a subgoal space through graph learning that leverages both state representation and spatial connectivity. By generating subgoal representations through graph learning, we ensure that they capture their relative positions in the docision chain, thereby forming a more effective subgoal space. In estimating the distance between the current state and the intended subgoal in the original space, we use the distance between their corresponding representations in the subgoal space. This distance is then used to calculate the intrinsic reward for curiosity-driven exploration, aiming to improve sample efficiency across both high-level and low-level agents.
The main contributions of this paper are as follows:
We propose a novel architecture that employs a graph encoder-decoder to embed spatial information into subgoal representations, enabling the evaluation of newly visited states. This architocture can be integrated into any GCHRL algorithm operating in environments with primarily symmetric and reversible transitions, enhancing performance across this class of problems.
2

---

Published in Transactions on Machine Learning Research (09/2025)
We present method for the online construction of the state graph as graph-based world model
(Ha & Schmidhuber 2018: Zhang et al. 2021 for the HRL agent by sampling from trajectories.
We use novelty-based auxiiiary rewards (Simsek & Barto 2006 subgoal representations to improve sample efficiency for both high-level and low-level agents.
Nehmzow et al. 2013) derived from
We tested our approach on several environments (Todorov et al., 2012 to assess the siguificance of our experimental results. The findings indicate that our method can significantly enhance the performance of the underlying HRL approach in terms of both sample efficiency and success rate/ aatucuuulative reward.
2
Preliminaries
2.1 Markov Decision Processes
As the most common framework for modeling reinforcement learning scenarios, the Markov Decision Process (MDP) (Puterman) 2014) introduced as tuple R, >, defined as follows: At each time step 1 the agent observes the current state 8t S provided by the enviromment and chooses an action or E A according to its internal policy T(ag)s)) which specifies the probability of choosing action a given state The action is then exccuted, and the interaction with the environment leads the agent to new state 81+1 according to a transition probability function P( 18,, at) which is known only to the enviromment. Subsequently, the agent receives a reward Tt, determined by the reward function R(s;, a,) that evaluates the action taken the current state and also only visible to the environment. The agent aims to learn an optimal policy to maximize the expected discounted cumulative reward E,, Ei= r're]. where y (with 0 7 < 1) is pre-defined discount factor used to prioritize immediate rewards over distant future rewards, thereby ensuring that the total reward remains finite.
2.2 Goal-conditioned Hierarchical RL (GCHRL)
Goal-Conditioned Reinforcement Learning (GCRL) trains agents to achieve specifie goals, which are the target states. The agent reccives additional goal input Ie along with the state input 8t and learns policy <<a, let, 8.) that aims to achieve this goal. Goals are represented explicitly in the input to the policy, guiding the agent's is actions towards desired outcomes. The reward function is often goaeddep,,,nnt, providing positive feodback when the agent successfully reaches the desired goal statc.
To deal with large and complex environments, Coal-Conditioned Hierarchical Reinforcement Learning (GCHRL) (Nachum et al.) [2018b; [Zhang et al. 2022 Wang al.) (024)) decomposes the learning task into hicrarchy of smaller, more manageable sub-taske Typically, there two levels of agents. At time step t, the high-level agent chooses subgoal 9t, representation of target state, and assigns it to the low-level agent to achieve as part of the overall task. This choice is made by sampling from the high- level policy Tx (g)((()))) where ò + Rd is the state representation function which gives condensed representation of the state.
Each state 8 can be mapped to its subgoal feature g(s:) by subgoal feature extractor. Note that g(s:) is not the same as gr- The former, (st), is the learned subgoal feature of the current state St, while the latter, 9t, is the target state aim to reach from the state s, in one step multiple steps.
Given the subgoal ge sampled from the high-level policy TA (gr (o(s:)) for the current time step t and the state representation vector d(s:), a low-level agent executes action ae based on the low-level policy mi(a, ((s)), 9t). The low-level agent is trained using the intrinsic reward signal rine (8t, It, at, 8641) = - -116(*++1) - ge 11 to encourage it to achieve the subgoal.
Both agents can be implemented by any policy-based based methods, including those introduced in previous works policy gradients such as Fujimoto et al. (2018) Haarnoja et al. (2018) and /Shhlman et al. 20117).
3

---

Published in Transactions on Machine Learning Research (09/2025)
2.3 Graph and MDP
Graph is generic data structure, which model complex relations among objects in many real-world problems. A graph is defined as G = (V,E), where V {1, .,, } is the set of nodes and & {egy is the set of edges without self-loops. The adjacency matrix of g is denoted by A = (Ai,) E RNx with As, = =1 if there is an edge between nodes and j. otherwise A.j = 0. The adjacency matrix can be extended to a weighted adjacency matrix, where A. is a weight of the edge eij.
In MDP, a node can represent state, while the edge weights can model the transition probabilities or reachability statistics between states.
3 Methods
This section presents our framework, Graph-Guided subGoal representation Generation (G4RL). Our method reshapes the subgoal space utilizing state graph to incorporate the relative spatial information of visited states.
One drawback of previous hierarchical reinforcement learning algorithms (Nachum et al. 2018b Kim et al, 2021 Zhang et al., 2022 Luo et al., 2024) that the Euclidean distance calculated in the original state representation space between the current state and the intended goal does not accurately reflect the true progress of the low-level agent, as there is rarely straight path between the current state and the subgoal in the space. As a result, the low-level agent trained with such information may receive inaccurate reward signal, thus impairing its performance, Another issue is that, without appropriate constraints, the high-level agent may propose subgoals that are too difficult to reach, wasting exploration steps on pursuing infeasible targets (Zhang et al., 2022). Our proposed method aims to mitigate both problems by calculating the distance a subgoal representation space between subgoal representations given by a graph encoder- decoder. This graph encoder-decoder captures the actual connectivity between states, ensuring that the generated subgoal representations respect adjacency information.
3.1 State graph
To record the visited states and their connections, we maintain a state graph g = EV,E) with a fixed number N of nodes This graph is built and updated during training, with no pre-training using expert data or handcrafted process involved in its construction.
Each node is labelled by the corresponding state and for each node st, the corresponding state representation vector (s:),, which is also referred as the node feature, stored. Edges in the graph represent connectivity between states. The graph is constantly updated during exploration.
We choose the state graph to be undirected for mainly two reasons: (1) Efficiency: An undirected graph requires fewer resources for storage and computation compared to directed graph, and (2) Method com- patibility: Our method relies on defining a deterministic distance between state/subgoal representations for each node pair. This is straightforward an undirected graph but becomes problematic a directed setting, where distances can asymmetric or undefined. Related details will be explained in Sections 3.2 and 3.3
This choice is based on the assumption that G4RL is designed for environments with symmetrical and re- versible dynamics. However, some of our experiments demonstrale that G4RL can still enhance performance in partially asymmetric environments; please refer to the experiments section for further details.
3.1.1 Graph construction
The graph is initialized with N empty nodes and no edges. The corresponding weighted adjacency matrix A is set to an Nx N zero matrix. We perform the GCHRL exploration process using randomly initialized policy
The number of training states for the graph encoder-decader grows quadratically with N because the adjacency weight matrix has 2 elements. The choice N depends on the machine's capabilities.
4

---

Published in Transactions on Machine Learning Research (09/2025)
Th and Ri. Once the agent encounters state representation never seen before, that is, the representation is different from any state representations stored in the graph, as described in equation 11, it stores the state
representation d(st as the node feature of an empty node in the graph and build an edge between this node and the node corresponds to the previous state:
V.LEV; (10(s)) - (((8))122 Ed.
(1)
A = A =1,
(2)
where 6d is a hyper-paramette controlling the distance threshold between state representations. When the agent encounters state &t with feature ((s;) that similar to several node representations already stored in the graph, it finds the state whose representation is the closest the current state feature:
8,
arg min
[1x(#)) - ((8u)l12-
(3)
Then the node sy is relabeled as 8t and the weight for the edge (s) is updated as follows:
A0(s)))) A Ab(s)))) +
(4)
Note that a large weight indicates more frequent transitions between the underlying states.
We have used the Euclidean norm to define the distance between feature vectors. Since some elements may contain more spatial information than others, one can use a weighted Euclidean norm define the distance between state representations instead.
3.1.2 Graph updating
The graph has a fixed number of nodes. Suppose the graph is now full. When new state is encountered, if from equation (3] exists, as before we relabel the node as and perform edge update as shown in equation 4 Otherwise, we replace the oldest state node in the graph with the current state node, delete edges previously linked to that node, and create an edge (é(s;-1), ((s))) with weight A = Alss 1) 1. Alternatively, we could replace the state node that is most weakly connected to the other nodes that is, the node with the lowest sum of edge weights.
3.2 Graph encoder-decoder
To enable the assignment of suitable subgoal representations to every possible state, including unseen ones, we use node representations and edges to train a graph encoder-decoder. The parameter updates of the graph encoder-decoder and the policies during policy training are performed alternately in each episode.
The encoder-decoder starts training after the graph is full and continues periodically after processing few trajectories. Section 3.3 will show the details of the training schedule.
The encoder E maps every state representation Φ(s) to a subgoal representation g(s). We use a feed-forward network (FFN) with several layers as the encoder E:
g(x) E(((s)) = PFN(
(5)
The weight parameters of the feed-forward network will be learned through training. The decoder D takes two subgoal representations input and outputs the inner product of these reprosentations:
D(g(su).gss)) g(sw) ( gss)).
(6)
We choose dot-product similarity based on the assumption that the similarity between two nodes, such as the overlap their local noighbourhoods, is well captured by the dot product of their enbeddings. This
assumption is supported by prior work the graph embedding literature (Ahmed al. 2013 Cao et al.
2015 Ou et al. 2016).
5

---

Published in Transactions on Machine Learning Research (09/2025)
The aim is to use the encoder-decoder structure to predict node relations. Naturally can use o as a measure of the relation between nodes 6(su) and ((8x). But for the sake of numerical stability in the training process, we use Asss.) maXg(s)))(ss) Ao(s)))((ss) as a measure. Thus the loss function is defined as:
C= Σ (((((s)) ols))) max A ((s)).
(7)
((
This loss function can enforce the subgoal representation provided by the encoder to respect neighbouring features in the graph.
Note that in each training phase of the graph encoder-decoder (except the first one), we use the values of the parameters obtained from the last training phase as the initial point, which helps save computation cost.
3.3 Adaptive training schedule of the graph encoder-decoder
The graph stores evolving data, including state representations node features and connection information in the weighted adjacency matrix A, which are continuously updated during online training. Since the graph structure and content change at varying rates across episodes, training the graph encoder-decoder at fixed intervals can cause several issues: (1) high variance in earlier episodes, where sparse or unstable graph data may lead to unreliable model updates; (2) data underutilization, where intermediate graph states are overwritten before being used for training; and (3) overfitting in later episodes, as the model repeatedly trains on increasingly redundant data. To address these issues, we introduce an adaptive training schedule for the graph encoder-decoder, described in the following paragraph.
There are two types of data changing in the graph: node replacement and edge update. We introduce variable to track the weighted number of data changes. Since the replacement of nodes has a much higher impact on the data than the edge update, we add N - to e if node replacement occurs, and add to c if an edge update happens:
c N+N-L a node replacement happens, c+1, if an edge update happens.
(8)
When this variable excceds certain value, specifically tolerance & multiplied by the total number of non- diagonal elements N2 - N in the matrix A. we perform one gradient update for the graph encoder-decoder, and then we reset c to
3.4 Hierarchical agent with graph encoder-decoder
Our proposed method involves traditional goal-conditioned settings and subgoal representation extractor implemented by graph encoder-decoder.
The high-level policy (x(o((((s)) nominates subgoal every K steps and is trained using the external environmental reward Text.
The policy can be implemented by any policy-based RL algorithm that takes transition tuples (St, Gt. at Tt, St41 9t+1) as input. To encourage it to propose a subgoal that not too difficult to reach from the current state s for more efficient exploration, we add an intrinsic term the high-level reward, considering the distance between the subgoal features of 8 and gr. in the subgoal space:
(8ts It, Se+1) = Text + Tint = Text + an D(E((ss)) E(g:)),
(9)
where aj is a hyperparameter that controls the significance of the intrinsic term in the high-level reward.
The low-level policy x (p(s))):)) 9t), however, operates in the subgoal space. While it still takes d(sz) and as input and outputs an atomic action dz, we compute the reward based on distances in the subgoal space:
(8(8 It. at, 81+1) =- -1(((+++1) - 2 on D(E()(#)+1), E(g)))
(10)
6

---

Published in Transactions on Machine Learning Research (09/2025)
where ay is hyperparametter controlling the significance of the reward term in the low-level reward. By computing the intrinsic reward in the subgoal space rather than in the state space, the function provides high values when proposed subgoals are easy to reach from the current location and low values when subgoals are close in the original state space but difficult to reach from the current location. The low-level agent can also be any policy-ban algorithm.
3.5 Balancing between speed and performance
Due to the excessive comparisons between the current state representation and the node features during graph updates, as well as the training cost of the graph encoder-decoder on large graph, our experiments show that the GCHRL method, after incorporating our method, takes approximately twice as long as the original GCHRL method.
To reduce the additional cost, we can either decrease the frequency of sampling candidates for node features, train the graph encoder-decoder with subset of all available training data, or do both.
For the sampling frequency, instead of comparing the state representation with node features in each time step, we do every to time steps. This may significantly speed up our method while maintaining satisfactory performance.
To reduce the training data in each graph encoder-decoder training cycle, we randomly sample node pairs in the graph instead of using every node pair for training.
In the experiments section, we will present the training time and performance of G4RL with the above two techniques applied.
3.6 Algorithm: GCHRL + G4RL
The detailed description of how our proposed strategy, G4RL, can be incorporated into typical GCHRL algorithms is given in Appendix D
4 Experiments
In this section, empirically evaluate the effectiveness of integrating G4RL into existing GCHRL methods. The experiment results remonst.rate substantial improvements in both convergence speed and overall success rates achieved by our proposed approach. Additionally, we provide empirical evidence that the generated state graph accurately represents the underlying structure and relationships within the task environments.
4.1 Environment settings
We used AntMaze, Anthather, AntPush, AntFall and AntMaze-Sparse environments from the GYM MuJoCo library [Todorov et al. 2012). The first four involve complex navigation and manipulation tasks performed by a simulated multi-armed robot, while AntMaze-Sparse presents a particularly challenging scenario due to sparse reward signals, providing feedback only upon reaching the goal. Note that AntPush and AntFall contain asymmetric transitions which lead to irreversible state changes that can be difficult to learn with undirected graph. We added these environments to show G4RL's robustness in inherently asymmetric environments. For the state representation, we selected a subset of raw state dimensions that contains spatial information (c.g. coordinates and arm angles) to serve as the node represcntation for the graph encoder- decoder. We deliberately aligned our choice of environments with prior work to ensure a lair and consistent comparison between the backbone algorithms and their GARL-angmented versions. This allowed us to use the same environments, hyperparameters, and codebases provided by the original studics. Our goal was demonstrate that G4RL can consistently enhance the performance of these backbone algorithms under comparable conditions.
7

---

Published in Transactions on Machine Learning Research (09/2025)
4.2 Experimental Comparisons
We incorporated G4RL in the following existing GCHRL methods:
HIRO Nachum al.)
This is the first method which describes how the Goal-conditioned information can be integrated into hierarchical agents.
2018b):
HRAC (Zhang et al.)
2022 This method enhances the performance of HIRO by training an adjacency network that produces subgoals easier to reach from the current subgoal.
HESS Li et al.
2022 This method applies a regularization term on consecutive subgoal represen- tations each update stabilize the representation across episodes.
HLPS (Wagg et al. 2024 : This method applies the Gaussian process on subgoal representations for a smoother reprosentation update.
In addition to comparing these four GCHRL-G4RL methods with their native counterparts, we also compared them with the following non-hicrarchical method:
TD3 Fujimoto et al. 2018
This is a well-known non-hicrarchical policy-based method designed for continuous action spaces and we use it to implement both high- and low-level agents.
Although reward is a key metric of an agent's learning ability, for AntMaze and AntMaze-Sparse, we compare success rates of these methods instead of their rewards on the corresponding tasks. This is because higher rewards in Aattaa/e//AaaataSSraaaa do not necessarily indicate better performance; the agent may achieve high rewards without reaching the goal.
The learning curves of baseline methods and G4RL-applied versions are plotted in Figure ! and Note that all the curves reported in Section 4 averages from 20 independent runs and they have been equally smoothed for better visualization.
AntMaze
AntMaze-Sparse
AntGather
3-
HRAC+34RL
0.8
0.8
2.5
HRAC
Suo0es Rale
Success Rate
HIRO
0.8
0.6
Roward
TDS
1.5
0.4
1-
0.2
0
0
Episodes
x10*
Episcdes
x10
Eplsodes
x10*
(a)
(b)
(c)
Figure 1: Success Rate on (a) AntMaze (b) Ant Maze-Spaase and Reward on (c) AntGather, using HIRO, HIRO-GARL, HRAC, HRAC-GARL, and TD3. Incorporating G4RL in HIRO and HRAC significantly en- hances their performance.
we observe that, in all enviromments, incorporating G4RL in the base GCHRL methods significantly enhances their performance, further improving the already strong results of these hierarchical methods compared to the non-hicrarchical method. Notably, G-RI-augmented methods not only achieve higher final success rates but also converge substantially faster, with the most significant improvements observed during the early stages of training.
From Figures [nd
To demonstrate the proposed method's effoctiveness in environments with imago- state representations, we conducted experiments on AntMaze, AntPush, and AntFall, utilizing images as states, and compared the results with HESS and HLPS, along with their G4RL variations. We use Mean Squared Error (MSF)
8

---

Published in Transactions on Machine Learning Research (09/2025)
AntMaze
AntMaze-Sparse
AntPush
0.8
0.8
0.8
Role
Succese Rele
Suecess Rata
0.6
0.6
HESS+G4RL
HESS
0.2
0.2
0.2
HLPS+GARL
HLPS
0
0
Episodes
x101
Episodes
x10"
Episodes
x10*
(a)
(b)
(c)
Figure 2: Success Rate on (a) AntMaze (b) Sapaasee and (c) AntPush, using HESS, HESS-G4RL, HLPS, HLPS-GARL. Incorporating G4RL in HESS and HLPS significantly enhances their performance.
to measure the pixel-wise differences between image states to decide whether a new node should be added to the graph. The test results, given in Figure 3 show that methods incorporating G4RL exhibit faster convergence and achieve higher performance across all tested image-based environments.
AntMaze w/ image input
AntPush w/ image input
AntFall Image Input
0.8
0.8
Success Ralt
Suocess Rale
Suo086s Rale
0.6
0.8
0.8
0.4
0.4
0.4
HESS+GARL
HESS
HLPS+G4RL
HLPS
Episodes
x101
Episodes
x10"
Episodes
X10"
(a)
(b)
(c)
Figure 3: Success Rate on (a) AntMaze (b) AntPush and (c) AntFall with image state features, using HESS, LLESS-GARL, HLPS, HLPS-GARL. Incorporating G4RL helps convergence and achieves higher performance across all tosted imago-based environments.
4.3
Ablation study
4.3.1
The effect of high/low-level intrinsic reward
We consider the following variants G4RL to show the effectiveness of adding high-level and low-level intrinsic rewards:
Low-Leel intrinsics: Apply both equation p and equation 10 to the high-level and low-level rewards respectively.
High-level intrinsic only:
Apply equation 9 to the high-level rewards and set aq 0 in equation
[0) when it is applied to the low-level rewards.
Low-level intrinsic only: Apply equation
10 to the low-level rewards and set 0 in equation
gb when it is applied to the high-level rewards.
MIIRO/RAA HESS HLPS: Vanilla baseline methods.
Same before, all the curves reported in this section are drawn from results averaged across 20 independent All curves have been equally smoothed for better visualization. Additionally, have added the
9

---

Published in Transactions on Machine Learning Research (09/2025)
un-smoothed versions of figures 4 smoothing.
and 5 in Appendix E to confirm that the conclusions unaffected by the
AntMaze
AntMaze-Sparse
AntGather
2.5
Nigh+Loww- level intrinsics
LogheLowwlevel intrinsics
High Ligh-Low-leve intrinsice
0.8
High-leval Intrinsic only
0.6
Hgh-leve Intrnnii only
2-
Hgh-levai Intrinsie only
Success Rale
Low-lovel intrinsc only
Success Rate
Low-level intrinsic onl
Low-level intrinsic only
0.6
HIRO
0.4
HIRO
Reward
1.5
HIO
0.4
0.3
1
02
0.2
0.5
9.9
Episodes
x10*
Episodes
X10
Episodes
x10*
(a)
(b)
(c)
Figure 4: Success Rate on (a) AntMaze (b) Sazeazzeee and Reward on (c) AntGather using HIRO-- G4RL, HIRO + High-level intrinsic, HIRO Low-level intrinsic and HIRO. All curves have been equally smoothed for better visualization. The combination of high-level and low-level intrinsic rewards results in the highest success rates and fastest convergence.
AntMaze
AntMaze-Sparse
AntGather
3
0.8
0.8
2.5
Success Raie
Supcess Rete
2:
0.8
Reward
1.5
0.4
0.4
HighLLowieve intrinsics
High-LLowleeel intrinsics High-ioval Intrinsic
HighLLoweer evel intrinsics
Hig-lavel Intrinsic only
Hih-lavel Intrinsic only
0.2
Low-evell intrinaic only
0.2
Low-lovel intriisic only
0.5
Low-ovell intrinsic only
HRAC
HRAC
HRAC
6
6
Episodes
X101
Episodes
X10
Episodes
X10
(a)
(b)
(c)
Figure 5: Success Rate on (a) AntMaze (b) Ant Sparse and Reward (c) AntGather using HRAC- G4RL, HRAC + High-level intrinsic, HRAC + Low-level intrinsic and HRAC. All curves have been smoothed equally for better visualization. The combination of high-level and low-level intrinsic rewards results in the highest success rates and fastest convergence.
Figures 4 to show that, across all tested environments and algorithms, the combination of high-level and low- level intrinsic rewards results the highest success rates and fastest convergence. The high-level intrinsic- only variant outperforms the low-level intrinsic-only variant, especially in sparse reward tasks, indicating that high-level intrinsie rewards play crucial role in facilitating efficient exploration by encouraging the agent to select reachable and meaningful subgoals. In contrast, low-level intrinsic rewards have limited effect exploration, primarily refining the exccution of local bchaviors. These results demonstrate that intrinsic rewards at different hierarchy levels serve complementary functions, and their combination yields superior performance.
4.3.2 Balancing between time and performance
To assess the tradc-off between computational efficiency and performance, we evaluate two acceleration stralegies mentioned in Section 3.5 First, we vary the sampling frequency of node features by testing intervals of 1, 5, and 10 steps in HLPS. As shown Figure increasing the sampling interval substantially roduces computation time, as docreases the number of interactions with the graph, with only minor degradation in success rates across both AntMaze and AntPush tasks. Second, we vary the proportion of training data used for the graph encoder-decoder, testing 50%, 75%, and 100% subsets. Results in Figure g
10

---

Published in Transactions on Machine Learning Research (09/2023)
AntMaze
AntMaze-SSarse
AntPush
0.8
0.8
0.8
Success Rale
Suncoss Rele
Success Raia
0.6
0.6
HightLoowlleee intrinsies
High+Low-leve intrinsics
HightLLow-lave intrinsics
High-level intrinsic only
High-level intrinsic
High-level intrinsic only
0.2
Low-level intrisic only
02
Low-level intrinsic only
0.2
Low-level intrinsic only
HESS
HESS
HESS
0
0
Episodes
X101
Episodes
x10"
Episodes
x10*
(a)
(b)
(c)
Figure 6: Success Rate on (a) AntMaze (b) Maz-azzeSparse and (c) AntPush using HESS-GARL, HESS + High-level intrinsic, HESS + Low-level intrinsic and HESS. All curves have been equally smoothed for better visualization. The combination of high-level and low-level intrinsic rewards results in the highest success rates and fastest convergence.
AntMaze
AntMaze-Sparse
AntPush
0.8
Success Rate
Success Fale
Success Rata
0.6
0.6
0.6
0.4
HightLowe oowllvel intrinsies)
0.4
High+L Lww-lvel intrinsics
0.4
HightL oow-lvvl intrinsics
High-level intrinsic only
High-level intrinsic only
High-level intrinsic only
0.2
Low-levei Intrinaic only
Low-level intrinsic only
0.2
Low-lave intrinaic only
HLPS
HLPS
HLPS
6
Episodes
x101
Episodes
x10*
Episodes
x101
(a)
(b)
(c)
Figure 7: Success Rate (a) AntMaze (b) Saz-are-e and (c) AntPush using HL.PS-4RT, HLPS High-level intrinsic, HLPS Low-level intrinsic and HLPS. All curves have been equally smoothed for better visualization. The combination of high-level and low-level intrinsic rewards results in the highest success rates and fastest convergence.
indicate that reducing the amount of training data leads to only marginal improvements in computational efficiency and has negligible impact on final performance.
These findings suggest that the primary computational bottleneck of G4RL lies in the graph construction and node comparison processes described in Section 3.1.1 rather than in the encoder-decoder training itself. Adjusting the sampling frequency is therefore an effoctive approach for reducing time cost while largely preserving the benefits of G4RL integration.
4.4 Subgoal space visualization
This section shows how the subgoal space evolves in the AntMaze environment as the number of train- ing episodes grows. We recorded state representations encountered in specific episodes and then used the corresponding graph encoders from those episodes to map these state representations to the subgoal repre- sentations. The subgoal representations are projected into 2D using t-SNE for visualization.
The distributions of subgoal representations in the subgoal space across different episodes are shown in Figure 10
From the figure we can conclude that the graph encoder-decoder gives better subgoal representations when the number of episodes grows. At the 2000th episode, the subgoal representations form few chains in the
11

---

Published in Transactions on Machine Learning Research (09/2025)
AntMaze
AntPush
0.8
0.8
Suogess
0.4
step, 1.9x time
0.4
step, 2.0x time
steps, 1.5x time
steps, 1.7x time
10 steps, 1 4x time
10 steps, 1 1.6x time
HUPS,
HLPS. 1.0x time
Episodes
x10
Episodes
x10
(a)
(b)
Figure 8: Success Rate on (a) AntMaze and (b) AntPush using HLPS- - G4RL and HLPS. The number of steps in the legend indicates the selection of Le as described in Section 3.5 and the timescale is calculated
w.r.t. the vanila HLPS algorithm. Increasing the sampling interval substantially reduces computation time, with only minor degradation in success rates across both AntMaze and AntPush tasks.
AntMaze
AntPush
0.6
0.8
Suiress
Success
0.6
0.4
100% data, 1 9x time
0.4
100% data, 2 time
75% data, 1.8x time
75% data, 1.9x time
50% data, 1.8x time
0.2
50% data, 1.8x time
HLPS, 1.0x time
HLPS, 1.0x time
Episades (a)
X10
Episades
X10*
(b)
Figure 9: Success Rate on (a) AntMaze and (b) AntPush using HLPS+G4RL and HLPS. The percentage of data in the legend indicates the amount of data used in the training of the graph encoder-decoder described in Section 3.5 and the timescale is calculated w.r.t. the vanila HLPS algorithm. Reducing the amount of training data leads to only marginal improvements in computational efficiency and has nogligible impact on final performance.
Epicode 2000
Epiiode 4000
Episode 6000
Dmm1
(a)
(b)
(c)
Figure 10: The distribution of subgoal representations in the subgoal space projected by t-SNE at the (a)
2000th (b) 4000th (c) 6000th episode. The subgoal space evolves as the training comes into later episodes.
2D space, indicating that the current graph is partitioned into many isolated sub-graphs. The graph encoder
12

---

Published in Transactions on Machine Learning Research (09/2025)
has not yet learned to understand the environment well, and the agent cannot find relationships between different state representations yet. By the 4000th episode, the representations are less isolated, suggesting that the graph encoder has begun to learn the connections between different groups of state representation. Finally, at the 6000th episode, the subgoal representations are distributed more evenly across the entire space, indicating that the encoder can organize them into a highly connected graph. The subgoal space now better represents the underlying problem's 's structure.
The evolution of the graph encoder demonstrates that it can indeed learn to represent the task structure the number of episodes grows.
5 Conclusion
We have presented novel approach using a graph encoder-docoder address the challenges of poor subgoal representations and sample inefficiency in GCHRL. The proposed architecture is designed to efficiently evaluate unseen states by operating in the graph representation space. It is easy to implement and can be scamlessly integrated into any existing GCHRL algorithms to enhance their performance in primarily symmetric environments. Our experiments on both sparse and dense control tasks have demonstrated the effectiveness and robustness of our method.
References
Amr Ahmed, Nino Shervashidze, Shravan Narayanamurthy, Vanja Josifovski, and Alexander J Smola. Dis- tributed large-scale natural graph factorization. In Proceedings of the 22nd international conference on World Wide Web, pp. 37 48, 2013.
Shaosheng Cao, Wei Lu, and Qiongkai Xu. Grarep: Learning graph representations with global structural information. In Proceedings of the 24th ACM international on conference on information and knowledge management, pp. 891 900, 2015.
Fenxiao Chen, Yun-Cheng Wang, Bin Wang, and C-C Jay Kuo. Graph representation learning: survey. APSIPA Transactions on Signal and Information Processing, 9:c15, 2020.
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International conference on machine learning, pp. 1329-1338. PMT.R, 2016.
Ben Eysenbach, Russ R Salakhutdinov, and Sergey Levine. Search on the replay buffer: Bridging planning and reinforcement learning. Advances in neural information processing systems, 32, 2019.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pp. 1587-1596. PMTR, 2018.
Robert Gieselmann and Florian T Pokorny. alamning-augmented hierarchical reinforcement learning. IEEE
Robotics and Automation Letters, 6(3):5097 5104, 2021.
Shangqi Guo, Qi Yan, Xin Su, Xiaolin Hu, and Feng Chen. State-temporal compression in reinforcement learning with the reward-restricted geodesic metric. JEEE Transactions on Pottern Analysis and Machine Intelligence, ((59))5572-00 55572-5589, 2021.
David Ha and Jürgen Schmidhuber. World models. arXio preprint 103. 2018.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint XrXiv:1812. 181500901, 2018.
William L Hamilton. Graph representation learning. Synthesis Lectures on Artifical Intelligence and Machine Learning, 14(3):1-155) 2020.
13

---

Published in Transactions on Machine Learning Research (09/2025)
Zhang-Wa Wei Hong, Tao Chen, Yen-Chen Lin, Joni Pajarinen, and Pulkit Agrawal. Topological experience replay. arxie preprint arXiv:2203. 15845, 2022.
Yang Hsu, Kyle Cox, Jiawei Xu, Zhen Tan, Tianhua Zhai, Mengzhou Hu, Dexter Pratt, Tianlong Chen, Ziniu Hu, and Ying Ding. Thought graph: Generating thought process for biological reasoning. In Companion Proceedings of the ACM Web Conference 2024, pp. 537-540, 2024.
Zhiao Huang, Fangchen Liu, and Hao Su. Mapping state space using landmarks for universal goal reaching, Advances in Neural Information Processing Systems, 32, 2019.
Shima Khoshraftar and Aijun An. A survey on graph representation learning methods. ACM Transactions on Intelligent. Systems and Technology, 15(1):1-55, 2024.
Junsu Kim, Younggyo Seo, and Jinwoo Shin. Landmark guided subgoal generation in hierarchical reinforce- ment learning. Advances in neural information processing systems, 14:28336-283449 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
44 6980, 2014.
Martin Klissarov and Marlos C Machado. Deep laplacian- based options for tenporally-estennde exploration. arXiv preprint or xrx:22301.11181 2023.
Martin Klissarov and Doina Precup. Reward propagation using graph convolutional networks. Advonces in Neural Information Processing Systems, 33: 53:12895 12908, 2020.
Tejas D Kulkarni, Karthik Narasimhan, Ardavan Sacedi, and Josh Tenenbaum. Hicrarchical deep reinforce- ment learning: Integrating temporal abstraction and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
Seungjae Lee, Jigang Kim, Inkyu Jang, and H Jin Kim. Dhrl: a graph-based approach for long-horizon and sparse hicrarchical reinforcement learning. Advances in Neural Information Processing Systems, 35: 13668-13678, 2022.
Andrew Levy, George Konidaris, Robert Platt, and Kate Saenko. Learning multi-level hierarchies with hindsight. arXiv preprint arXio:1712.0094 2017.
Siyuan Li, Jin Zhang, Jianhao Wang, Yang Yu, and Chongjie Zhang. Active hierarchical exploration with stable subgoal reprosentation learning. ar preprint arXiw:215.. 105.:2105.14750, 2022.
Yu Luo, Tianying Ji, Fuchun Sun, Huaping Liu, Jianwei Zhang, Mingxuan Jing, and Wenbing Huang, Goal- conditioned hicrarchical reinforcement learning with high-level model approximation. IEEE Transactions on Neural Networks and Learning Systems, 2024.
Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Near-optimal representation learning for hierarchical reinforcement learning. arXiv preprint 1xXi::1810.01951 2018a.
Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement lcarning. Advances in neural information processing systems, 31, 2018b.
Clrich Nehmzzw, Yiannis Gatsoulis, Emmett Kerr, Joan Condell, Nazmul Siddique, and T Martin McGuin- nity. Novelty detection as an intrinsic motivation for cumulative learning robots. Intrinsically Motivated Learning in Natural and Artificial Systems, pp. 185 207, 2013.
Mingdong Ou, Peng Cui, Jian Pei, Ziwei Zhang, and Wenwu Zhu. Asymmetric transitivity preserving graph embedding. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 1105-1114, 2016.
Martin L Puterman. Markow decision processes: discrete stochastic dynomic programming. John Wiley & Sons, 2014.
14

---

Published in Transactions on Machine Learning Research (09/2025)
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithins. arXiv preprinl arxi::1707 1111770706447 2017.
Wenling Shang, Alex Trott, Stephan Zheng, Caiming Xiong, and Richard Socher. Learning world graphs accelerate hierarchical reinforcement learning. arXiv preprint ar orXim:1907.00664 2019.
Ozgur Simsek and Andrew G Barto. An intrinsic reward mechanism for efficient. exploration. Proceedings of the 23rd international conference on Machine learning, pp. 833 840, 2006.
Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181 211, 1999.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model- based control. In 2012 IEEE/RSJ international conference intelligent robots and systems, pp. 5026-5033. IEEE, 2012.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In International conference on machine learning, pp. 3540 3549. PMLR, 2017.
Guojia Wan, Shirui Pan, Chen Gong, Chuan Zhou, and Gholamreza Haffari. Reasoning like human: Hi- erarchical reinforcement learning for knowledge graph reasoning. International Joint Conference on Artificial Intelligence. International Joint Conference on Artificial Intelligence, 2021.
Vivienne Huiling Wang, Tinghuai Wang, Wenyan Yang, Joni-Kristian Kämäräinen, and Joni Pajari- nen. Probabilistic subgoal represenlations for hierarchical reinforcement learning. arXiv preprinl arXiv:2406.166 16707, 2024.
Lunjun Zhang, Ge Yang, and Bradly C Stadie. World model graph: Learning latent landmarks for planning. In International conference on machine learning, pp. 12611-12620. PMIR, 2021.
Tianren Zhang, Shangqi Guo, Tian Tan, Xiaolin Hu, and Feng Chen. Adjacency constraint for efficient hierarchical reinforcement learning. IEEE Transactions Pattern Analysis and Machine Intelligence, 45 (4):4152-4166 2022.
Hong Zhao, Yao Fu, Weihao Jiang, Shiliang Pu, and Xiaoyu Cai. Simulate human thinking: Cognitive knowledge graph reasoning for complex question answering. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pp.
Springer, 2022.
Deyao Zhu, Li Eran Li, and Mohamed Elhoseiny. Value memory graph: A graph-structured world model for offline reinforcement learning. arXiv preprint arXiv:2206.048 04384, 2022.
15

---

Published in Transactions on Machine Learning Research (09/2025)
A Limitations and Future work
Despite the advantages demonstrated by G4RL in HRL tasks, its effectiveness depends significantly on several hyperparameters (e.g., Ed. Og, and oy), which require careful tuning to achieve optimal performance across different environments.
In future work, we aim develop methods for automatically selecting these hyperparamcters based on environmental dynamics, thereby reducing the need for manual tuning. Also, we plan extend our work by exploring how to generate subgoals with more interpretable representations to facilitate knowledge transfer, potentially leveraging alternative graph representations (e.g. Graph Laplacian). Another promising direction is to transfer the knowledge embedded in the graph structure to new tasks by analyzing graph topology and establishing mappings between nodes of different state graphs. Additionally, more evidence on asymmetric environments is needed to demonstrate G4RL's robustness when syymmetric/irrevosiiii transitions are present.
16

---

Published in Transactions on Machine Learning Research (09/2025)
B Implementation details
B.1 Environment details
AntMaze This environment is part of the Syymnasium-Roooieie libraries. The size of the environment is 24 x 24. Both of the state space and the action space are continuous, with a state dimension of 31 and an action dimension of 8, The reward of each step is defined by its negative Euclidean distance from the current location the target position. At evaluation time, the goal is to (0, 16) and an episode is recognizod successful if the agent is within Euclidean distance of 5 from the goal.
AntMaze Sparse This environment is variant of Antmaze. The size of the environment is 20 x 20. The state and action spaces are the same as those in AntMaze. The reward of cach step is 1 if and only if the agent reaches within an Euclidean distance of 1 from the goal, which is set to (2,9).
Gather This environment is described in Duan et al. (0166). The size of the environment 20 x 20. The state and action spaces are continuous. The task involves gathering apples to the designated place. The agent will be awarded +1 for each apple gathered and -1 for each bomb gathered. Apples and bombs are randomly placed in the 20 20 world.
AntPush uush The size of the environment is 20 20. The state and action spaces are continuous. A challenging task that requires both task and motion planning. The agent needs to move to the left then move up and push the block to the right in order to reach the goal.
B.2 Network architecture details
Our network architecture for the HRL agents is the same described in Nachum et al. (0188)), Zhang et al. 20022], Li el al. (2022) and Wang el al. (2024). For HIRO, HRAC, HESS and HLPS both the high-level and low-level agents use the TD3 algorithm. The size of each hidden layer in actor and critic networks in TD3 is 300.
For the graph encoder-decoder, activation function used is ReLUU The decoder is dot product of the two input subgoal representations.
four-layer fully connected network with the hidden size of 128. The
Wc use Adam as the optimizer for the actor network, critic network and the graph encoder is Adam optimizer Kingma & Ba 2014).
17

---

Published in Transactions on Machine Learning Research (09/2025)
C Hyperparameters
In this section we list all hyperparameters used in our experiments.
Hyperparameters
Values
High-level agent
Actor learning rale
0.0001
Critic learning rate
0.001
Batch size
128
Discount factor ?
0.99
Policy update frequency
High-level action frequency
10
Replay buller size
20000
Exploration strategy
Gaussian
Low-level agent
Actor learning rate
0.0001
Critic learning rate
0.001
Batch size
128
Discount factor 7
0.99
Policy update frequency
Replay buffer size
20000
Exploration strategy
Gaussian(r = 1)
Table 1: Hyperparameters used in high- and low-level TD3 agents.
Hyperparameters
Values
Number of nodes N Batch size
200
128
Optimizer learning rate
0.0001
Ed
0.1 for Ant.aaee/Oa 00. for others
O
0.1
aq
0.1
8
0.2
Table 2: Hyperparameters used the graph encoder-decoder.
18

---

Published in Transactions on Machine Learning Research (09/2025)
D Algorithm
Now we describe our proposed method in Algorithm
Algorithm GCHRL+G4RL
Require:
High-level policy Tx (x))((()), low-level policy =)(a((()),)) g), replay buffer B,
graph encoder E, graph decoder D, high-level action frequency K, significance hyperparamcter an and a,, tolerance hyperparameter B, number of episodes N, nunber of steps in episode T.
=0
while S N do ==
3:
4:
c=0
5:
while <
if mod = then
Execute the high-level policy (9:)((()))) sample the subgoal Jt. clse
Keep the subgoal Si unchanged.
10:
Execute the low-level policy maa (o(*), gt) to sample the atomic action az
11:
Sample reward ra and next state 81+1-
12:
Calculate rh (8t, 9t, 8t-1) and (1(, Dt, 0t, 8141) using 9 and 10
13:
Collect experience (st, It, Qe, Th; Fi, 81+1) and update the replay buffer B. Updale node representations and edge weights using collected experience.
14:
16:
Update using 8
16:
if 2/ 3 then
17:
Update graph encoder with node representation and edge information in the graph. 0=
18:
19:
t=t+1
20:
Update low-level policy maa(((s)))) using the chosen HRL algorithm.
21:
Update high-level policy Ty (g) (s)) using the chosen HRL algorithm.
22:
-=++
19

---

Published in Transactions on Machine Learning Research (09/2025)
E Un-smoothed experiment results
In this section, we present the un-smoothed versions of figures 4 and 5 in the main text to show the full details of the experiments.
AntMaze
AntMaze-Sparse
AntGather
2.b
lived inbiralce High leot only
Hot+Loww Insiraic
0.8
"igh- evsl mac only
-co- uni iitit sacc
Lowvitee inmios only
Joan and intrraic
Success Ralts
1800
Sacos Rala
0.5
H NN
+RO
0.0
Reward
1.5
0.4
0.2
0.2
0.5
Episodes
x101
Episodes
x10f
Episodes
xnof
(b)
(c)
Figure 11: Success Rate on (a) AntMaze (b) AntMaze-Sparse and Reward (c) Gnthet using HIRO- G4RL, HIRO High-level intrinsic, HIRO Low-level intrinsic and HIRO. Smoothing is removed in contrast to figure
AntMaze
AntMazo-Sparso
AntGather
3.5
0.8
0.8
Succega Rele
2.5
0.6
Reward
2
1.5
0.2
intrrvatc
0.2
Hightowoll intrrdo
intrieaic only
Love monisse only
0.5
Low intreod
0
0
Episodes
x10
Episodes
X10
Episades
x104
(a)
(b)
(c)
Figure 12: Success Rate on (a) AntMaze (b) Ant Ant-aaze-Spasee and Reward on (c) AntGather using HRAC- G4RL, HRAC + High-lovel intrinsic, HRAC + Levelevel intrinsic and HRAC. Smoothing is removed contrast to figure
20

---

Published in Transactions on Machine Learning Research (09/2025)
F Exploring the subgoal space through diverse visualization techniques
In this section, first extend the visualization results presented in Figure 10 by applying multiple dimen- sionality reduction techniques namely PCA, I-SNE, and UMAP to demonstrate that the observations
discussed in Section 4.4 are not specific to any single method. The results are shown in Figure to
Episode 2000
Episode 4000
Episode 6000
(b)
Figure 13: The distribution of subgoal representations in the subgoal space projected by PCA at the (a)
2000th (b) 4000th (c) 6000th episode.
Episode 2000
Episode 4000
Episode 6000
(b)
Figure 14: The distribution of subgoal representations in the subgoal space projected by I-SNE at the (a)
2000th (b) 4000th (c) 6000th episode.
Episode 2000
Episode 4000
Episode 6000
(c)
Figure 15: The distribution of subgoal representations in the subgoal space projected by UMAP at the (a)
2000th (b) 4000th (c) 6000th episode.
21

---

Published in Transactions on Machine Learning Research (09/2025)
While the precise projections vary across methods, consistent trend emerges: the representations pro- gressively evolve from forming scattered clusters to a more distributed pattern. This trend aligns with the objective of G4RL- to learn & subgoal representation extractor that structures subgoals within a space fune- tioning as a cognitive map, effectively capturing the underlying structure of the original state space. Such a map can be interpreted as an internal model of the environment, offering valuable information for planning and decision-making.
Figures 13 illustrate how the graph encoder progressively learns to utilize the subgoal space more effectively. Initially, it relies only limited region of the space, but over time, its usage becomes more dis- tributed. This progression further supports the claim that the learned subgoal representations are mcaningful and hold potential for providing continuous guidance.
We then visualize trajectories sampled at different stages of training in the AntMaze environment, projecting ((s;) into the state space and g(st) into the subgoal space, respectively. Results are shown in Figures
* gual
+ start
+ end
(a)
(b)
Figure 16: Projection of (a) state representations ((x) in the state space and (b) subgoal representations g(s:) in the subgoal space the 2000th episode. The colour spectrum indicates the relative time step (red corresponds to carlier steps), while the green and red crosses mark the projected start and end points, respectively. The green star denoles the projected ultimate goal in the same space.
Figures 16 to 19 demonstrate that the learned subgoal space evolves progressively to reflect the agent's observations of the state space. A clear correspondence emerges between the connection structure of state representations ((s;) in the state space and subgoal representations g(st) in the subgoal space. This supports the claim that the subgoal space captures the connectivity of the original state space and that the proposed subgoals offer accurate and consistent guidance toward the final goal.
22

---

Published in Transactions on Machine Learning Research (09/2025)
goal start
end
(a)
(b)
Figure 17: Projection of (a) state representations (8)) in the state space and (b) subgoal representations g(s)) in the subgoal space at the 4000th episode. The colour spectrum indicates the relative time step (red corresponds carlier steps), while the green and red crosses mark the projected start and end points, respectively. The green star denotes the projected ultimate goal in the same space.
* goal start
end
(a)
(b)
Figure 18: Projection of (a) state representations ((8)) in the state space and (b) subgoal representations g(s:) in the subgoal space at the 6000th episode. The colour spectrum indicates the relative time step (red corresponds to earlier steps), while the green and red crosses mark the projected start and end points, respectively. The green star denotes the projected ultimate goal in the same space.
23

---

Published in Transactions on Machine Learning Research (09/2025)
* goal
+ start
end
(b)
Figure 19: Projection of (a) state representations ((x)) in the state space and (b) subgoal representations g(s)) in the subgoal space at the 8000th episode. The colour spectrum indicates the relative time step (red corresponds earlier steps), while the green and red crosses mark the projected start and end points, respectively. The green star denotes the projected ultimate goal in the same space.
24


Reviews:
[{"review_id": "kX4mkMv4D0", "reviewer": "Reviewer_aBsC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n* Novel integration of graph learning with hierarchical RL:\n  The paper introduces a graph encoder–decoder that enables subgoal representations to capture spatial connectivity among states. This provides a new perspective on addressing both high-level exploration inefficiency and low-level reward signal quality within a unified framework.\n\n* Clear formulation and implementation details:\n  The method is described systematically, including graph construction, updating, encoder–decoder training, and reward shaping. Pseudocode and hyperparameter tables in the appendix improve clarity and reproducibility.\n\n* Extension to image-based inputs:\n  Demonstrating applicability beyond low-dimensional state features adds credibility to the method within robotics-style environments.\n\n\n### Weaknesses\n\n* Decoder assumptions and symmetry restriction:\n  The dot-product decoder implicitly assumes that the adjacency structure of the state graph can be embedded into a symmetric Euclidean space (i.e., low-rank approximation). This design enforces undirected graphs, which may be reasonable for reversible navigation tasks but limits applicability to environments with asymmetric or non-Euclidean connectivity (e.g., discrete rooms, teleportation, one-way transitions). As a result, the claim of being “applicable to any GCHRL method” may be overstated.\n\n* Baseline limitations:\n  The non-hierarchical baseline (TD3) predictably underperforms in long-horizon sparse-reward tasks. Without including stronger goal-conditioned baselines such as TD3+HER, it is difficult to fully assess the comparative advantage of G4RL.\n\n* Evaluation domain narrowness:\n  All experiments are conducted in MuJoCo Ant-based environments, which share similar geometric properties. This may bias results toward methods leveraging spatial embeddings, and leaves applicability to broader domains (e.g., strategic video games, non-robotics tasks) uncertain.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper addresses the challenge of learning effective subgoal representations and improving exploration efficiency in Goal-Conditioned Hierarchical Reinforcement Learning (GCHRL). The key contributions can be summarized as follows:\n\n1. Graph-Guided sub-Goal representation Generation (G4RL):\n   The authors propose a novel architecture that integrates a graph encoder–decoder into GCHRL. The encoder maps state representations into a subgoal embedding space, while the decoder—implemented via dot product—ensures that the learned embeddings preserve adjacency relations in the state graph. This design enables the system to evaluate both visited and unseen states more effectively.\n\n2. Online construction of state graphs:\n   The method maintains an evolving undirected state graph during exploration, where nodes correspond to visited states and edges encode observed transitions. The graph is updated dynamically as training proceeds, avoiding reliance on handcrafted or pre-defined graphs.\n\n3. Intrinsic rewards at both hierarchy levels:\n   By leveraging distances in the learned subgoal space, the framework introduces novelty-based intrinsic rewards for both the high-level and low-level agents. This aims to guide the high-level agent toward reachable and meaningful subgoals and provide the low-level agent with more accurate progress signals.\n\n4. Empirical validation across multiple benchmarks:\n   Experiments are conducted on several MuJoCo tasks (AntMaze, AntGather, AntPush, AntFall, and AntMaze-Sparse) under both dense and sparse reward settings, as well as image-based state inputs. Results indicate that incorporating G4RL consistently improves sample efficiency and final success rates across multiple backbone GCHRL algorithms (HIRO, HRAC, HESS, HLPS).\n\n5. Additional analyses:\n   The paper includes ablation studies that separately examine the effects of high- vs. low-level intrinsic rewards, as well as computational trade-offs such as graph update frequency and training data subsampling. A visualization of the subgoal embedding space further illustrates how the learned representations evolve over training.\n\nOverall, the submission contributes a general mechanism for embedding spatial information into subgoal representations via graph learning, which can be combined with existing GCHRL methods to enhance performance in long-horizon, sparse-reward environments.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "### Critical\n\n* Clarify and justify the decoder design and undirected graph assumption:\n  The paper currently employs a dot-product decoder with an undirected graph, and explains this mainly in terms of computational efficiency and compatibility. However, this design implicitly imposes strong assumptions on the environment’s dynamics:\n\n  * It assumes that the adjacency matrix can be embedded into a symmetric Euclidean space (i.e., low-rank approximation).\n  * It enforces undirected connectivity, thereby biasing the representation toward environments with reversible dynamics (e.g., physical navigation tasks), while making it unsuitable for settings with inherently asymmetric transitions (e.g., teleportation, one-way actions, irreversible state changes).\n\n  This implicit bias directly affects the claim of being “applicable to any GCHRL method.” The authors should explicitly discuss these assumptions in the paper, clarify the extent to which their method is restricted to approximately symmetric/Euclidean domains, and justify why dot-product similarity is chosen over other possible decoders. Without such discussion, the generality claim remains questionable.\n\n\n### Optional (would strengthen the work, but not critical for acceptance)\n\n* Strengthen baselines:\n  Since TD3 predictably fails in sparse-reward long-horizon tasks, incorporating stronger goal-conditioned baselines such as TD3+HER would provide a more balanced comparison.\n\n* Broaden evaluation domains:\n  The evaluation is limited to MuJoCo Ant-based tasks that share similar geometric properties. Adding results on domains with less geometric regularity (e.g., discrete rooms, symbolic or video game environments) would better support the claim of broad applicability.", "broader_impact_concerns": "no", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "I7SzGrLIgk", "reviewer": "Reviewer_ZrNt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Please note that I am not confident in commenting on the novelty / strength of the contribution of this work.\n### Strengths\n- Good Preliminaries section.\n- I think Section 3 is really written well with each major component of the proposed method being given text-space to be explained clearly.  \n- Appreciate Section 3.5 that analyses the speed / performance trade off in their method. \n- Extremely pleased to see that a large number of repetitions of the experiments was carried out!\n- Good experiments with good ablation studies.\n\n#### Weaknesses\n- The main weakness is in writing style and consistency of the exposition. These can be easily fixed — see Requested Changes below.\n- Are you using a directed or undirected graph? From Equation (4) I expect it is an undirected graph? If so, why is this a better choice than a directed graph; I would expect that state transitions aren't generally symmetric.\n- From what I can see, code isn't made available.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides a method for use in Goal-Conditioned Hierarchical Reinforcement Learning that tackles issues of high-level exploration,  effective subgoal space creation and sample efficiency (of both the high- and low-level agents). This method can be added to any existing method and uses a graph-based method for mapping spatial information to subgoal representations; a spatial graph that leverages the state representation and the connectivity to make the mapping to subgoals is maintained and updated (as new states are visited). Experiments were carried out on several variants of the Ant-X environments on MuJoCo, with baselines from various techniques; the experiments suggest that this additive method provides meaningful improvements.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Page 1: There seems to be a link attached to the footer / page number on Page 1; is this intentional?\n- Introduction, Paragraph 1; \"its resemblance to the human thinking process\" -> Is there a reference for this statement?\n- Introduction, Paragraph 3; \" no prior work has attempted to address them in a unified framework.\" -> This sentence suggests that your method is a 'unifying framework' — is this true? If so, are you trying to unify the:\n\t1.  \"enhance(ment of) the efficiency of high-level exploration\"\n\t2. \"find(ing of) a good subgoal space to boost GCHRL’s performance in complex environments\"\n\tif not, then please be more specific.\n- Introduction, Paragraph 3; \"Yet, such integration is crucial, as uniting an effective high-level agent with an accurate low-level agent within a cohesive framework can yield performance improvements that surpass simple additive effects.\" -> This sentence makes it sound like other works don't train the low-level and high-level agent jointly and cohesively. Is this true to say?\n- Introduction, Paragraph 5; \"Additionally, most existing works...\" -> Please add references to \"most existing works...\".\n- Section 2.1; \"Notation\" -> Perhaps a better title for this section is \"Reinforcement Learning\" or \"Markov Decision Processes\".\n- Section 2.2, Paragraph 1; \"Goal-Conditioned Reinforcement Learning (GCRL) trains agents to achieve specific goals, which are the target states. The agent receives an additional goal input along with the state input and learns a policy that aims to achieve this goal. Goals are represented explicitly in the input to the policy, guiding the agent’s actions towards desired outcomes\" -> Wouldn't it make sense to add math notation here? E.g., \"The aganet receives an additional goal input $g_t$ along with the state input $s_t$ and learns a policy $\\pi(a_t | g_t, s_t)$ that aims to achieve this...\".\n- Section 2.2, Paragraph 2 -> Is the norm missing a subscript 2?\n- Equation (2) -> According to this equation, is the graph stored in terms of the raw state $s_t$, or the state representation $\\phi(s_t)$? If not the state representation, what is the reason? Since the state representation is used in computing the distances in Equation (1) and Equation (3), wouldn't it be more computationally efficient to store the state representation?\n- Figure 1 -> can the legend be moved to the top of the figures.\n- Figure 2, 3 -> can the arrangement of the legends be the same as Figure 1.\n- Figure 4 and 5 -> Please add the fact that the curves were smoothed in the caption. Furthermore can the un-smoothed figures be added to an Appendix for verification that any conclusions weren't affected by the smoothing. \n- Figures in general -> I think it is useful to write the main conclusion of each figure in the figure caption.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xBkceK7kcO", "reviewer": "Reviewer_v6Rq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n* Method is conceptually simple.\n* Outperforms previous GCHRL methods (though sometimes it seems marginal).\n\n**Weaknesses**:\n* In general, the writing of this paper could benefit from extensive polishing (“a target state that the high-level agent wants the low-level agent to achieve as a part of the overall task.” For example, I don’t think “wants” is a formal language for a paper. There are many such examples throughout the paper.). It feels more like a “project report” rather than a paper.\n* Method is simple in essence, but includes complex implementation details such as tuning the adaptive training schedule of the network. I think that Section 3.3 requires some more clarifications, the schedule design was not entirely clear to me.\n* The method introduces increased time complexity due to training and updating the graph. While the authors propose a way to alleviate this by performing the operations every pre-defined time interval, I believe that with more complex tasks this will be detrimental to the performance.\n* Simple and limited environments and tasks (mostly aligning with previous work). State-based only (i.e., no image-based inputs).\n* I believe the results section requires more discussion.\n* Lacking discussion of limitations.\n\n**Minor**:\n* Notations: Section 2.2 – I found the notations $g_t$ and $g(s_t)$ very confusing and I highly recommend choosing a different notation for $g(s_t)$.\n* Page 4, first paragraph: I find the statement “This approach provides more accurate reward signals and leads to more reasonable target subgoals.” a bit too strong. If the author could provide more insights as to why, or better yet, a reference to empirical results that prove that, it would be great. Otherwise, I suggest toning it down.\n* Page 4: “We perform the GCHRL explore process” -> “exploration process”\n* Figure 1: something is wrong with the legend. While all graphs include all colors, the legend is divided between figures.\n* Figure 7: why 5 steps take longer than 10 steps? Or is it a mistake in the figure?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper addresses several issues with previous GCHRL methods, such as better utilization of graph representations and formulation of better graph-based sub-goal mechanisms. The proposed method is a graph autoencoder to represent sub-goals (thus allowing to encode “new unseen states”), from which they design a novelty-based intrinsic reward for improved exploration and sample-efficiency. Notably, the “decoder” is designed as the inner product between two state representations to predict their adjacency value, making it possible to predict how close two states are.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Overall, I’m not certain the paper provides enough evidence for its claims, for two main reasons: (1) the writing and language make it seem like a “project report” rather than a journal paper; and (2) the experimental results are limited to few state-based environments, and while it seems that they somewhat outperform previous methods, it comes at a computational cost (>=X 1.5). My main request here is to improve the writing and maybe present the results in a more convincing manner (e.g., by adding more discussion to the results sections as I mentioned above).\n* See “Weaknesses” and “Minor” above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)