A tale of two goals: leveraging short term goals performs best in multi-goal scenarios
Authors: Olivier Serris
Stephane Doncieux
Olivier Sigaud
Venue: Accepted by TMLR ()

Abstract: When an agent must learn to reach far away goals, several hierarchical reinforcement learning methods leverage planning to create a sequence of intermediate goals  guiding a lower-level goal-conditioned policy. The low-level policy is typically conditioned on the current goal, with the aim of reaching it as quickly as possible. However, this approach can fail when intermediate goals can be reached in multiple ways, some of which may prevent continuing toward subsequent goals. To address this issue, we introduce an enriched Markov Decision Process (MDP) framework where the optimization objective not only considers reaching the current goal, but also subsequent ones. Using this framework, we can specify which goals the agent prepares to achieve ahead of time. To study the impact of this design, we conduct a series of experiments on navigation, balancing and locomotion tasks in which sequences of intermediate goals are given. By evaluating policies trained with an off-policy actor-critic algorithm on both the standard goal-conditioned MDP framework and ours in these tasks, we show that learning policies conditioned on the next two goals generally require less interaction data than all other types to reach the same level of performance or better.

Published in Transactions on Machine Learning Research Mlarch/2020)
A tale of two goals: leveraging short term goals performs best in multi-goal scenarios
Olivier Serris
Dlivier.Serris@isiin. upme.f
ISTR Sorbonne Université, CNRS
Stéphanc Doncicux
Stephane. www.eeewcoooiiiiiiiiiio... SpppharneDDadieeuiiiiiu.u...p.p upmo..
ISIR
Sorbonne Université, CNRS
Olivier Sigaud
Olivier.Sigaudl Wisir upmo.fr
ISIR
Sorbonne Université, CNRS
Reviewed on OpenReview:
https: /ttps:// opwrreveew.reeee.ooeeeeee net/ forum? id- gsUeLubErp
Abstract
When agent must learn to reach far away goals, several hierarchical reinforcement learn- ing methods leverage planning to create a sequence of intermediate goals that guides lower-level goal-conditioned policy. The low-level policy is typically conditioned on the cur- rent goal, with the aim of reaching it as quickly as possible. However, this approach can fail when intermediate goals can be reached in nultiple ways, some of which may prevent reaching subsequent goals. To address this issue, we introduce an enriched Markov Decision Process (MDP) framework in which the optimization objective not only considers reach- ing the current goal, but also subsequent ones. Using this framework, we can specify the goals that the agent prepares to achieve in advance. To study the impact of this design, we perform series of experiments on navigation, balance and locomotion tasks in which sequences of intermediate goals are given. By evaluating policies trained with an off-policy actor-critic algorithm on both the standard goal-conditioned MDP framework and ours in these tasks, we show that learning policies conditioned on the next two goals generally re- quire less interaction data than all other types to reach the same level of performance or better.
1 Introduction
In reinforcement learning (RI.), an agent learns to control system to accomplish task over sequence of actions by maximizing some reward signal. Standard RL methods struggle when the sequence of actions is long and the reward signal is sparse. Goal-conditioned (GC) agents (Schaul et al. 2015). where the agent must learn a GC-policy to reach any goal in given goal space, are not. exceptions, often failing to reach distant goals. To address this, several hicrarchical reinforcement lcarning (HRL) approaches propose decompose the task into a sequence of intermediate goals, with the GC-policy iteratively conditioned on each goal until the final one is reached (Eysenbach et al. 2019 Levy et al. 2019).
However, as shown in Chenu et al. 2023 when intermediate goals do not fully specify the states in which they are achieved, a chaining issue can arise: the system may achieve a goal in a state that is incompatible with the achievement of the next goal. To counteract this, Chenu et al. (2023) propose a framework in which the agent prepares to reach future goals while aiming at the current one. Their solution consists in integrating the successive goals into the state space of the underlying Markov Decision Process (MDP) together with adapting the transition and reward functions. This results in sequential goal reaching MDP,

---

Published in Transactions on Machine Learning Research (March/20P)
but their approach is limited to the case where the agent must learn to follow a unique predefined sequence of goals from a given starting state.
In this paper, we address the much more general context in which the agent must learn to achieve any final goal from any starting state given a sequence of intermediate goals. In this context, the GC-policy must be able to reach the current goal in a way compatible with the rest of the sequence which itself depends on the final goal. Moreover, while learning to reach goal, it must maintain its ability to reach any other goal from any other sequence
Drawing inspiration from Chenu et al. 2023 we propose a Markov Decision Process (MDP) in which the state, transition and reward functions account for sequential goal reaching and make it possible to fully benefit from goal relabeling provided by an adapted version of the Hindsight Experience Replay (HER) mechanism Andrychooicz et al. 2018). Building on this framework, we focus on learning the low-level GC-policy and assumc access to an expert high-lovel planner that supplies the sequence of intermediate goals given start state and final goal. We propose an RL algorithm, GCP-N, in which the agent is conditioned on both the first and the N-th goal. In this formulation, the agent optimizes its actions to reach all intermediate goals up to the N-th one.
Then, to evaluate these instances, we implement three agents on top of the same actor-critic architecture: an agent conditioned only on the final goal, a myopic agent that only learns to reach the current goal before discovering the next one, and our GCP-N solution. We compare these agents on various navigation, balance and locomotion benchmarks in which the agent is trained to reach any goal from any state given some well-chosen sequence of intermediate goals.
In summary, our contributions are the following: (1) we investigate the failure modes of myopic agents, relating them to the terminal conditions of the underlying MDP; (2) we propose new MDP formalization which accounts for sequential goal reaching; (3) we propose the GCP-N method, which combines Soft Actor- Critic (SAC) with a modified version of HER adapted for multi-goal conditioning; (4) Given the possibility of preparing up to N goals advance, we compare how different choices of N affect different performance criteria: the episode return, the success rate or the time to goal, depending on the environment.
Our results suggest that policies conditioned on the two most immediate successive goals (N = 2) can generally be learned from less interaction data than myopic agents and non-sequential agents to reach the same level of performance or better. Our analysis shows that the GCP-2 agent benefits from the simplicity of learning short-horizon objectives while still being able to complete the full sequence of goals.
2 Background
In this work, we study goal-conditioned reinforcement learning agents that leverage hindsight relabeling and can be seen as the low-level part of a hierarchical approach. In this section, we provide background information about these elements and present some related algorithms.
2.1 Reinforcement learning
In reinforcement learning (RL), the interactions of an agent with an environment are defined as Markov Decision Process (MDP) M = (S, A, R, P, Po T), where S is the state space, A is the action space, R is the reward function, P is the transition probability function, y is the discount factor and po is the initial state distribution. In the cpisodic case, at the beginning of each episode, an initial state su is drawn from the initial state distribution Po- Then at each step, the agent observes 8, S, chooses an action d, A, moves to a new state 8441 according to P and receives reward r. from R. Finally, T corresponds to the set of all terminal states. The goal of RL is to learn agent policy ne that maxinizes the expected cumulative reward until a terminal state is reached, that is
E ET]
Eawit E
Σ rr II
1 1
(s,
somPo
1=0
1=
2

---

Published in Transactions on Machine Learning Research MMarch/2020)
2.2 Goal-conditioned reinforcement learning
Goal conditioned reinforcement learning (GCRL) can be formalized as an extension of MDPs that we call goal-conditioned MDP (GC MDP) and that note Mge (Sye= A, Rges P, Po; T), with Sye = S G. Now, at the beginning of each episode, a single goal g E G is selected in addition to 5o. A goal specifies a state or a set of states. For example, goal might correspond to the 3D position of the center of mass of a robot, and achieving the goal could mean reaching any position within a sphere of diameter e around that point. A mapping function o: + G can project a state into the goal space. At cach step, the agent now observes (st. .) E gg =
and receives a reward ri from the goal-conditioned reward function Rge- In particular, if Sg is the set of all states for which g is achieved, a sparse reward function defined as: Rge(St, It: Se+1) 1 [[s4+1 s..
Finally, T(g) defines the set of terminal states based on the current goal g. Some states may be terminal independently from the current goal (e.g. robot falling on the floor). Reaching a goal can be terminal or not depending on the task at hand. For instance, in a maze environment, the task ends as soon as the agent reaches the goal position, while in a pole-balancing task, the pole must stay at equilibrium in the goal state until the episode ends, hence goals are not terminal.
The objective of GCRL algorithms is to find GC-policy g)) that maximizes the expected cumulative reward:
8
IE
Σ
* Rgc( 9, 8(+1)
II 1ss e T())]]
(1)
Sorepo go))
A common way to tackle the GCRL objective is through actor-critic methods. These algorithms learn a critic, which estimates a goal-conditioned action-value function Q(s, a,g), measuring the expected return of taking action in state while pursuing goal g, and then following the current policy. The actor, represented by the policy x(8,g) is updated to maximize the expected value given by the critic. Off-policy algorithms are particularly useful in GCRL, as they can reuse past experience across different goals. In practice, both the actor and the critic are trained using transitions (st, 9, ae, St+1,) sampled from a replay buffer, which stores past experience collected by the agent. In this article, we use Soft Actor-Crità (SAC) (Haarnoja et al. 2018), which augments the actor-critic Iramework with an entropy term to encourage exploration.
2.3 Hindsight relabeling
GCRL with sparse reward functions is difficult because, as long as the agent does not reach the goal, it does not collect any reward and does not learn anything. The Hindsight Experience Replay (HER) algorithm (Andrychowicz et al. 2018) is designed to mitigate this difficulty. During off-policy updates, where the agent learns from transitions {st. 9, ae, Rge 9, s141), $441,, most of the time it does not receive a reward. HER modifies transitions by substituting the current goal with goal achieved later in the trajectory, resulting in more dense reward signal.
This is shown in the following notation, where Emar represents the last step of a given trajectory:
Roelt B1r at 8++1) > (St, ((8k), (8t, ((81), st+1), at, 8141), where ls E [t, (mar]-
(2)
2.4 Sequential planning in GCRL and myopic policies
When the agent's goal is too far from its current state, combining GCRL with HER is not enough. In that case, established approaches in the HRL literature often combine a GC-policy together with a planner. We formalize three main points. (1) How a planner outputs plans, (2) How plans are updated and (3) How combine planner with a GC-policy.
We define the planner as a function that outputs a sequence of intermediate goals. For i 2 1, let G :- G x G x ... G denote the set of all sequences of length i over G. We then define G+ == U1 G so that
i times
3

---

Published in Transactions on Machine Learning Research (March/200)
G+ is the set of all finite non-empty sequences of clements of G. The planner is thus mapping
plan : S G G+.
At the start of an episode, the planner provides a plan from so to the final goal fg = So plan( ʃg). We denote Bi (R) the k-th goal of the plan at step t. To reduce computational cost, the plan is not recomputed at every step. Instead, it is updated only when the first goal in the current sequence has been reached. Formally:
gr
if 81+1 $ $
B++1 fuext plan 18 Bt f9)
(3)
plaan(se11: otherwise.
For convenience, we refer to the first goal in the plan as the behavioral goal: bg. := gi (1)
To solve GCRL problem, a common approach is to combine planner with GC-policy. The GC-policy is trained to reach any goal from any starting state. It uses transitions sampled from the replay buffer (st.g, at, Tt, $t+1), where the goal is fixed for the episode, optimizing Mge (section 2.2p, often combined with HER. During environment interaction, however, the GC-policy is conditioned not on the final goal fg but on the behavioral goal provided by the planner: ag ~ T(5,, bg.). This ensures the policy is queried only on nearby goals, where it is most accurale, while progressing toward the final goal.
2.5 Following a single sequence of low dimensional goals
When the goal and state spaces are identical, each behavioral goal by specifies a unique larget state, so the policy can reach it independently from future goals. In contrast, when the goal space is defined as low-dimensional projection of the state space, bg can often be reached through multiple distinct state configurations. As shown in (Chenu et al., 2023). although all these configurations are valid to reach the current goal, they may differ in how compatible they are with subsequent goals. As result, the agent may produce suboptimal trajectories or even fail.
The STIL algorithm CCeen et al., 2023 is an imitation learning algorithm designed to deal with the case in which chaining two subsequent, goals can be an issue. In STH, the agent learns behavior from a single The problem is defined
demonstration that is split into a unique sequence of goals TG extended MDP Mas {Sata, A, Retil, P.y, Pu. TT Tg). The agent observes (st, bgt, in) Sat = = x G x N, where 69: G is the current goal the agent is targeting and is the index of the goal in sequence Tg. When the agent takes an action, it moves to a new state, where 8141 P(.|st, ax) and (441) follows:
b9e41. h41 (sr bgt, i.) = [ (1++1)
is +
if 8t-1 E Sig. otherwise.
As soon as the agent reaches goal, a goal switch is triggered: it simply switches to the next one. The agent is rewarded each time reaches its current goal with Rota(St, bge, st+1) = 1[$441 E Sbn. The low-level agent combines SAC and HER and maximizes the expected cumulative reward:
1
EE
Σ y Rata(St bgt, S141)
(N Nowteel]] II 1ss, T(T ==0
(4)
The above formulation has two advantages. First, when the agent aims for its current goal, it also trics to reach it in configuration that is valid for also reaching the rest of the sequence. Second, since the agent is also conditioned on the current goal index i,, it can differentiate intermediate goals from terminal goals, which helps appropriately handling terminal conditions and using goal relabeling.
3 Related Work
Our method enables GC-policies follow multiple soquences of low-dimensional goals toward any final goal. The focus of our work is the chaining problem: ensuring that each intermediate goal is reached in a way that remains compatible with completing the rest of the sequence.
4

---

Published in Transactions on Machine Learning Research (March/200)
We first situate method within hierarchical reinforcement learning (HRL), where what doing can be seen as focusing on learning a low-level policy given high-level plan. We then discuss how the chaining problem is addressed by methods that condition GC-policies on full Markov states, on final goals, or on single intermediate goals.
3.1 Hierarchical Reinforcement Learning
Hierarchical reinforcement learning provides a framework for decomposing complex tasks into simpler sub- tasks. popular approach to HRL the options framework (Sutton ct al. 1999; Bacon et al. 2017) which defines temporally extended actions called options, each consisting of policy, a termination condition and initiation set. A high-level policy selects an option, and the option's policy is executed until it ter- minates. However, the majority of option-based approaches do not explicitly use goals or leverage general goal-conditioned policies. Feudal methods (Dayan & Hinton 1992 Levy et al. 2019 learn a hicrarchy where a high-level policy proposes goals and a low-level GC-policy executes them. In graph-based methods (Fy- senbach et al., 2019), graph is constructed with nodes as states or goals and edges as feasible transitions; a shortest path algorithm then yields a sequence of goals, executed by conditioning the GC-policy on the first goal of the current plan. In our work, we also follow sequences of goals with a low-level GC-policy, as in feudal and graph-based methods, but condition the low-level policy two goals instead of one. We use fixed expert planner to abstract high-level planning and compare methods following sequences of goals assumed to be feasible.
3.2 Conditioning GC-policies on Markov states
When intermediate goals are specified as Markov states, the chaining problem is entircly solved by the high- level policy: the low-level policy only needs to reach each intermediate state, without considering the full sequence.
This is the case in HIRO (Nachum al. 2018) and LEAP (Nasiriany et al., 2019) where the intermediate goals are specified as Markov states, with the GC-policy iteratively conditioned on these states. More recently, Zadem et al. (024) proposed two-stage formulation: a first component selects region of the state space, and sccond component generates sequence of states that progressively guides the policy toward that region. However, learning a low-level policy that reliably reaches Markov states is challenging, As shown in Gehring et al. (2021) the dense reward used in HIRO for the low-level policy is dominated by the (z, v) position, such that the agent often learns to reach (z, y) positions while ignoring other state dimensions (e.g.: joint angles). This effectively reduces the problem to conditioning on low-dimensional goals, which can lead to ambiguity: a given goal may be achieved through multiple configurations, introducing chaining issues when composing goals. Moreover, in some practical settings, it is undesirable to specify the goal as Markov state (e.g., user may wish to specify only the final position of robot, not its velocities or sensor states). In our method, we avoid conditioning on Markov states, focusing instead on oin-dinnsiional goals while explicitly addressing the chaining problem.
3.3 Conditioning GC-policies on the final Goal
Another line work trains GC-policies to reach final goals directly, using intermediate goals only as an auxiliary learning signal. Though this sidesteps chaining issues during execution, the auxiliary objectives must be carcfully constructed to avoid suboptimal behaviors the Iearned low-level GC-policy.
In RIS (Chane-Sane et al. 2021 intermediate goals are part of auxiliary loss to constrain the GC-policy to perform the same action to reach distant goal as it would when targeting an intermediate goal along the same path. Since these intermediate goals represent the Markov state, chaining issucs are avoided. However, as previously stated, policies conditioned on the Markov states raise specific issues. In GSP (Lo et al., 2024) the authors propose to solve a high-level options SMDP which navigates between way-points. The value function of the high-level SMDP then used to define a dense, potential-based reward for training GC-policy that directly targets the final goal. This potential baase based reward guides the agent without changing the optimal policy of the original MDP [gg et al. 1999), ensuring that chaining problem arises
5

---

Published in Transactions on Machine Learning Research March/2020
from approximating the high-level value. Their approach, however, was primarily applied in tabular and discrete-action RL. whereas we target continuous states and actions problems.
3.4 Myopic GC-policies
Many approaches use GC-policies that depend on a single, low-dimensional goal, e.g. several graph-based (Huang et al. 2019; Lee et al., 2022; Kim et al., 2023). feudal (Levy et al. 2019 Zhang et al. 2022) and options-based [Baariia et al. 2021) methods. Such policies optimize only for reaching the current goal, without regard for whether the stale reached al an intermediate goal is well-suited for completing the rest of the trajectory. This assumption works well in environments like AntMaze (de Lazcano et al.) [024), where the stable body configuration of the ant makes the precise state reached at cach goal largely irrelevant. In contrast, for humanoid agent, reaching the correct spatial position may result in a body configuration that restricts its ability to reliably move farther in all directions. Rather than resolving chaining at the GC-policy level, feudal and graph-based approaches mitigate these issues by adjusting the plan to keep the remaining trajectory feasible given the capabilities of the GC-policy.
Foudal: In feudal methods, the high-level policy rewarded when it proposes a. sequence of goals that successfully guides the low-level GC-policy toward the final goal. Extensions typically constrain the high- level policy by requiring goals to be reachable within & steps with the current low-level policy (Levy et al,, 2019 Zhang et al. 2022). While this encourages feasible chaining of goals, it also restricts the planner: the planner can only propose sequences that are alrcady compatible with the current low-level skills, potentially excluding otherwise valid solutions.
Graph-based:
Graph-based methods construct a graph whose nodes correspond to states sampled from the replay buffer, with odges representing the cost of moving from the state stored in one node to the goal representation of another Huang et. al. (2019). This formulation implicitly assumes that achieving a agent may
goal corresponds to reaching the exact state saved in the target node. In practice however, reach the intended goal position through many possible states, some of which can make reaching subsequent goals harder to achieve. To mitigate this mismatch, methods such as Huang et al. (2019) frequently replan, adapting to the trajectory executed by the policy. PIG (Kim et al. 2023) introduces a self-imitation loss, similar to RIS, which encourages the policy to take the same action for a distant goal as for an intermediate goal along the path. Yet, this approach overlooks that the optimal action for an intermediate goal can depend on the subsequent goals. [Bonavaud et al. (2024) report high sample efficiency on AntMaze by combining pre-trained GC-policy with graph planning. This efficiency is possible under a clearly stated condition: transitions between any two states associated with the same goal are always feasible and costless. This assumption reduces the class of applicable environments and goal abstractions.
Options skill chaining: R-DSC [Baaaria et al. 2021 constructs a sequence of options to reach a final goal, where each option corresponds to executing a goal-conditioned policy. To execule a sequence, the first option is selected and its policy is conditioned on a goal that overlaps with the initiation set of the next option. Once the first option reaches its goal, the next option is executed in the same way, and so on, until the final goal is reached. R-DSC updates the initiation set of each option using a classifier that identifies the states from which the current option can reach its goal. It also adjusts the conditioning goal of the policy so that the goal includes the most promising states for subsequent chaining.
However, because each intra-option policy is rewarded simply for reaching a low-dimensional goal before termination, it is optimized to reach any state within the goal set, without regard for whether that state facilitates exccution of the subsequent option. Morcover, R-DSC constructs a sequence of options targeting a single final goal, whereas our approach is designed to reach any final goal in the goal space.
A common weakness of all myopic approaches is that the high-level mechanism must adapt to steer the low-level policy along the subset of trajectories that a myopic GC-policy can achieve, which corresponds to a restricted set of trajectories with respect to what a more informed low-level policy could make. In this work, we examine myopic GC-policies in settings that highlight the limitations of focusing exclusively on the current goal. We assume access to an expert planner that proposes sequences of feasible goals, yet myopic policies can still fail or behave sub-optimally. Our approach solves these sequences by explicitly considering future goals when deciding how to reach the current one.
6

---

Published in Transactions on Machine Learning Research Mlarch/2020)
3.5 Chaining low dimensional goals
White (0117) introduces a transition- based discount (1s, a, s,),,, that termination and discounting depend specific transitions, thereby decompasing single MDP into subtasks. With soft discount formulation, the value can be partially propagated across subtask boundaries, enabling the agent to account for rewards that occur beyond the current subtask.
Our work closer to Chenu et al. (2023), described in Section 2.5 where the agent is trained to imitate a single trajectory composed of a fixed sequence goals. Our approach extends the work of et al. (2023) the general HRL setting, where the agent can produce multiple possible sequences leading to different final goals.
4 Methods
In the following sections, we present the problem we address and our method for learning low-level policies that can follow arbitrary sequences of goals. We leverage insights from STIL to construct a specific sequential MDP called xGC-MDP, where the agent prepares to reach future goals while aiming at the current one.
Based on this MDP, we propose two agents: GCP-final, which prepares for the full sequence, and GCP-N, which prepares only for the next N goals.
4.1 Problem statement
We study the problem solving GC-MDPs, where an agent must learn a policy capable reaching any specified final goal from any starting state. In our setting, reaching the final goal framed as following a sequence of intermediate goals. To abstract away high-level planning, we assume access to an expert planner that provides these sequences of intermediate goals.
We focus on the case where goals are low-dimensional relative to the state space, which allows the agent to achieve the same goal through multiple configurations. We study how to design low-level policies following a sequence low-dimensional goals. In particular, we analyze the chaining issues that arise when GC-agents are ioned on a single goal and how an agent conditioned on two goals can achieve each goal a way that remains compatible with the rest of the sequence.
4.2 Sequential Goal-Conditions MDP Formulation
In this section, we propose a sequential MDP framework, where the agent is rewarded for each intermediate goal, integrates goal switches inside the transition function, correctly handles terminal transitions, and is compatible with HER relabeling.
To guide the agent through sequence of goals, we define a reward function that grants positive feedback not only for reaching the final goal but also for reaching each intermediate goal. In standard Mge; the reward is defined as Rge(St, (St, G, 8c+1) on goal y that is assumed to be fixed along the episode. In contrast, our setling explicitly models the fact that the next goal the agent must reach evolves over time. Consequently, we define the reward as
Rge bgt, S1+1) = 1[s E Sbge]
where the behavioral goal byt denotes the first goal of the current plan gt. Because the reward now depends on bgt, which changes over time, bg: must be included in the state, and its dynamics must. be described in the transition function of the MDP. the state is defined only as (st, bgt), then transitions take the form
(st: bgt) - (st+1, bg++1)
where bg++1 depends on the unobserved remaining goal sequence gr. This leads a Partially Observable Markov Decision Process.

---

Published in Transactions on Machine Learning Research (March/2026)
(1) GC-Agent: SAC+HER
(2) Myopic Agent: SAC+HER+SEQ
(3) GCP-final
(4) GCP-2
Fixed final goal and rolling behavioral goal
relling behavioral goal and next goal < Ag
(a) success
fallure
(a) unseen next goals Issue
turn
x(K.
(
xis.by
r(e. by
=
V(sogg
V(s0.g
V(so, bg
V440, =
.19
V(so, bg
86
Σ (g
Rgc(
Bgolor bg)
Σ 1m0
7 Rgn hgg)
Σ
Rgo
100
1-0
7"
7
Figure 1: In this didactic example, we compare goal-conditioned formulations a simple maze. Reaching the final goal terminal, policies are deterministic.
(1) A GC-agent conditioned only on the final goal (a) ignores intermediate goals and can, in principle, reach any goal, but (b) using approximate methods, it often fails long-horizon goals. (2) A GC-agent using the same loss in (1), but conditioned the next goal while interacting with the environment (a) cannot know in advance how to reach its current goal, it depends unknown future goals and (b) fails to reach the final goal as the value function ignores the evolution of the behavioral goal. V(so, bg = sums rewards only until the next goal is reached, since goal reaching is torminal. (3) With GCP-final agent, the behavioral goal evolves over time both during environment interactions and the learning objective. The value V(sg; by Sg
includes rewards for all future goals until fa, where the episode terminates upon reaching the final goal. (4) With GCP-2 agent, both behavioral and final goals evolve along the trajectory. The final goal is assumed fixed when learning, V(Su, bg fg conditioned on the next two goals, the agent completes the trajectory.
sums rewards up to the terminal blue goal at St,i by being iteratively
Markovian goal transitions can be preserved by defining augmented MDP with state (st, gr, ft)) where the agent's policy is conditioned on the complete goal sequence. In this formulation, the next sequence is updated as g++1 = fnext Slanxt_pan((_ (t, (see Equation 33) ensuring that transitions remain Markovian.
It is also possible to preserve the Markov property with lover-dimensional state by defining the augmented state as (st, bg.. fg). In this formulation, the next goal bgt+1 is implicitly determined from the state:
bgt+1 = fnext, (0a4(1141 bg., [g) fg) = (141, if 8441 E Sag.
bge
otherwise.
In order to properly formalize this process, we propose the following MDP formulation: xGC-MDP (Syeegs A, Rger P. Po. T). At the beginning the episode, the planner computes the sequence intermedi- ate goals up to the final goal fg. At each step, the agent observes (st. bg.. E Sgseg = S x G which is composed of 8t, behavioral goal byt that the agent must reach, and the final goal fg that stays fixed during the episode. When the agent takes action, moves to new state (5t+1, b9t+1.ff)) where st+1 P(-sst, az) fg). The reward function is defined as Ryo(Se, bg.. $141) = 1 Sbg.
and bg.+1 = fneat [oettt_ thus the agent is rewarded for each behavioral goal reached along the trajectory. The objective is to find a policy that maximizes the expected cumulative reward
1
E
Σ t=0
r Rgg (St, bge, S141)
I 1ss T(f9)) 1=0
(6)
Suropg P(tg)
bg + next goat (80, Sg)
8

---

Published in Transactions on Machine Learning Research March/2020
where T(fg) is the set of terminal states according to the final goal fg. The corresponding temporal difference target is given by:
(ss byt, fg. a) - (r st, bgr, 81+1) + 71/00+1 & T(fg) max ( b9c41,
As illustrated in Figure 1 the advantages of this formulation are the following. First, because the agent receives reward for each achieved goal, the optimal policy is the one that completes the entire sequence. Second, it enables proper handling of terminal states, as it can differentiate intermediate and final goals. With the problem framed as an MDP, we can directly use SAC to learn the policy.
Hindsight Relabeling: Since the agent is conditioned on two goals, each of them can be relabeled.
Given a sample (st: bg.. fg. s bgt+1); we can relabel bg: < o(sk,) and fg t- Φ(sk2) where kn min(t,, 12) and 42- max(t1, 12), with t2 sampled independently and uniformly from (t, fmax). Since xGC-MDP integrates successive goals in the transition dynamics, once the current goal is reached, the next goal must be updated to be the next one the planner would have selected to reach the final goal. When bg: and fg are relabeled, b94+1 is updated according to Equation ( Skipping this step would cause the agent to learn from transitions between goals that violate the transition function. The pseudocode is provided in Appendix A.1
4.3 Generalizing from final goal to N-th Goal Conditioning
To solve a GCRL problem with access to a planner, we first propose a method that directly optimizes the entire sequence of goals using the xGC-MDP. We then introduce variants that focus on limited horizon of future goals.
We denote GCP-Sinal the method that optimizes xGC-MDP using SAC and leverages our modified version of HER. The GCP-final policy is conditioned on the state (st, bgt. which specifies both the current intermodiate goal bg and the final goal fg. In order to maximize the objective function, the agent should reach bg in a configuration from which it can achieve the rest of the sequence.
In practice, achicving by in a way compatible with the full sequence may depend primarily on the first few upcoming goals rather than all goals up to fg. For example, in a car navigation task, if the next three goals cover an entire turn, at each moment the car can plan for just these three goals. By always preparing for short horizon of upcoming goals, it can handle a whole sequence of multiple turns without needing to plan for the entire route, similarly to how Model Predictive Control optimizes over a short horizon at each step rather than the full trajectory.
To examine this hypothesis, we introduce a variant, denoted GCP-N. This variant employs the same learning algorithm as GCP-final but differs in its action selection strategy during environment interaction. Rather than conditioning on the final goal, the policy is conditioned on the N-th goal in the sequence. More formally, the agent samples actions
((nn(N)e)(())))))
a₂ Tst, byc, gr
where g. is updated with Equation 3,, and len(g:) corresponds to the number of goals in the sequence ga-
Given that the GCP-N policy is learned on xGC-MDP, selects an action that optimizes reaching all intermediate goals up to the N-th one. For a GCP-N agent to achiove good performance, it must (1) learn policy that can always reach the next N goals, and (2) operate in a problem setting where optimizing only the next N goals is sufficient. Failure to satisfy (2) can occur in two ways: the agent may reach an intermediate goal in configuration that is not compatible with reaching the final goal leading to failure, or it may complete the full sequence but with suboptimal behaviors. In contrast, GCP-final does not require property (2), since it directly optimizes the full sequence. However, for GCP-final, property (1) may be harder to achieve. Shorter horizons improve value function estimation (Park et al., 2023). and during training the agent is more likely to observe transitions involving nearby goals than distant ones.
The pseudocode for GCP-final and GCP-N is provided in Appendix A.1
9

---

Published in Transactions on Machine Learning Research Mlarch/20200
5 Experiments
Our experimental study is designed to evaluate three types of agents: (i) SAC+HIER, an agent conditioned on the final goal without leveraging planning, (ii) SAC+IER++SQQ an agent conditioned iteratively on the successive goals provided by the expert planner, corresponding to the formulation adopted by most feudal and graph-based approaches (sce Section 2.40, and (iii) GCP-N: our agents conditioned on both the first
and N-th goals of the plan, as described in Section 4.3
All these agents rely on SAC+HER with the same seper-parameters (see Appendix A.3), conclusions are therefore only drawn under these configurations, without specific finc-tuning for cach agent.
We evaluate them in four environments with various goal chaining difficulties and various terminal conditions to highlight the advantages and limitations of the different approaches. All agents are trained on goals sampled uniformly from the goal space. Evaluations are performed on fixed set of challenging (start state, goal) pairs specified for each environment in Figure 2
Evaluation metrics reported using both figures and tables. Figures show the mean performance over the evaluation set during training, including success rate, number of rollout steps required to reach the goal, cumulative reward. The associated tables report, for the same runs, the final success rate and the number of training time steps required to reach a 99% success rate.
5.1 Environments
Dubins Hallway
Humanoid Evaluation
Humanoid Planning
GC-Cartpole
SnakeMaze5
Figure 2: Each environment shows the evaluation goals together with the expert graph used for planming. In Dubins Hallway, the agent is evaluated on five goals, starting in the main corridor and aiming for each goal shown as a red circle. GC-Cartpole, the agent starts at the bottom center and is evaluated on two goals, shown as red rectangles. In SnakeMaze5, the agent starts the green area and is evaluated only on the hardest goal, shown as red circle. In GC- Humaooddd Walk, the agent is evaluated on all 16 goals, shown as red circles. The starting position is always at the center of the arena, facing the current evaluation goal. In this environment, the planner does not use graph. Instead, it outputs uniformly spaced points between the agent and the final goal.
We evaluate the methods in four environments, each highlighting different goal-chaining challenges. In Dubins Hallway, ignoring future goals can lead to collisions with walls, resulting in failure. PointMaze and Cartpole illustrate how terminal and non-terminal conditions, respectively, can affect performance in sequences of goal following. Finally, GC-Humanoid- Walk tests scalability to high-dimensional action and state spaces, where each intermediate goal must. be reached in a sufficiently balanced configuration allow continued locomotion toward the final goal.
in 2D maze. The state {a, y, cos(O), (in) includes the position and orientation of the agent. Each goal g is defined position (z. y) and is considered reached if the agent is within ball of radius 0.1 centered at that position. The agent moves forward al fixed speed al each step, the action controls the variation of the orientation for the agent. During a training episode, both the initial position the agent and the goal are randomly sampled within the maze boundarics. During an evaluation episode, the agent is tested on fixed start state and multiple goal locations illustrated in Figure 2 If the agent hits a wall, it stays stuck until the episode ends. The state is only terminal when the agent reaches the goal. A key feature of this environment. is that, when
Dubins Hallway is navigation task in which the agent controls
10

---

Published in Transactions on Machine Learning Research (March/2026)
the agent is in the central hallway, it should not prepare for the next goals in the same way depending whether the final goal is on the left- or right-hand side. In contrast, the velocity of the agent being constant, the sime-to-goal does not vary much in this environment, so we do not present results on this aspect.
In GC-Cartpole the agent must reach a given position while balancing a pole. The state s = ( 0 O) contains velocity à; angle 0, and angular velocity Ó. As explained in more detail in Appendix A.2 the agent's goal is the difference between its current position I and a fixed target position Idg. The goal is reached once || - Eagll < 0.05. Actions are continuous values in [-1,1] proportional to the force applied to the cart. During training episode, the agent targets random uniform goal in the range [-5,5] 5] starting from [0, 0). During evaluation episode, the agent is tested on multiple goal locations illustrated in Figure 2 A state is terminal when 0 leaves the [-0.12", 0.12%] interval. As reaching a goal is not terminal, the agent must learn to reach its target position and stay there while keeping the pole balanced.
In MnzkeMaze5 (de Lazcano et al., 2024), the agent controls a ball navigating in a 2D maze. The state = {ar, y, :/} includes the position and velocity of the agent. Each goal g is defined as a (ar y) position and is considered reached if the agent is within a ball of radius 0.45 centered at that position. The action represents the linear force exerted on the ball in the x and y directions. In de Lazcano et al. (2024), the agent's 's velocity was limited to the range [-5,5] m/s. Within this range, the agent could easily take sharp turns at maximum speed, making the task relatively simple. To increase the difficulty, we extended this limit to [-10, 10] m requiring the agent to anticipate turns and decelerate in advance. During a training episode, both the agent's 's initial position and the goal are randomly sampled within the maze boundaries. During evaluation episode, the agent is tested on multiple goal locations illustrated in Figure 2 is only terminal when the agent reaches the goal.
The state
In CC-Humanoid- Walk, we adapted the original humanoid- 55 environment (Towers et al. 2024 to the goal-condite conciiondt case, The agent must learn to move toward target positions in an empty arena. The state s E R348 includes joint positions, velocities, inertial properties, center-of-mass velocities, actuator forces, and external forces on body parts. As in GC- Cartpol,,, goals are not given as (æ, y) absolute positions, but as relative differences between the current agent position and a target position. A goal is reached if [xtorso Storeo) go ygoat) < 0.1|I. When the z-coordinate of the torso (the height) is not in the closed interval [1, 2] the episode terminates and negative reward of -10 is given. During cach training episode, we sample a random uniform goal in the square (Ir, v) E [-5,5] 5] x [-5,5] 5]. During evaluation episode, the agent is tested on multiple goal locations illustrated in Figure 2 The humanoid agent is always initialized at the center of the environment, oriented to face the sampled goal. In this task, reaching the final goal does not terminate the episode; instead, the agent must learn to reach the target position and remain in equilibrium indefinitely.
Planning: In the CC-Humanoid-- Walk environment, the planner sets intermediate goals evenly spaced between the agent and the final goal. In all other environments, the planner computes the shortest path using the expert graphs shown in Figure?
5.2 Conditioning on increasingly distant goals
We study the impact of the index of the second goal N used to condition the agent policy (Section 4.3 where N is between 2 and the index of the final goa
Learning curves are shown in Figure 3 while Table reports the mean final success rate and the mean timestep at which a 99% success rate was reached. Learning a policy with smaller values of N yields com- parable performance with fewer environment interactions in SnakeMaze5 and Dubins Hollway and improved performance in GC-Carlpole and GC-Humanoid- Walk. In particular, the best performances sample effi- ciency are observed for N = 2, corresponding to the setting in which the agent is conditioned on the next two upcoming goals, while it tends to be the lowest for N = final.
The upper bound was determined empirically; beyond this value, collisions become unstable and the agent can occasionally pass through maze walls. 2GCP-1 would correspond conditioning the agent twice on the next goal, which similar SAC+HER+SFO, studied in Section 5.3 and Section 5.4
11

---

Published in Transactions on Machine Learning Research March/2028
Dubins-Hallway
SnakeMaze5
GC-Cartpole
GC-Walk-Humanod--5
Sucress rale
Success rate
0.75
Success
0.75
0.4
0.50
0.25
0.25
0.00
0.00
0.00
100
200
300
100
200
300
100
200
10
Env step (x10")
step (x10")
(x10')
step (x110)
GCP-2
GCP-3
GCP-4
GCP-final
Performance of GCP-N as function of N, the number of upcoming goals included in the planning horizon. Smaller N improves sample efficiency overall performance (statistical tests supporting this claim are reported in Table D Performance is reported as the mean success rate on the evaluation goal set defined in Figure 2 Each evaluation is performed 10 times, with results reported as the mean and 95% confidence interval (computed with bootstrapping) over 50 seeds. Stars denote statistically significant differences (Welch T-test, p-value < 0.05) between GCP-2 and the algorithm of the corresponding color.
Figure 3:
Table 1: Mean + standard deviation of the final success rale and the first timestep at which a 99% success rate (SR) was reached over 50 seeds. indicates statistically significant difference from GCP-2 (Welch T-test, p-value < 0.05). Blank entries indicate that there was at least one run in which 99% SR was not reached.
Env
Metric
GCP-2
GCP-3
GCP-4
GCP-final
Dubins
Final SR
1.0 + 0
1.0 + 0
1.0 + 0
1.0 + 0
99% SR at
63.4k + 11.9k
86.3k + 14.8k
120.3k
+ 30.2k
124.7k + 25.0k
SnakeMaze5
Final SR
1.0 + 0
1.0 +
1.0 + 0
0.94 + 0.22 *
99% SR at
20.7k + 4.9k
25.6k
+ 6.6k
26.9k
6.1k
49.7k
+ 30.0k
Cartpole
Final SR
0.99
+ 0.07
0.90
+ 0.25
0.77
+ 0.26
0.78
+ 0.32
99% SR at 53.0k
+ 13.4k
Humanoid
Final SR
0.46
t 0.17
0.18 + 0.16
0.18 + 0.17
0.00 + 0*
99% SR at
One challenge for GCP-Sinal is that the second goal used to condition the policy can be any goal in the goal space. This requires the policy to generalize over a much wider range of goals, which increases learning difficulty. By contrast, in GCP-2, the second goal corresponds to a restricted subset of goals (the second goals of any plan), rather than the entire goal space. Moreover, when HER is applied, the final goal is often replaced by goals that were uclla achieved in successful trajectories. On average, these substituted goals are closer to the current state of the agent, so training transitions contain fewer examples with distant goals, which are necessary for GCP-final but not GCP-2
The main advantage of GCP-final compared to GCP-2 is that GCP-final can prepare not only for the next two goals but also for all subsequent ones. This benefit depends on the ability to pack-propagate rewards from distant goals through the value function. If this propagation fails, due to limited generalization, bias from HER, or the difficulty of long- horizon credit assignment, the policy cannot use the additional information. To examine this point, we analyze the value functions learned by cach GCP-N variant.
For each GCP-N variant, we assess whether the learned value accurately estimates the expected return and correctly petro-propagates rewards from the first N goals.
Formally, we compare the learned value V (st, bgt, gi (1))) with the theoretical value V V* bg..g. (2)) of the current policy. Since SAC does not directly compute VyJ but only Q7 estimate:
p (st, bgt. gi ((1))) 27 (st, byc, ( (s6, bit, gi (())))-
12

---

Published in Transactions on Machine Learning Research (March/200) /0026)
(a) Trajectories
(b) Value evolution
VISI
1
20
v(s)
3
2
20 Trajectory Step
Trajectory Step
GCP-2 GCP-3
GCP-4
GCP-final
Single palicy rollout return
Figure 4: Qualitative analysis of GCP-N agents in Dubins Hallway. (a) Set of trajectories of trained agents (b) Value function and single policy rollout return over episodes matching trajectories in (a). GCP-2 shows the best sample efficiency. As N increases, the value estimates deviate more from the single policy rollout return.
In deterministic environments, with deterministic policies, a single policy rollout return and V coincide. as it is the deterministic counterpart of the unbiased Monte Carlo value estimate Sutton & Barto 11998 3
This policy rollout return is computed as
T (N)
MC(I,N = Σ
T
Figure [b) shows that the discrepancy between V and MC increases with N. This result suggests that the critic trained with SAC has only propagated the value from the first few goals of the sequence. According to this interpretation, since the agent only prepares for reaching the first few intermediate goals and mostly ignores the influence of more distant goals, conditioning the policy on the last goals in the plan does not provide a practical benefit.
Other factors might come into play, such as the fact that the agent accumulates fewer experience in reaching goals that are more distant. We further investigate the question in Appendix A.7
The key takeaway of these experiments is that GCP-2 consistently achieves equal superior performance by preparing for the first two goals. Thus, we focus on GCP-2 in subsequent experiments.
5.3 Comparison to baselines
In this section, compare success rates of our agent conditioned two goals with two approaches in which the policy is conditioned on single goal.
As Figure 5 and Table 2 show, SAC+HER consistently achieves high success rates in Dubins Hallway, where the time horizon is short (40 timesteps), and in GC-Cartpole, where the dynamics is simple. However, in environments such as SnakeMaaee5 or CC-Humanood- Walk, SAC+IER catastrophically fails, never reaching goals that are distant from the initial position.
In contrast, SAC+ SEQ shows the best sample efficiency in GC-Cartpole and SnakeMaze5. However, the resulting behaviors are not always optimal, as discussed in Section 5.4 In Dubins Hallway, insufficient anticipation of future goals can cause colliding with walls. In GC-Humanoid Walk, it can result in loss of balance (Figure G This illustrates that this method fails in environments where preparing for future goals is critical.
Our method, GCP-2, consistently achieves the highest or comparable success rates in all environments.
ASAC employs stochastic policy; however, stochasticity vanishes the end training across repeated rollouts.
13

---

Published in Transactions on Machine Learning Research March/2022
Dubins: Hallway
SnakeMaze5
GC-Cartpole
GC-Walk-Humanodd-5
1.00
Success rale
Success rete
rate
0.75
0.75
0.75
0.50
0.50
0.50
0.2
0.25
0.25
0.25
0.00
100
200
300
100
200
300
100
200
10
step (x10*)
step (x10%)
(x10")
(x119)
SAC+HER
+
GCP-2
Figure 5: Success rates. SAC+ achieves the best sample efficiency in GC-Cartpole, but fails in GC- (C--Humnoidd- Walk and Dubins Hallway. In contrast, GCP-2 attains similar better success rates across all environments (statistical tests supporting this claim are reported in Table 2). Results are reported as the mean success rate on the evaluation goal set defined in Figure 2 Each evaluation is performed 10 times. Shaded regions indicate the 95% confidence interval (computed with bootstrapping) over 50 seeds. Stars denote statistically significant differences (Welch T-test, p-value < 0.05) between GCP-2 and the algorithu of the corresponding color.
(b) SAC+HER+SEO
t=0
18
t=30
t=
t=50
t=
t=
t=110
(a) GCP-2
t=0
t=18
t=30
t=40
t=50
t=70
t=90
t=110
Figure 6: Trajectorics of fully trained GC-Humanoid- Walk agents. (a) HAR SEQ reaches the first goal in a way that prevents completing the full sequence (e.g., taking an overly wide step). (b) GCP-N reaches the first goal in a configuration compatible with the second, enabling the chaining of multiple goals.
Table 2: Mean + standard deviation of the final success rate and the first timestep at which 99% success rate (SR) was reached over 50 seeds. indicates statistically significant difference from GCP-2 (Welch T-test, p-value < 0.05). Blank entries indicate that there was at least one run in which 99% SR was not reached.
Env
Metric
GCP-2
SAC+HER
SAC+HER+SEQ
Dubins
Final SR
1.0 + 0 11.9k
0.86 + 0.16
0.20 +
99% SR
63.4k
Maze5
Final SR
1.0 + 0
0.02 + 0.14
0.98
+ 0.07
99% SR at
20.7k + 4.9k
31.0k 9.6k
Cartpole
Final SR
0.99
0.07
0.93 + 0.17
0.99
+ 0.07
99% SR at
53.0k = 13.4k
100.3k
+ 30.6k
23.6k
+ 5.0k
Humanoid
Final SR
0.46 = 0.17
0.00 + 0*
0.00 + 0.00
99% SR
14

---

Published in Transactions on Machine Learning Research Alarch/20200
5.4 Effects of terminal conditions
Terminal conditions play an important role GC-RL and can change the nature of the task. We consider two cases. (i) In environments with terminal goals, reaching the goal terminates the episode. In that case, the optimal behavior to reach the goal as quickly as possible. (ii) When goals are non-terminal, reaching them does not terminate the episode. In that case, the optimal behavior is to reach the goal and remain in equilibrium, since this yiclds an infinite sum of discounted positive rewards.
(a) Trajectories in SnakeMaze5
(b) SnakeMaze5
(c) GC-Cartpole
Episode return
400
Time To Goal
500
200
400
0
100
200
300
0
100
200
Env step (x10²)
Env step (x10²)
(d) Trajectory velocities in MnakeMaze5
(e) Trajectory velocíties in GC-Cartpole
10
2.5
0
0.0
100
200
300
400
500
100
200
300
400
500
600
AA
10
2.5
0.0
100
200
300
400
500
100
200
300
400
500
600
10
2.5
0
0.0
100
200
300
400
500
100
200
300
400
500
600
rollout steps
rollout steps
HER+
SAC+HER+SEO
GCP-2
Figure 7: Analysis of behaviors in Snake.aazc5 and GC- CC-Cartpole, whose goals are terminal and non-terminal respectively. The evaluation goal sets defined in Figure 2 (a) Some trajectories generated by agents trained in Snake:Maze5. (b) Mean time to goal in Snake.Maze.. (c) Mean episode return in CCCCarllple. (d, e) Velocities of trajectories generated by agents trained in SnakeMaze5 and GC-Cortpole. For (b,c), results are reported as the mean and 95% confidence interval (computed with bootstrapping) over 50 sceds. Stars denole statistiaalyy significant differences (Welch T-test, p-value < 0.05) between GCP-2 and the algorithm of the corresponding color. SAC+ HER-HER--SEE converges a suboptimal behavior compared to GCP-2 both environments (statistical tests supporting this claim reported in Table 2). GCP-2 manages intermediate goals with more stable velocity.
Table 3: Mean standard deviation over 50 seeds. For SnakeMazo5, we report final time goal (lower is better). For Cartpole, we report episode return (higher is better). indicates a statistically significant difference from GCP-2 (Welch's t-test., p < 0.05).
Eav
Metric
GCP-2 SAC+HER
SAC+
SnakeMaze5
time to goal 4 334.9 :: 16.7 546.1 m.*
510.0 23.2"
Cartpole
return f
406.8
383.1 r2
307.1 25.0*
Figures d e) illustrate the velocities of agents in SnakeMaze5 and GC- CC-Cartpole. In these figures, black vertical dotted lines correspond to time steps where the agent reaches intermediate goal.
15

---

Published in Transactions on Machine Learning Research (March/200)
In SnakeMaze5, SAC+ reaches goals at high speed (Figure (d)). However, this behavior makes
it difficult to turn, resulting in bouncing off walls (Figure 7 (a)). As shown in Figure 4b) and Table 3 the
time the agent needs to reach the final goal is longer than with GCP-2 because the SAC+IER+SEQ agent does not distinguish intermodiate from final goals.
In GC-Cartpole (Figure 7 (e)), the HAC+ agent decelerates before each goal to ensure it can stabilize there. This behavior results in overall slower movement compared to other methods (Figure c), Table 3
In contrast, HAC+ which ignores intermediate goals, obtains a high return in GC- GCrCare but frequently fails in Snake:Maze,, as the agent struggles to reach distant goals. Finally, GCP-2 performs equally or better in both environments, with more constant velocity in SnakeMaze5 and GC-Cartpol (Figure dd. e)). Conditioning on the two upcoming goals allows the agent to anticipate future turns in Snzke and to cross intermediate goals in GC- GC-Cart Cartplle with only minimal deceleration.
5.5
HER ablation
(a) GCP-2
Dubins-Hallway
SnakeMaze5
GC-Cartpole
GC-Walk-Humanodd-5
Success rate
1.0
Success rate
1.0
Success rate
1.0
Success rate
0.50
$ pbrrrree
0.5
0.5
0.5
0.25
0.0
0.0
0.0
0.00
100
200
300
100
200
300
100
200
10
Env step (x10²)
Env step (x10²)
Env step (x10²)
Env step (x105)
(b) GCP-final
Dubins- Hallwayy
SnakeMaze5
GC-Carppole
Success rate
1.0
1.0
Success rate
1.0
Success rate
0.010
Success
0.5
0.5
0.5
0.005
0.0
0.0
0.0
0.000
100
200
300
100
200
300
0
100
200
5
10
Env step (x 10
Env step (x10²)
Env step (x10²)
Env step (x10%)
GCP-final
GCP-2
bg relabel
relabel
Figure 8: HER ablation, showing that suecess rate is generally higher with HER than without for GCP-2 (top row) and GCP-final (bottom row), statistical tests supporting this claim are reported in Table 4 Results are reported as the mean succoss rate on the evaluation goal set defined in Figure Each evaluation is performed 10 times, with results reported as the mean and 95% confidence interval (computed with bootstrapping) over 50 seeds. Stars denote statistically significant differences (Welch T-test, p-value < 0.05) between GCP-2 (top row) or GCP-final (bottom row) and the algorithm of the corresponding color.
We evaluate the role of HER by examining the effect of relabeling both the behavioral goal and the final goal in the GCP-2 and GCP-final settings across all previously considered environments.
Figure S and Table show that, in most environments, omitting the relabeling of either the behavioral goal or the final goal leads to a drop of success rate or sample efficiency. In particular, HER proves to be more effective GC-Humanoid- Walk, where relabeling considcrably improves the final success rate of the GCP-2 method. An exception is observed in GC- CC-Cartpole, where the impact of relabeling is low with the GCP-2 method and results in small drop of success rate for GCP-final.
16

---

Published in Transactions on Machine Learning Research (March/2022
Table 4: Mean + standard deviation of the final success rate and the first timestep at which a 99% success rate (SR) was reached over 50 scods. * indicates a statistically significant difference (Welch T-test, p-value < 0.05) between GCP-2 or GCP-final and their respective ablations. Blank entries indicate that there was at least one run in which 99% SR was not reached.
GCP-2
GCP-2 w/o fg
GCP-final GCP-final w/o bg
GCP-final
Env
Metric
GCP-2
w/o bg
w fg
Dubins
Final SR
1.0 o
0.86- 0.10"
0.65 0.25"
1.0 +
0.75 0.17
0.23 0.20
99% SR at
63.4k 11.9k
124.7k 26.02
SuakeMaze5
Final SR
1.0 + o
0.97 0.19
0.94 + 0.21
1.0 +
0.98 + 0.08
0.14 0.30
99% SR at
20.7k a 4.9k
66.7k 14.3**
46.9k 15.0k"
31.2k 6.8k
89.5k 25.2k*
Cartpole
Final SR
0.99 + 0.07
0.94 = 0.19
0.97
4 0.18
0.78 +: 0.32
0.73 = 0.35"
0.92 = 0.21
99% SR at 53.0k + 13.4k
74.9k + 22.4k
Humanoid
Final SR
0.46 0.17
0.00 0.00"
0.11 + 0.10"
0.00
0.00 0.00'
0.00 0.00
99% SR at
6 Conclusion
In this paper, we have studied the problem of following diverse sequences of low-dimensional goals from a given plan. We showed that strategics relying only on the immodiate next goal can lcad to failure cases, To address this, we proposed new MDP formulation, in which we formalize goal transitions and terminal states to ensure that the objective is to reach all goals in the sequence. Within this framework, the GCP-N method, where the agent prepares to reach up to N goals. Through navigation, pole-balancing and locomotion experiments, we have shown that in most environments, agents trained with GCP-2 are more sample-efficiet, or achieve higher performance than non sequential, myopic and GCP-N agents with a larger N.
proposed
A first limitation of our approach is that GCP-N variants consider only the first N goals in the sequence. Theoretically, this means that the optimal policy of the corresponding problem can be suboptimal compared to directly targeting the final goal optimizing over the whole sequence, such as GCP-final. However, in practice, particularly in the challenging GC-Humanoid- Walk environment, learned policies are far from their theoretical optimum, and we showed that restricting optimization to the first few goals facilitates learning and producos empirically strong performance. An interesting direction for future work would be to exhibit environments where using a larger N is mandatory and to make N adaptive to the complexity of the sequence, so that the agent. prepares further ahead only when necessary.
In this work, we used fixed oracle planner, which allowed us to focus analyzing the low-level lcarning of GC-policies for goal sequences. Our method would still work with a learned planner, provided that the planner is trained first and provides good enough intermediate goals, before learning the GC-policy. We leave the case where intermediate goals are misleading for future work. A second limitation of our approach arises when GC-policies and the planner are learned in parallel.
If the planner evolves over time, the low-level MDP can become non-stationary, depending on how the behavioral goal dynamics are defined. When the behavioral goal is fixed throughout the episode (as in HER+ the dynamics remain stationary. In contrast, if the behavioral goal switches to the next goal provided by the planner (as in GCP-N), each change in the planner modifies the low-level MDP and can introduce learning instabilities. However, the less the agent prepares for future goals, the fewer planner- dependent behavioral goal transitions. An extreme case occurs when the agent is conditioned on the next two goals of a plan: in this case, the evolution of behavioral goals becomes cases, a GCP-N agent conditioned on π ((.., bg A, = B) can only switch to T(.., bg = B, fg = B) once the goal A has been reached. Consequently, GCP-N with smaller N is likely to be more stable. Exploring strategies for the simultaneous learning of both components is left for future work.
indeeer-ndeeenddent In such
17

---

Published in Transactions on Machine Learning Research Alarch/20200
Broader Impact Statement
As any work in reinforcement learning, our work may be relevant for applications robotics, navigation, or other sequential decision-making lasks. One must be aware that policies learned through reinforcement learning can behave unpredictably in novel situations, which can be problematic in safety-critical or high- stakes contexts, highlighting the importance of carcful evaluation and human oversight. Most importantly, these potential issues are not specific to our work.
Reproducibility
The code for this study is publicly taleetttaioo-tlllo-----o--
available at httpes//gtaab.ccooovvvreerrrrrr
Acknowledgments
This was performed using HPC resources from GENCI-IDRIS (Grant 2022-AD0110159611 Tt also received funding from the European Commission's s Horizon Europe Framework Program under grant agreement No 101070381 (PILLAR-robots project). The authors also thank Alexandre Chenu for helpful discussions.
References
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Picter Abbeel, and Wojciech Zaremba. Hindsight Experience Replay, February 2018. URL arXiv:1707.01495 [cs].
http://arxiv www.:rrg//aaoo / hatpb///17700110
Lierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the Thirly- First AAAI Conference on Artificial Intelligence, AAAF17, pp. 1726 1734, San Francisco, California, USA, February 2017. AAAI Press.
Akhil Bagaria, Jason Senthil, Matthew Slivinski, and George Konidaris. Robustly learning composable options in deep reinforcement learning. In Zhi-Hua Zhou (ed.), Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pp. 2161-2169. International Joint Conferences on URL https://doi. org/10.
Artificial Intelligence Organization, 2021. doi: 10.24963 11///////cai../. 2021/298...
24963/1jcai. 2081/298. Main Track.
Hedwin Bonnavand, Alexandre Albore, and Emmanuel Rachelson. Learning state reachability graph in translation invariant goal-based reinforcement learning tasks. Transactions Machine Learning Research, 2024. ISSN 2835-8856. URL httpps:/ nee/forrumii--PPIPPPPg
Elliot Sanne Cordelia Schmid, and Ivan Laptev. Goal-Conditioned Reinforcement Learning with Imagined Subgoals. In Proceedings of the 38th International Conference on Machine Learning, pp. 1430- 1440. PMLR, July 2021. URL httpps://proceedings.nt mlr. http:////113//aaa-aaaaaa html ISSN: 2640-3498.
Xinyue Chen, Che Wang, Zijian Zhou, and Keith W. Ross. Randomized ensembled double q-learning: Learning fast without a model. In International Conference on Learning Representations, 2021. URL
https://openrevior net net/ a
Alexandre Chenu, Olivier Serris, Olivier Sigaud, and Nicolas Perrin-Gilbert. Leveraging Sequentiality in Reinforcement Learning from Single Demonstration, April 2023. URL http://aaxiv..om abs/ 22211 04786 arXiv:2211.04753 [cs].
Peter Dayan and Geoffrey E Hinton. Feudal reinforcement learning. Advances in neural information pro- cessing systems, 5, 1992.
Rodrigo de Lazcano, Kallinteris Andreas, Jun Jet Tai, Seungjae Ryan Lee, and Jordan Terry. Gymnasium robotics, 2024. URL
Robotics.
18

---

Published in Transactions on Machine Learning Research March/2024
Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. Search the Replay Buffer: Bridg- ing Planning and Reinforcement Learning, June 2019. URL http://arxiv.. wwwpaaiwwi......aaooooooooeoooot www.aaas......// 05253 arXiv:1906.05533 [cs].
Jonas Gehring, Gabriel Synnaeve, Andreas Krause, and Nicolas Usunier. Hierarchical skills for efficient exploration. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Newral Information Processing Systems, 2021. URL http:////ppppppoeee....o/o//raaa
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement, Learning with a Stochastic Actor. Tn Proceedings of the 35th International Conference on Machine Learning, pp. 1861 1870. PMLR, July 2018. URL httpp::/proceeeding.. mmr. press//80/haarrojal8Bb... html ISSN: 2640-3498.
Zhiao Huang, Fangchen Liu, and Hao Su. Mapping State Space using Landmarks for Universal Goal Reach- ing, August 2019. URL http:://axxx wrgg/bs////a/ 00t//99900.005511111 arXiv:1908.055: 0.055 [cs, stat].
Junsu Kim, Younggvo Seo, Sungsoo Ahn, Kyunghwan Son, and Jinwoo Shin. Imitating Graph-Basee Basee Planning with Goal-Conditioned Policies, March 2023. URL. http://arxiv.. http://xxxvv..og/aaa/ has///b///330000 11166. arXiv:230..11166 [cs].
Seungiae Lee, Jigang Kim, Inkyu Jang, and H Jin Kim. Dhrl: graph-based approach for long-horizon and sparse hierarchical reinforcement learning. Advances in Newrul Information Processing Systems, 35: 13668-13678 2022.
Andrew Levy, George Konidaris, Robert Platt, and Kate Saenko. Learning multi-level hierarchies with hindsight. In Proceedings of International Conference on Learning Representations, 2019.
Chunlok Lo, Kevin Roice, Parham Mohammad Panahi, Scott M. Jordan, Adam White, Gabor Mihucz, Farzane Aminmansour, and Martha White. Goal-space planning with subgoal models. Journal of Machine Learning Research, 25(330):1 57, 2024. URL http://jmlr.
wwwplrroggggppggea wwtpoorggg/////////oooo wtggpappers//5//4-0000
httm
Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine Data-Efficient Hierarchical Reinforcement Learning, October 2018 URL http:://araai..rggbbbb///// 08296 arXiv:1805.0822 082996
[cs,, stat].
Soroush Nasiriany, Vitchyr H. Pong, Steven Lin, and Sergey Levine. Planning with Coal-Conditioned Policies, November 2019. URL
banggabs//10 1911.9911111111 901.9111..851331 arXiv:1911.08453 [cs, stat].
Andrew Y. Ng, Daishi Harada, and Stuart J. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Proceedings of the Sizteenth International Conference on Machine Learning, ICML '99, pp. 278 287, San Francisco, CA, USA, 1999. Morgan Kaufmann Publishers Inc. ISBN 1558606122.
Seohong Park, Dibya Ghosh, Benjamin Eysenbach, and Sergey Levine. HIQI: Offline goal-conditioned RL with latent states as actions. In Thirty-seventh Conference Neural Information Processing Systems, pp. 34866-34891, 2023. URL https:://ponreviiie wwt.:eeeeooorumnddcccttttttttto
Tom Schaul, Danicl Horgan, Karol Gregor, and David Silver. Universal Value Function Approximators. In Proceedings of the 32nd International Conference on Machine Learning, pp. 1312 1320. PMLR, June wtt.ggg........eeee////aaaaaaaaa
2015. URL htttps://proceeednngg mlr. www.ppressvvvi wwwppp/sssssssseeepe wttps///////chaal11 html ISSN: 1938-7228.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. Cambridge, MA, 1998.
The MIT Press,
Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-1):111-211, 1999.
Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. Gymnasium: A standard interface for reinforcement learning environments. Xi preprint arXin:247..1 17032, 2024.
19

---

Published in Transactions on Machine Learning Research (March/200)
Martha White. Unifying task specification in reinforcement learning. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 3742 3750. PMLR, 06 11 Aug 2017. URL https://proceedding: mlr.
press/v70/hhitel1 html
Mehdi Zadem, Sergio Mover, and Sao Mai Nguyen. Reconciling spatial and temporal abstractions for goal representation. In The Twelfth International Conference on Learning Representations, 2024. URL
https://openrevieet
http:eerrwiieeeeetoooourrreeieee net/fforuunidioodsPPPP1
Tianren Zhang, Shangqi Guo, Tian Tan, Xiaolin Hu, and Feng Chen. Adjacency constraint for efficient hicrar- arXiv:2111.000
chical reinforcement learning, August 2022. URL [cs].
A Appendix
A.1 Pseudo-code
Algorithm D provides the pscudocode for the main loop, illustrating how GCP-N and GCP-final operate within the environment using planner. Additionally, Algorithm 2 outlines the relabeling mechanism.
Algorithm Main Loop
Require: Q, n, planner, env, N, 6
RB + |
for N Nepisodes do
8, s,ewu &- env.resel()
path t 1g ,(() ...,, enuyl + plonner.path(s,, enty
bg + path[0]
if N + None then
fg - path [N- 1]
else
fg - envg
end if
done +- false
while not done do
+ T bg. fg)
s', term, trunc + env.step(a)
if E by then
path () gg ...,,,,,,,,, + planner.path(s enu,)
bg' + (0
if N + None then
fy path[N - 1]
end if
end if
RB t RB + [(s, a, bg. fg. s', bg', term)]
8, by = s ,b'
Q, Learn Step(Q, m, RB, planner, env. awv.terminal func, Φ)
done +- term V trunc
end while end for
A.2 Absolute and relative goals
Absolute goals are goals that correspond absolute positions in the goal space, while relative goals are goals whose positions relative to the agent. The relative goal formulation is better suited in GC- GC-Carlpole.
20

---

Published in Transactions on Machine Learning Research Mllrch/20200
Algorithm 2 Learn Step
Require: Q, n, RB, planner, terminal fune
Sample trajectory from RB
St, de, by,, Tt: fg: 81++1, byt+1, lermy Sample transition from relobel + (random(0.0, 1.0) < 0.8)
if relabel then
states + [s0, $1. Stmoa + Extract all states from the trajectory т I randomm(, (max)
t2 random(t, (t,t)a))
kn min(li, (2)
22 mar(ti,t2)
bg:, fg + ((states((a]) (sstates(ka))
end if
S1+1 bg: then
b9c+1 + planner, planner.path(s:: [0]
clse
bit+1 bg:
end if
4-1 if 8 E bg, clse 0
term, t- terminal ffnc(s,,nsss $1+1, fg)
transition = St, at, byt, Tt, 81+1, bgt+1 1, fg, term,
7,Q+ + SAC Upplate(transition...
return n.
Indeed, with an absolute goal formulation, going from z = 2 to x and from == to x = 10 are two different things, whereas with a relative goal formulation, the agent must only learn how go Frei solve both cases at once. Similarly, in CC-Humanoid- Walk, the goal is defined independently of the agent's absolute location: the agent is required to reach target (x,y) position specified in relative coordinates.
A.3 Hlpeerparameter
The lyper-parameters used for each environment are presented in Table 5 All baselines and methods
share the same arper-parameters, taken from the original SAC paper Haarnoja et al. (2018), except for SAC+HER directly targeting the final goal, where we use higher discount factor 0.995 for long horizon tasks (SnakeMaze5, GC- Carttaee, GC- (CC-Hmmnood- Walk). We only increased the neural network size for the most complex lasks, as Chenu al. (003)).
Table 5: hyper-parameters used for SAC
Hyper-parameters
Dubins Hallway
SnakeMaze5
GC-Cartpole
GC-Ilumanoid Walk
Random actions
5K
5K
5K
50k:
Critic hidden size
256,, 256
[512, 512]
[256, 256)
512,512,512)
Policy hidden size
225,, 256]
5122,122]
[256, 256]
512,512,512
Activation functions
ReLU
ReLU
ReLU
ReLU
Batch size
256
256
256
256
Discount factor
0.98
0.99
0.99
0.98
Critic lr
3 x 10-4
x 10-4
3 x 10
3 x 10-4
Policy Ir
3 x 10
3 x 10
3 x 10
3 x 10 4
temperature
5 x 10-3
5 x 10-3
5 x 10-3
5 x 10-3
HER Relabel
80%
80%
80%
80%
21

---

Published in Transactions on Machine Learning Research (March/2026)
(a) Tabular MDP
(b) GCP-N trajectriees
9.1
0.2
9.3
9.4
GCP-2 O-O
X-3
(0.0) ml4:Ag==1, 1,5o
(0, 1)
(1,
(2, 1)
y-0
(1,00
GCP-3
-b4 155
(0, 1) - (0, 2)
1,155)1 (1,
0.11
(2.1)
(0, 0)
215, (2, 21 (3,2)
02
B..
GCP-4
(0, 1(40
(U,1) stbulg-1 (0,2) 1(4949911199-1) (0,
bo 11115))) (1.3) 2( (2,3) 31 (3)3)
3
(1 (4,3)
Figure 9: An example of tabular MDP where only conditioning the agent the last goal can result in performing optimally. All agents conditioned on intermediate goals fail to discover the optimal sequence of actions.
A.4 Failure Modes of GCP-N Agents
In this section, we present a simple tabular task designed to illustrate a failure mode that occurs for GCP-N agents when N is lower than the index of the final goal (here the maximum index 4). The corresponding MDP is shown in Figure g States represent the agent position (x, y) so the state space is S C IR². The agent starts at state (0, 0) and can move deterministically along directed edges connecting cells. Additionally, wait action allows the agent to remain in its current state. The goal space contains only the z-component of the state. Consequently, the planner outputs a sequence of intermediate goals corresponding to monotonically increasing x- z-values. For instance, if the agent starts from (0, 0) and is tasked to reach x = k; the resulting plan is (k.2....,k). The reward function provides a value of +1 whenever an intermediate goal is reached, and 0 otherwise.
The trajectories executed by different GCP-N agents when pursuing the final goal are depicted in Figure g Looking at the trajectories for GCP-N agents up to N 4, see that only the GCP-4 agent can successfully reach the final goal.
For instance, with GCP-2, the planner provides the goal sequence (x: = 1, 2, = 4). Initially, the policy is conditioned on i(s = (0,0), bg = 1, = 2). Since the agent was trained on transitions in which the final goal (Jg) remains fixed throughout the episode, its optimal trajectory is the shortest path from (0, 0) to reach x 1, and then æ This corresponds to the path: [(0,0) (0, 1) - (1. + (2, 1)]. Following this policy, GCP-2 moves from (0, 0) to (0, 1), and then to 11. Upon reaching (1, which corresponds to the first planner-provided subgoal (x = 1), the agent updates its conditioning to i(ss (1, 1), 2, fg = 3). At this point, however, the only reachable states are (1, and (2, and from these states there is no available path to reach x = 3. This prevents the agent from completing the planned trajectory. The agent thus transitions to (2, 1), sceking the next intermodiate reward. Once it reaches 1), the agent updates its conditioning to (bg = 3, fy- = 4). However, from the state (2, 1), the only available action is the "wait" action, leaving the agent trapped in that state.
This specific example can casily be extended to any size to create failure cases for any GCP-N agent where N is smaller than the length of the planner's longest sequence of goals.
A.5 Asymptotic performance of GCP-N Agents
In this section, we study whether a longer training budget allows GCP-n variants with higher N achieve better performance.
22

---

Published in Transactions on Machine Learning Research March/2028
GC-Walk-Humanoid-s5
SnakeMaze5
SnakeMaze5
0.8
success rate
0.6
rate
0.8
to goal
500
0.6
0.4
400
0.4
0.2
300
0.0
0.0
0
Env step (x109
Env step (x109)
Env step (x10%)
GCP-2
GCP-3
GCP-final
Figure 10: Success rate of GCP-2 and GCP-3 in GC-Humanoid- Walk with 20 millions steps, and of GCP- 2 and GCP-final in SazkeMazeō with 10 millions stops. Each evaluation is performed 10 times, with results reported as the mean and 95% confidence interval (computed with bootstrapping) over 10 seeds. For SnakeMaze5, we also report time-to-goal, which better distinguishes agents (lower better). Stars denote statistically significant differences (Welch T-test, p-value < 0.05) between GCP-2 and the algorithm of the corresponding color.
As shown in Figure 10 in SnakeMaze5, GCP-final performs similarly to GCP-2, though with higher variance, without significant. statistical difference. To quantify this, we computed per-seed averages by taking the mean time to goal over the last 20 evaluation checkpoints for each of the 10 independent random seeds. This yielded 10 independent samples per method, each representing the final or asymplotic performance of single seed. Comparing these samples using Welch's T-test, which accounts for unequal variances, resulted in p-valuc of 0.73, indicating the absence of statistically significant difference between both methods.
In the case of GC-Humanoid Walk, 20 million steps are still insufficient to reach asymptotic behavior, and due to computational constraints, we could not train any further, In principle, GCP-final could outperform GCP-2 in scenarios where anticipating further into the future required. This is the case for instance in the GC-Humanoid- Walk task, where behaviors such as running efficiently loward distant goal may only emerge if the agent plans sufficiently ahead. However, in practice, we did not observe such cases: the learned policies likely remain far from optimal, and GCP-2 achieves overall better performance.
A.6 GCP-N success rate under higher UTD ratio
In this section, we evaluate whether increasing the update-to-data (UTD) ratio benefits GCP-N agents. As shown in Fi GCP-2 with UTD does not improve performance, and GCP-3 is negatively affected by the higher UTD ratio. Although higher UTD ratios can improve sample efficiency in some settings, this typically requires additional stabilizing modifications. For example, REDQ Chen et al. (2021 reports that large UTD ratios in SAC tend to amplify Q-value overestination and instability; they counteract this using an ensemble of Q-functions, and in-target minimization across a random subset of Q functions from the ensemble.
In our work, we focus on the standard SAC algorithm as the low-level agent. Since high-UTD methods are not the focus of our study, we do not investigate such extensions.
A.7 Evaluation of the value function during training
In this supplementary analysis, we qualitatively observe in Figure 12 the value function learned by GCP-N agents in Dubins Hallway using a single run. Our goal is to assess whether the learned critic Vi matches a Monte Carlo estimate of the true value of the current policy v along training, from a single start state sg.
Interestingly, the Monte Carlo return converges to higher value for policies conditioned on more distant goal, as it includes more rewards for intermediate goals.
23

---

Published in Transactions on Machine Learning Research Alarch/20200
GC-Walk-Humanoid-v5
GC-Walk-Humanoid-w5
0.6
0.3
Success rate
Success rate
0.4
0.2
0.2
0.1
0.0
0.0
5
10
0
5
10
Env step (x10%)
Env step (x10%)
GCP-2
GCP-2 (UTD=2)
GCP-3
GCP-3 (UTD=2)
Figure 11: Performance comparison of GCP-2 and GCP-3 in GC-HHmanoid Wolk for UTD values of 1 and 2. Each evaluation is performed 10 times, with results reported as the mean and 95% confidence interval (computed with bootstrapping) over 10 seeds. Stars denote statistically significant differences (Welch T-test, p-valuc < 0.05) between the compared algorithms.
4
4
3
S
2
150
150
300
300
Env step (x10²)
Env step (x10*)
Env step (x10')
Env step (x107)
GCP-2
GCP-3
GCP-4
GCP-final
Monte Carlo Returns
Figure 12: Evaluation of the value function learned by GCP-N agents in Dubins Hallway. V (so) denotes the critic's estimated value for the start state so along training, for evaluation goals shown the lower-left panel of Figure 2 At each evaluation step, we also report the mean of 10 Monte Carlo rollouts from 8o: which is a coarse estimate of the true value of the policy (so)..
Regarding the critic value, the initial portion of the curves corresponds to randomly initialized critic. As training progresses, smaller values of N lead to faster convergence of the critic toward an accurate estimate of the current performance of the policy, We see that GCP-final is slower to converge, even though the agent reliably achieves the next five planned goals after 150K steps.
24


Reviews:
[{"review_id": "DyzWfH8yPJ", "reviewer": "Reviewer_teVv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The primary contribution of this paper is a method for a goal-conditioned policy that, instead of considering a single subgoal, takes two subgoals as input: the next subgoal and a subgoal N steps in the future. The paper also presents a modification to hindsight experience replay to handle these two subgoal policies. \n\nThe paper's strength is that it offers a relatively simple approach to overcoming the limitations of a subgoal space that considers only position, thereby making it difficult for the agent to transition from one goal to the next. \n\nThe paper's weaknesses lie in its clarity and the lack of support for its claims. In terms of clarity, the problem area, assuming that the abstract planner cannot use a subgoal space that would enable a myopic goal-conditioned policy to perform well, is not well motivated; it is, however, an acceptable problem area. There are also numerous imprecise statements in the paper; some are listed in the claims section.  The writing also lacks conciseness and a clear flow of ideas that build on one another; it is at times repetitive. These writing issues can be fixed.", "claims_and_evidence_yn": "No", "claims_explanation": "Below is a list of claims I found in the paper and a statement regarding the sufficiency or lack of evidence. Overall, the claims made are not supported by direct evidence or are not scoped to the evidence provided. \n\n**Performance-related claims**:\n\nFor all performance-related claims, there are several key problems: 1) there is a lack of a clear metric used to make the claim and perform statistical analysis, judgments of superior or faster optimization are based on subjective observations of learning curves, 2) it is unclear what methods are used to compute confidence intervals and the number of trials is likely too low to be sufficient for rely on them, 3) there is no correction for multiple comparisons, 4) lastly all algorithms are run with the same hyperparameters, this means one algorithm will be better than another just do to the choice of hyperparameters. No discussion is given on how the hyperparameters are chosen or tuned and this process was not taken into account for the uncertainty of performance. The claims are not scoped to account for this narrow interpretation of performance. \n\n1. “in most cases, preparing to reach the next two goals improves stability and sample efficiency over all other approaches.”\n\n  Source: abstract\n\n  Issues: stability is undefined and undiscussed in the paper. Sample efficiency is also undefined. \n\n2. “Our results suggest that conditioning the policy on the two most immediate successive goals (N = 2) is\nthe most efficient variant. This approach outperforms both myopic agents and non-sequential agents.” \n\n  Source: end of introduction\n\n  Issues: same as given above. \n\n3. “As Figure 2 shows, smaller values of N tend to yield both higher sample efficiency and better performance\nin all environments tested. In particular, the best performance is observed for N = 2,”\n\n  Source: Section 5.2, paragraph 2. \n\n  Issue: same as above. \n\n\n4. “Our method, GCP-2, consistently achieves the highest or comparable success rates in all environments. It\nis the only method that achieves nontrivial success in the GC-Humanoid-Walk task, and it also achieves the\nbest sample efficiency in Dubins Hallway.”\n\nSource Section 5.3, last paragraph\n\nIssue: Same as above. What does nontrivial success mean? This is not a defined or precise term. \n\n\n\n**Claims in regard to conditioning on a future subgoal lead to optimizing actions for intermediate goals**:\n\nThe paper makes numerous claims suggesting that having a single future subgoal will lead to the selection of actions that are optimal for the intermediate subgoals. This claim does not make sense and is not supported by the evidence in the paper. Consider an agent in a 1D environment where the final goal is to go to the right, but the first half of the subgoals are all to the left. Conditioning on any goal that is to the right tells the agent nothing about how it should go to the left or when it should prepare to go to the right. If the agent were instead given a sequence of N subgoals, then it could make decisions to achieve all of them in sequence, but this is not what is proposed or what is discussed. \n\nThe claims listed below are taken from the paper and are stated without direct or sufficient evidence. \n\n5. “Thus, the policy must balance immediate progress toward bg with preserving the feasibility of future goals.”\n\n  Source: Section 4.3, paragraph 2. \n\n6. “Given that the GCP-N policy is optimized on xGC-MDP, it selects an action that optimizes reaching all intermediate goals up to the N-th one…N = final has the lowest performance and sample efficiency.“\n\n  Source: Section 4.3, second to last paragraph\n\n7. “only when using GCP-final the agent can prepare for the entire sequence of future goals.”\n\nSource: Section 5.2, third paragraph\n\n8. “In such cases, the actor primarily learns to select actions that prepare the agent for reaching the first few intermediate goals, while the influence of more distant goals becomes negligible.“\n\nSource: Section 5.2, last paragraph\n\n\n**Other claims**:\n\nThe following claims are made in the paper but are not covered by the two categories above. \n\n9. “However, the shorter horizon reduces sensitivity to compounding errors overtime”\n\nSource: Section 4.3, second to last paragraph\n\nIssue: What are the comounding errors? This is the only time this was discussed in the paper, and it is unclear what it is referencing or what evidence is used to make the claim\n\n10. “This indicates that the critic trained with sac has only propagated the value from the first few goals of the sequence instead of all goals up to the final one.”\n\nSource: Section 5.2, second to last paragraph\n\nIssue: This plot shows the difference in prediction error and does not directly show how value is propagated across subgoals. Thus, there are multiple ways this difference between the different values of N could have been observed. For example, the other methods (N=2,3) were able to reach the final goal faster, so they have had more time to learn the value function for a policy that goes through all the goals. This faster learning could just be due to beneficial hyperparameters or some factor other than value propagation, this this claim is unsupported. Looking at value estimates and propagation during learning would be interesting and a possible route to explain the behavior. However, this experiment alone is insufficient to draw such a conclusion. \n\nThere is another issue, where it is claimed that \"In deterministic environments, the Monte Carlo return and Vπ coincide,\" but this is only true if the policy is deterministic. In this case, the policy is not deterministic, and the value function cannot be evaluated from a single episode. Furthermore, if a deterministic policy was created from a stochastic policy, then the value function would have been trained on the data from the stochastic policy, and there is no reason to assume it should match unless the policy is deterministic. \n\n11. “The agent does not differentiate between intermediate and final goals, leading it to decelerate before each goal to ensure it can stabilize there.”\n\nSource: Section 5.4, paragraph 2.  \n\nIssue: This is likely true, but there is no evidence for this claim. It would make sense that this is the desired behavior, but this behavior needs to be measured. \n\n12. “In SnakeMaze5, sac+her+seq instead reaches goals at high speed. However, this behavior makes it difficult to turn, resulting in bouncing off walls (Figure 6 (a)). As shown in Figure 6(b), the time the agent needs to reach the final goal is longer than with GCP-2.”\n\nSource: Section 5.4 \n\nIssue: Similar to the other statement, there is no direct evidence that it reaches the goals at higher speeds. The paths shown in Figure 6 (a) do not show velocity. The paths shown in Figure 6 (a) would be evidence that conditioning on the second goal helps the policy over just having the first goal, but not sufficient on its own. This is because, even without any goal information, a good policy should be able to navigate to the chosen goal as quickly as possible. This suggests (i.e., needs investigation) that two or more subgoals used to select actions may influence the policy (change the objective surface) to prefer a smooth path between goals, rather than simply seeking to reach a single goal as quickly as possible. This could be a route to understanding Figure 6 (a), but it is not the only one. \n\n*Question: Were these paths smoothed in the plotting?* They look unusually smooth for an RL agent.", "audience_yn": "No", "audience_explanation": "I say it is not of interest due to the lack of support for its claims. Future versions of this work, with problems fixed, could be of interst to the community.", "requested_changes": "**Critical changes**: \n\n1. All claims are either properly supported or scoped to the available evidence. For many of these claims, it would likely require new experiments that look beyond simple performance metrics. If ALL claims currently in the paper were only scoped to the available evidence, the paper would no longer have interesting results. \n\n\n2. It is unclear why one would want to condition on a subgoal N>2 steps in the future and not include the subgoals between the current one and the Nth one. The value of N=2 is the only choice that actually chains together subgoals. It should be made clear why N>2 is even considered or could be beneficial. The stated reasons (as discussed above) do not follow. \n\n**Important but not critical**:\n\n3. It should be made clear why we are looking at planners that provide bad subgoals. For example, in the walker problem, the sequence of subgoals is just a series of vectors to move straight ahead. Why not have just a single goal for this problem to reach the final position? There are many examples (and I have personal experience) that make the humanoid move to different positions and require the agent to change directions. What does providing a single", "broader_impact_concerns": "No broader impact concerns", "additional_comments": "- It should be stated how $k_1$ and $k_2 are chosen in the method.", "extra_scores": {}, "extra_text": {}}, {"review_id": "kNj86hQnjq", "reviewer": "Reviewer_hR8T", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the problem of multi-goal conditioned reinforcement learning, which extends the goal-conditioned setting to the case where an agent must reach a sequence of goal locations in an MDP. The main contributions of the paper are:\n\n1) The observation that agents conditioned on a single goal may fail to ensure that each intermediate goal is reached in a way that remains compatible with reaching the rest of the sequence of goals;\n\n2) A generalization of goal-conditioned policies (GCP) in which a policy is conditioned not only on the next current goal, but also on the N-th goal from a provided sequence of goals given by a fixed planner; and\n\n3) A practical implementation of this method in an actor-critic framework that extends Hindsight Experience Replay (HER) to the multi-goal setting. The experimental results suggest that GCP-2, i.e., conditioning only the two most immediate goals, provides the best performance in the benchmarks.\n\nStrengths:\n- The idea is simple and easy to incorporate into existing goal-conditioned methods.\n\nWeaknesses:\n- Although the authors acknowledge the limitation of using a fixed oracle planner, this point is of high relevance and deserves a more in-depth discussion. Assuming a fixed planner is a significant assumption and may limit the method's applicability to the community.\n- The paper relies solely on empirical results, while some of the claims could also be backed up by theoretical/mathematical results.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The main claims of the paper are backed up by empirical results. As discussed below, the paper could be strengthened by a more mathematical/theoretical analysis that can better characterize the problem and the trade-off resulting from the method’s hyperparameter $N$.", "audience_yn": "Yes", "audience_explanation": "I believe this paper would gather interest due to the fact that goal-conditioned RL is a highly studied topic. Moreover, the proposed technique is straightforward to implement, and extending HER to multi-goal settings is also an interesting contribution.", "requested_changes": "- A smaller value of $N$ may lead to actions that are suboptimal with respect to the entire sequence of goals. However, the shorter horizon reduces sensitivity to compounding errors over time. It would significantly strengthen the paper if the authors could provide a more mathematical/theoretical justification of this claim, e.g., a novel theorem characterizing the trade-off of choosing a good value for $N$.\n\n- I suggest expanding the discussion on the limitations of using a fixed planner. What are the challenges associated with learning a planner simultaneously? Do you think this problem is tractable? If not, in which types of scenarios is it reasonable to assume access to an oracle goal planner?\n\n- Should the set $G$ be part of the definition of $M_{cg}$ in Section 2.2.?\n\n- In Fig. 2, why not include a standard GC policy (GCP-1) that is conditioned only on the current goal? Is it possible that it would also work well, showing that the benchmarks do not have the property being evaluated, which is “balance immediate progress toward bg with preserving the feasibility of future goals”?\n\n- In Section 5.2, it is not surprising that smaller values of $N$ lead to higher simple efficiency. However, it would be relevant to investigate if higher values of $N$ could lead to better asymptotic performance if trained for a sufficiently long time. For instance, in SnakeMaze5, could GCP-4 outperform GCP-2 in the long term? Would it be possible to construct a scenario in which it is strictly necessary to condition on $N>2$ goals?", "broader_impact_concerns": "NA.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "oaKhP4lJQK", "reviewer": "Reviewer_ii1i", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper tackles the chaining issue in sequential low-dimensional goal-conditioned RL: an agent, achieving an intermediate goal, may arrive in a state that is incompatible with the next goal. The paper introduces an augmented MDP framework that integrates the current and final goal into the state and the transition function, enabling the agent to prepare for subsequent goals while reaching the current goal. The paper provides a solution method, GCP-N, that combines off-policy actor-critic (SAC) with modified Hindsight Experience Replay (HER). Through HER, the critic is trained to estimate action values conditioned on a wide range of goals. The actor is conditioned on the N-th goal, corresponding to selecting an action that optimizes reaching all intermediate goals up to the N-th one, with the extreme case being the final goal (GCP-final). The paper presents experimental results that demonstrate the impact of N, the better performance of GCP-2 against other choices of N and baselines across various environments, and an ablation on HER.\n\nThis paper is clearly written with substantive evidence for the proposed method in the experiments. The investigation into the choice of N was useful for justifying GCP-N, and the empirical methodology was overall sound. The framework is a tad complicated, as is typical for planning and abstraction, but clearly explained. Overall, the problem statement is clear, and the proposed solution addresses the problem. The augmented MDP framework that incorporates two goals into the state to preserve the Markov property is interesting, and GCP-N is a useful algorithmic contribution.\n\nThere are a few points to address. \n1. SnakeMaze has 30 runs and the others have 10 runs. Using 10 runs for Gaussian CIs is a bit low. What was the reason for this difference, and could it be remedied?\n2. There is a small change to SnakeMaze: “In de Lazcano et al. (2024), the agent’s velocity was capped between the range [-5, 5] m/s; we updated this limit to [-10, 10] m/s. ” What is the reason for this change?\n3. Performance in Walk-Humanoid is poor for GCP-3 but not for GCP-2. The plots of value functions do help explain why this is the case (even though these plots are in Dubins Hallway). This might suggest that using more learning updates would be useful for GCP-3, GCP-4 and GCP-final. It might be worth testing this, seeing if values are more accurate and seeing if performance improves. If it does, then GCP-2 still has an advantage (computationally), but the advantage is more clearly a number of updates advantage rather than a learnability advantage. \n4. You give the hyperparameters in the appendix, and they are sensible choices. But how were they chosen? I am also curious how important it was to have different gamma across these problems.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims in the experiments section are generally well supported by the reported results.", "audience_yn": "Yes", "audience_explanation": "The practical problem being addressed and the proposed solution method in the paper may be of interest to those researching or applying sequential goal-conditional RL.", "requested_changes": "Minor comments:\n1. Pg 9, Fig 1: the font is small. \n2. Page 10: There is one sentence that is a paragraph, starting “This is unexpected…”, which should be part of the previous paragraph. As a note, this is not that unexpected, and this sentence slightly contradicts the purpose of introducing the Nth version (which should presumably be easier to learn than GCP-final). Maybe what is surprising is that final performance is also worse, but really, that is hard to judge without running for longer or accounting for the fact that learning is harder and so the agent may need additional tricks.\n3. In Section 3, it could be worth citing “Unifying task specification in reinforcement learning”, which uses transition-based gammas in an experiment in Taxi for an agent to learn an optimal policy between tasks, showing the importance of considering the next pick-up (goal) to get optimal policies. \n4. In Section 5.1, the body text and the caption under Fig 1 are inconsistent in describing how the start state and the goal are sampled for each env. For example, in SnakeMaze5, “the agent starts in the green area” is written in the caption, but “the agent’s initial position and the goal are randomly sampled within the maze boundaries” is written in the body text.", "broader_impact_concerns": "There are no concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)