Flattening Hierarchies with Policy Bootstrapping
Authors: John Luoyu Zhou
Jonathan Kao
Venue: NeurIPS 2025 spotlight ()
Keywords: reinforcement learning
goal-conditioned
hierarchical
offline
Primary area: reinforcement_learning
TL;DR: We propose an offline goal-conditioned RL algorithm that achieves state-of-the-art performance on complex, long-horizon tasks without needing hierarchical policies or generative subgoal models.

Abstract: Offline goal-conditioned reinforcement learning (GCRL) is a promising approach for pretraining generalist policies on large datasets of reward-free trajectories, akin to the self-supervised objectives used to train foundation models for computer vision and natural language processing. However, scaling GCRL to longer horizons remains challenging due to the combination of sparse rewards and discounting, which obscures the comparative advantages of primitive actions with respect to distant goals. Hierarchical RL methods achieve strong empirical results on long-horizon goal-reaching tasks, but their reliance on modular, timescale-specific policies and subgoal generation introduces significant additional complexity and hinders scaling to high-dimensional goal spaces. In this work, we introduce an algorithm to train a flat (non-hierarchical) goal-conditioned policy by bootstrapping on subgoal-conditioned policies with advantage-weighted importance sampling. Our approach eliminates the need for a generative model over the (sub)goal space, which we find is key for scaling to high-dimensional control in large state spaces. We further show that existing hierarchical and bootstrapping-based approaches correspond to specific design choices within our derivation. Across a comprehensive suite of state- and pixel-based locomotion and manipulation benchmarks, our method matches or surpasses state-of-the-art offline GCRL algorithms and scales to complex, long-horizon tasks where prior approaches fail. Project page: https://johnlyzhou.github.io/saw/

Flattening Hierarchies with Policy Bootstrapping
John L. Zhou University of California, Los Angeles john. ly. zhou@gmail com
Jonathan C. Kao University of California, Los Angeles kaoêseas wuw...lllee
Abstract
Offline goal-conditioned reinforcement learning (GCRL) is promising approach for pretraining generalist policies on large datasets of reward-free trajectories, akin the self-superrised objectives used to train foundation models for computer vision and natural language processing, However, scaling GCRL to longer horizons remains challenging due to the combination of sparse rewards and discounting, which obscures the comparative advantages of primitive actions with respect to distant goals. Hierarchical RL. methods achieve strong empirical results on long- horizon goal-reaching tasks, but their reliance on modular, limescale-speciffe policies and subgoal generation introduces significant additional complexity and hinders scaling to high-dimensional goal spaces. In this work, introduce an algorithm to train a flat (non-hierarchical) goal-conditioned policy by bootstrapping policies with advantage- weighted importance sampling. Our approach eliminates the need for a generative model over the (sub)goal space, which find is key for scaling to nigh-dimensional control in large state spaces. We further show that existing hierarchical and bootstrapping-boe -based approaches correspond specific design choices within our derivation. Across comprehen- sive suite of state- and pixel-based locomotion and manipulation benchmarks, our method matches surpasses state-of-the-ae of-thh-artt offline GCRL algorithms and scales to complex, long-horizon tasks where prior approaches fail.
loooooal-conditionn
Project page: http:///hmnlyzoouu.thuu....boo//
Introduction
Goal-conditioned reinforcement learning (GCRL) specifies tasks by desired outcomes, alleviating the burden of defining reward functions the state-space and enabling the training of general policies capable of achieving a wide range of goals. Offline GCRL extends this paradigm to existing datasets of reward-free trajectories, and has been likened the simple self-supervised objectives that have been successful in training foundation models for other areas of machine learning [58, 37]. However, the conceptual simplicity of offline GCRL belies practical challenges in learning accurate value functions and, consequently, effective policies for goals requiring complex, long-horiae horizon behaviors. These limitations call into question its applicability as a general and scalable objective for learning foundation policies [5, 39, 41] that can be efficiently adapted to diverse control tasks.
Hierarchical reinforcement learning (HRL) is commonly used to address these challenges and is particularly well-suited to the recursive subgoal structure goal-reaching tasks, where reaching distant goals entails first passing through intermediate subgoal states. Goal-conditioned HRL exploits this structure by learning hierarchy composed of multiple levels: one or more high-level policies, tasked with generating intermediate subgoals between the current state and the goal; and low- level actor, which operates over the primitive action space to achieve the assigned subgoals. These approaches have achieved state-of-the-s results both online [29, 33] and offline GCRL [36], and are especially effective in long-horizon tasks with sparse rewards.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

However, despite the strong empirical performance of HRL. it suffers from major limitations as scalable pretraining strategy, In particular, the modularity of hierarchical policy architectures, fixed to specific levels of temporal abstraction, precludes unified task representations and necessitates learning a generative model over the subgoal space to interface between policy levels. Learning to predict intermediate goals in a space that may be as high-dimensional as the raw observations poses a difficult generative modeling problem. To ensure that subgoals are physically realistic and reachable in the allotted time, previous work often implements additional processing and verification of proposed subgoals [62, 8, 19, 59]. An alternative is to instead predict in a compact learned latent subgoal space, but simultaneously optimizing subgoal representations and policies results in nonstationary input distribution to the low-level actor, which can slow and destabilize training [52, 29j. The choice of objective for learning such representations, ranging from autoregressive prediction [47, 60] to temporal metric learning [51, 35,31,11 31J, remains an open question and adds significant complexity to the design and tuning of hierarchical methods.
Following the tantalizing promise that flat, one-step policies can be optimal in fully observable, Markovian settings [42], this work aims to isolate the core advantages of hierarchies for offline GCRL and distill them into a training recipe for a single, unified one-step policy, We first conduct an empirical analysis on a state-of-the ther hierarchical method for offline GCRL that significantly outperforms previous approaches on a range of long- horizon goal-reaching tasks. Beyond the original explanation based on improved value function signal gegaalttonon noise ratio, we find that separately training a low-level policy on nearby subgoals improves sampling efficiency. We then reframe hierarchical policies as form of implicit test-time bootstrapping on subgoal- conditiood policies, revealing theoretical connection to earlier methods that learn subgoal generators and bootstrap directly from aubgoal-conditioned policies train a flat, unified goal-conditioned policy.
Building on these insights, we construct an inference problem over optimal subgoal states that unifies these ideas and yields Subgoal Sovantage-Weightee Policy Bootstrapping (SAW), a novel policy extraction objective for offline GCRL. SAW uses advantage-weighte importance sampling to bootstrap on in-trajectory waypoint states, capturing the long-horizon strengths of hierarchies in a single, flat policy without requiring a generative subgoal model. In evaluations across 20 state- and pixel- based offline GCRL datasets, our method matches or surpasses all baselines in diverse locomotion and manipulation tasks and scales especially well to complex, long-horizon tasks, being the only existing approach achieve nontrivial success in the humano iomanoidmaze dmaze- -giantt environment.
2 Related Work
Our work builds on a rich body of literature encompassing goal-conditioned RL [23], offline RL [25, 28), and hierarchical RL [9, 50, 22, 4, 52]. The generality of the GCRL formulation enables powerful self-supervised training strategies such as hindsight relabeling [2, 16] and state occupancy matching [30, 13. 63, 48]. These are often combined with approaches that exploit the recursive subgoal structure of GCRL: either implicitly by enforcing quasimetric structure on the goal-conditioned value function [56]. or explicitly through hierarchical decomposition into subgoals [33, 29, 17, 36). Despite these advances, lcarning remains difficult for distant goals due to sparse rewards and discounting over time. Many methods rely on the key insight that actions which are effective for reaching an intermediate subgoal between the current state and the goal are also effective for reaching the final goal. Such subgoals are typically selected via planning [21, 61, 18], searching within the replay buffer [12), or, most commonly, sampling from generative models. Hierarchical methods in particular generate subgoals during inference and use them to query "subpolicies" trained on shorter-horizon goals, which are generally easier to learn [49, 3]. Our method also leverages the ease of training subpolicies to effectively learn long-horizon behaviors, but aims to learn a flat, unified policy while avoiding the complexity of training generative models to synthesize new subgoals.
Our work is most closely related to Reinforcement learning with Imagined Subgoals [7, RIS], which, to our knowledge, is the only prior work that performs bootstrapping on policies, albeit in the online setting. Similar to coal-conditioned hierarchies, RIS learns a generative model to synthesize "imagined" subgoals that lie between the current state and the goal. Unlike HRL approaches, however, it regresses the coll-ooal-ond policy towards the subgoal-conditioned target, treating the latter as a prior to guide learning and exploration [Figure 1]. While RIS yields a flat policy for inference, still requires the full complexity of a hierarchical policy, including a generative model over the goal space. In contrast, work extends the core benefits of subgoal-based
Policy bootstrapping:
2

---

(a) HIQL
(b) RIS
(c) SAW
Figure 1: Learning with subgoals. Both HIQL and RIS "imagine" subgoals (thought bubbles) en route to the goal (red star) with generative models. However, HIQL samples actions directly from the subgoal-conditioned policy, while RIS regresses (black arrow) flat goal-conditioned policy towards the subgoal-conditionn uses "real" subgoals from the dataset DD weighting the regression more heavily towards distributions conditioned on good subgoals and less (gray arrow) towards bad ones.
action distribution during training. SAW also performs regression but only
bootstrapping to offtine GCRL with an advantage- from dataset trajectories, eliminating the need for subgoal generator altogether.
based based importance weight on subgoals sampled
3 Preliminaries
Problem setting: We consider the problem of offline goal-conditioned RL, described by a Markov decision process (MDP) M (S, A, R. P) where is the state space, A the action space, R. x S - IR the goal-conditioned reward function (where we assume that the goal space G is equivalent to the state space S), and P x A - S the transition function. In the offline setting, we are given a dataset 2 of trajectories T = (so, 4g, $1, Ai...., ST) previously collected by some arbitrary policy (or multiple policies), and must learn policy that can reach a specified goal state from an initial state E S without further interaction in the environment, maximizing the objective
( = EE (Bggppp))))
Σ r'r (st.g)
where p(g) is the goal distribution and yd (T) is the distribution of trajectories generated by the policy and the transition function P during (online) evaluation.
Offline value learning: We use a goal-conditioned, action-free variant implicit Q-learning [24, IQL] referred to as goal-conditioned implicit value learning [36, GCIVLJ. The original IQL formula- tion modifies standard value iteration for offline RL by replacing the max operator with an expectile actions. GCIVL replaces the critic (s,, a. g) with an action-free estimator ((s, g) = r(s.g) + >V (s',g), minimizing LGCIVL(V) = -x.a.g=p" [e (r(s,g) + + (s', V (+,
regression, in order to avoid value overestimation for out-of
(2)
where (x)
1xr < 0)(2 is the expectile loss parameterized by E [0.5, 1) and V(-)) denotes a target value function. Note that using the action-free estimator with expectile regression is optimistically biased in stochastic environments, since it directly regresses towards high-value transitions without using Q-values to marginalize over enviromment stochasticity.
Offline policy extraction: To learn target subpolicy, we use Advantage-Weighted Regression [40, AWR] extract a policy from learned value function. AWR reweights state-action pairs according to their exponentiated advantage with an inverse temperature hyperparameter a, maximizing
JAWR(T) = E.aaggD
a ()( V(saa)) log (a s, s.9)
and thus remaining within the support of the data without requiring an additional behavior cloning penalty, As with GCIVL, the frteon-free estimate Q(s, a, g) compute the advantage.

---

Understanding Hierarchies in Offline GCRL
In this section, we seek to identify the core reasons behind the empirical success of hierarchies in offline GCRL that can be used guide the design of training objective for a simpler, flat policy. We first review previous explanations for the benefits of HRL and propose an initial algorithm that seeks to capture these benefits a flat policy, but find that still fails to close the performance gap to state-of-the art method. We then identify an additional practical benefit of hierarchical training schemes and show how HIQI. exploits this from a policy bootstrapping perspective.
4.1 Hierarchies in online and offline GCRL
Previous investigations into the benefits of hierarchical RL in the online setting attribute their success to improved exploration [22] and training value functions with multi-step rewards [34]. They demonstrate that augmenting non-hierarchical agents in this manner can largely close the performance gap to hierarchical policies. However, the superior performance of hierarchical methods in the offline GCRL setting, where there is exploration, calls this conventional wisdom into question.
Our empirical investigation focuses on Hierarchical Implicit Q-Learning [36, HIQI-,, a simple yet effective hierarchical method for offline GCRL that learns a high-level policy over subgoals and low-level policy over primitive actions from single goal-conditioned value function [Figure 1]
Juh (0n) [e ((on.e4)) exp (Bh. . (sL, (er.8))))))))))))))))))))))))( log Thn (#L+* st.99]
J (0e) ((s
exp (8 AA' (5r, aes S(+x)) log Téé (at ses ((tR)
(4)
where 4h and f are inverse temperature hyperparameters and Ā is an approximation of the advantage function described further in Appendix F.1. Importantly, the value function is trained with standard one-step cemporal-diffreene
(TD) learning (GCIVL) instead of multi-step rewards [34], isolating HIQL's significant performance gains across a number of complex, long-horizon navigation tasks to improvements in policy extraction. While this does not preclude the potential benefits of multi-step rewards for offline GCRL, it does demonstrate that the advantages of hierarchies are not limited to temporally extended value learning, in line with previous claims that the primary bottleneck in offline RL is policy extraction and not value learning [38].
4.2 Value signal-to to-nal-t--noiee ratio in offline GCRL
Instead, HIQL claims to address a separate "signal-to-noise ratio" (SNR) issue in goal-conditioned value functions when the goal is very far away, where combination of sparse rewards and discounting makes it nearly impossible to accurately determine the advantage of one primitive action over another with respect to distant goals. This is also known the action gap phenomenon [14]. By separating
policy extraction into two levels, the low-level actor can instead evaluate the relative advantage of actions for reaching nearby subgoals and the high-level policy can utilize multi-step advantage estimates to get a better notion of progress towards distant goals.
To test whether improved SNR in advantage estimates with respect to distant goals is indeed the key to HIQL's superior performance, we propose to utilize subgoals to directly improve advantage estimates in simple baseline method term goal-conditioned waypoint advantage estimation (GCWAE). Briefly, we use the advantage of actions with respect to subgoals generated by a high-level policy as an estimator of the undiscounted advantage with respect to the true goal
A (st, Cte g) A (SL, ae, m (w) st, (s,
(5)
where the sg subseript indicates a stop- nop-gradient operator. Apart from using this advantage to extract a flat policy with AWR, we use the same architectures, sampling distributions, and training objective for h as HIQL. Despite large gains over one-step policy learning objectives several navigation tasks, GCWAE still underperforms its hierarchical counterpart, achieving 75% success rate on -arize-lrreee-avve
compared to 90% for HIQI. without subgoal representations and 16% for GCIVL with AWR policy extraction [Appendix H].

---

4.3
It's easier to find good (dataset) actions for closer goals
While diagnosing this discrepancy, we observed that training statistics for the two methods were largely identical except for a striking difference in the mean action advantage A(s,, Qz, w). The advantage was significantly lower for GCWAE, which samples "imagined" subgoals w ~ r²(. 8, {) from a high-level policy to train the flat poliey; than HIQI., which samples from the real k-step future state distribution w (s St) of the dataset to train the low-level policy (and then imagines subgoals during inference). This leads us to an obvious but important insight: in most cases, dataset actions are simply better with respect to subgoals sampled from nearby future states in the trajectory than to distant goals or "imagined" subgoals generated by a high-level policy. The dataset is far more likely to contain vanntage actions for goals sampled at the ends of short subsequences, whereas optimal state- aate-action pairs for more distant goals along the trajectory are much rarer due to the combinatorial explosion of possible goal states as the goal-sampling horizon increases.
The practical benefits of being able easily sample high-advantage state-actions acttonncion tuples are hinted at in Park et al. [37], who pose the question "Why can't It we use random goals when fraining policies?" after finding that offline GCRL algorithms empirically perform better when only sampling (policy) goals from future states in the same trajectory as the initial state. While their comparison focuses on in-trajectory versus random goals instead of nearer versus farther in-trajectory goals, we hypothesize both observations are driven by similar explanations: namely, the solutions reached by constrained policy optimization algorithms are dependent on the base sampling distribution over actions [28. 38].
4.4 Hierarchies perform test-time policy bootstrapping
Our observations suggest that training policies on nearby goals benefits both from better value SNR in advantage estimates and the ease of sampling good state-action-gol combinations. For brevity, we will refer to such policies trained only on goals of a restricted horizon length as "subpolicies," denoted by zsub and analogous to the low-level policies nH in hierarchies. Now we ask: how do hierarchical methods take advantage of the relative ease of training subpolicies to reach distant goals. and can we use similar strategies to train flat policies?
HIQL separately trains a low-level subpolicy on goals sampled from states at most steps into the future, reaping all the benefits of policy training with nearby goals. Similar other goal-conditioned hierarchical methods [33, 29], it then uses the high-level policy to predict optimal subgoals between the current state and the goal at test time, and "bootstraps" by using the subgoal-conditioned action distribution as an estimate for the full goal-conditioned policy.
5 Subgoal Sraatage-Weightee Policy Bootstrapping
We now seek to unify the above insights into an objective to learn a single, flat goal-reaching policy without the additional complexity of HRL. Following the bootstrapping perspective, direct analogue to hierarchies would use the subpolicy to construct training targets, regressing the full goal-conditioned policy towards a target action distribution nuu (a 8, > 8, s,)) given by subpolicy sud conditioned on optimal subgoals from a subgoal generator, similar to Equation 5.
Deriving such an objective in a principled manner leads us to a key contribution of this work: by viewing hierarchical policy optimization subgoals, we can (1) derive SAW, our training-tine bootstrapping objective that does not require generative subgoal models; and (2) recover RIS, related bootstrapping objective that still requires high-level policy [7]: as well as (3) the bilevel HIQL objective [Equation 4]. We present an abridged version below and leave the full derivations to Appendix B.
probabilistic inference over the optimal distribution of
5.1 Hierarchical RL as inference
The connection between RL and probabilistic inference is eeellesstblissed [10, 44, 26, 1, 404 and begins by constructing a probabilistic model over states, actions, and a binary optimality variable O, The joint likelihood of optimality (O, = for = (l..... T)) over a trajectory T is chosen as p(O = 1 ( x exp (E== r/r/).). where is a temperature parameter. Intuitively, O represents the event of maximizing return by choosing an action, i.e., maximizing progress towards the goal in the goal-conditioned case. Peng et al. [40] derive a similar form that instead maximizes the
5

---

expected policy improvement J(T) J(T)) of π over behavior policy n in terms of the advantage A* a) = Q a (s), which we use both for compatibility with HIQL and for practical improvements in stability. We provide more thorough introduction to these ideas in Appendix A.
To paint hierarchical policy optimization subgoals, where (U = I т, {w).g) 9) & exp (6 Et=o 7 (St, Wt, Wt, 9)). The binary variable U can be interpreted as the event of reaching the goal g quickly possible from state st by passing through subgoal state W. The subgoal advantage is defined as A(sr, w, g) = (Vr,
inference problem, define variable U over
( + Ei= r(su, 9). In practice, we follow HIQL and simplify the advantage estimate to V(w.g g) - (st. g). i.e., the progress towards the goal achieved by reaching w [Appendix F.1].
Without loss of generality (since we can represent any flat Markovian policy simply by setting r²( g) to point distribution on g). we use an inductive bias on the subgoal structure of GCRL to consider policies of factored hierarchical form
Pn (T 9) = p(s)) II p(s St, (n,aa))((a St, we )o) (wr st,
The distinctions between hierarchical approaches like HIQI and non-hierarchical approaches such as RIS and SAW begin with our choice of posterior distribution. For the former, we would consider similarly factored distributions, whereas for the latter, use a flat posterior o (r g) that factors as
of (r g) p(so) II p(s st, a:)a (at st,9),
1=0
assuming that the dataset policies are Markovian. We also introduce posterior q"({w) 9) which factors over a sequence of waypoints {w} = (wo, w1....} as
0
((w[ ()) p(5)) II p( st a.) sub (a, st, we we)gg(( st,g),
where we treat the target subpolicy sub as fixed. Using these definitions, we define the evidence lower bound (ELBO) on the optimality likelihood Pa (U 1) for policy π and goal distribution p(g)
log (U 1) = log f (Tg)p=(T, g)r(U = |r, [w], gdd w dr dg
p(U 1, [w) (g) q (1 (w) (g)
(eg log
[(g T).
Here, of and g² serve as auxiliary distributions that allow us to optimize their corresponding parametric policies Ty and Thhh via iterative wwwettatiooomaximizzatttt (EM) procedure. Expanding distributions according to their factorizations, dropping terms that are independent of the variationals, and rewriting the discounted sum over time an expectation over the (unnormalized) discounted stationary state distribution (x = E r'p(s: = 8 π) results in the final objective
T(g, m) Eu()).n(a) (gg(()))))) [A(s w. g)g)] (y((a))))) [DK (w 1|rh (w s.gg)]]
(urts.)))) [Du (a s, (/ w)) J. (6)
where we optimize an approximation of J by sampling from the dataset distribution (s((s) [45, 1, 40].
5.2 Eliminating the subgoal generator
Converting the KL penalty in the first line of Equation 6 to hard constraint (-1.) S allows us to find the closed form of the optimal posterior g² ("(w] 9. x T 8, exp(A(s, w,g)) [1, 40]. Both RIS and HIQL iteratively minimize the KL divergence between the parametric generative model n 9, and this sample-ber based posterior q². Then, RIS trains flat policy To using the remaining KL divergence term in the second line. This minimizes the divergence between the flat

---

Algorithm Subgoal Advantage- Weitggee Policy Bootstrapping (SAW)
1: Input: offline dataset D. goal distribution p(g).
2: Initialize value function Vo. target subpolicy Tgh, and policy mo-
3: while not converged do
4: Train value function: @ - XV (oCCVIIL( with (8t, 84+1) pp, 9 p(g) [Equation 2]
5: end while
6: while not converged do
7:
Train target subpolicy: t (WWRRW((W) with (st, a w) pd [Equation 3]
8: end while
9:
while not converged do
10: Train policy: t-6 0 AVoJsaw (0) with a, ~ ~p?. ~ p(g) [Equation 9]
11: end while
goal-conditioned policy posterior o' and the subgoal- conettioone policy x' in expectation over the optimal distribution of subgoals g as approximated by z²
(O) (w)))o [DL (To(a s, f) s, w.))
(7)
This brings us to the key insight underlying the SAW objective: instead of learning generative model over the potentially high-ding dimensiinnio space of subgoals to compute the expectation over q" (w), we can directly estimate this expectation from dataset samples using simple application of Bayes' rule:
p(w 8, s, U 1) x (w (U
w,g)
x p Pw |8) @eeeep((( w.g)),
which replaces the expectation over q to yield our subgoal advintage-weighted bootstrapping term Epthhhpp((oeehsgggglsspppppppp ((s,, (KL ())]l==(( (a] w))]-
(8)
We learn an approximation <a s, w) by training separate (sub)goal-conditione policy with AWR in a similar fashion to HIQL, whereas RIS uses an exponential moving average of the parameters of its full goal-conditioned policy as target. Since our regression target is parametric distribution, this conveniently allows us to directly model q² with another parametric policy Ie (a s,g).
While approximating p(w s, U 1) with q directly and using our importance weight the dataset distribution are mathematically equivalent, the latter does introduce sampling-based limitations, show empirically that the benefits from lifting the burden of learning a distribution over high-dimensional subgoal space far outweigh these drawbacks, especially large state spaces with high intrinsic dimensionality.
which discuss in Section 7. However,
5.3
The SAW objective
The importance weight in Equation 8 allows the policy to bootstrap from subgoals sampled directly from dataset trajectories by ensuring that only subpolicies conditioned on high-advantage subgoals influence the direction of the goal-conditioned policy [Figure 1]. We combine our bootstrapping term with an additional learning signal from a (one-step) policy extraction objective utilizing the value function, which improves performance in stitching -eevvnng-heavyy environments [Appendix I]. Here, one-step AWR [Equation 31, yielding the full SAW objective:
[(0) = (s..(())))))) (aA(),,))) log To(a 8. e 3A(e,w.)) DRL (To(a 8, g (a 8, w))
where and 3 are inverse temperature hyperparameters. This objective provides convenient dynamic balance between its two terms: as the goal horizon increases, the differences in action values and therefore the contribution of one-step term decreases. This, in turn, downweights the noisier value-based learning signal and shifts emphasis toward the policy bootstrapping term. Finally, we use GCIVL to learn V, resulting in the full training scheme outlined in Algorithm 1.
7

---

Figure 2: OGBench tasks. We train SAW on 20 datasets collected from 7 different environments (pictured above) and perform evaluations across 5 state-goal pairs for each dataset.
Experiments
To assess SAW's ability to reason over long horizons and handle high- high-dimensional observations, we conduct experiments across 20 datasets corresponding to locomotion and manipulation environments [Figure 2] with both state- and pixel-based observation spaces. We report performance averaged over 5 state-goal goal pairs for each dataset, yielding 100 total evaluation tasks. Implementation details and hyperparameter settings are discussed in Appendices F and G, respectively.
6.1 Experimental setup
We select several environments and their corresponding datasets from the recently released OGBench suite [37], a comprehensive benchmark specifically designed for offline GCRL. OGBench provides multiple state-goal pairs for evaluation and datasets tailored to evaluate desirable properties of offline GCRL algorithms,, such the ability reason over long horizons and stitch across multiple trajectories or combinatorial goal sequences. We use the baselines from the original OGBench paper, which include both one-step and hierarchical state-of-the-art offline GCRL methods. We briefly describe each category of tasks below and baseline algorithms in Appendix C, and encourage readers refer to the Park al. [37] for further details.
Locomotion tasks require the agent to control a simulated robot to navigate through a maze and reach a designated goal. The agent embodiment varies from simple 2D point mass with two-dimensional action and observation spaces to a humanoid robot with 21 degrees of freedom and di-dimensional state space. In the visual variants, the agent receives 64 x 64 x pixel-based observations, with its location within the maze indicated by the floor color. Maze layouts range from medium to giant, where tasks in the humano idmaze version of the latter require up to 3000 environment steps to complete.
Locomotion:
egocentric
Manipulation tasks use 6-DoF UR5e robot arm to manipulate object(s), including up to four cubes and a more diverse scene environment that includes buttons, windows, and drawers. The multi-cube and scene environments are designed test agent's ability to perform sequential, long-horizon goal stitching and compose together multiple atomic behaviors. The visual variants also provide 64 x 64 x 3 pixel-based observations where certain parts of the environment and robot arm are made semitransparent to ease state estimation.
Manipulation:
6.2 Locomotion results
State-based locomotion: As a method designed for long-horizon reasoning, SAW excels in all variants of the state-based locomotion tasks. It scales particularly well long horizons, exhibiting the best performance of 73% across all tasks in saanmazz-giantt-nvaiae
and is the first method to achieve non-trivial success in humanoi dwwze-aaziiieae giant-navvgast reaching 35% success compared to the previous state-of-the-are of 12% [37]. We demonstrate that training subpolicies with subgoal representations scales poorly to the giant maze environments [Figure 3] but are critical HIQL's performance, emphasizing fundamental tradeoff in hierarchical methods: compact subgoal represen- tations are essential for making high-level policy prediction tractable, but those same representations can constrain policy expressiveness and limit overall performance. While other subgoal representation learning objectives may perform better than those derived from the value function, as in HIQL, this highlights the additional design complexity and tuning required for HRL methods. We also implement an offline variant of RIS [Appendix CJ and find that it performs significantly worse than SAW with subgoal representations, which we suspect can be explained by insights in Section 4.3.

---

Environment
GCBC
GCIVL
GCIQL
CRL
HIQL
poistaaze aediun navigate
14
Lo
L5
29
-5
58 1.
97
pointmaze
pointaaze- large navigate-vd
29
45 a1
34 sa
86 21
39
58. =3
68 213
85 2.1
pointraze-gg neviggte-ee
68
27
46
67
68
untaaze-nediue www.--dd-----iaae-------
71
96
antaaze Large navigate
antaaze glant navigate-00
0.
0.a
D Lo
16
65 14
73
8*
27
60
89
73
88
huranoidraze
hunameidnaze- - waw.re---argg--erraa-....
2
2
0
24
49
21
46
Junanoldmaze giant savaagggeo
0.
3-42
333
bube-eingle-playyv
63
nube-domble-p apyyy--00 ay-v0
1:1
10
36 sa
4 =7
cube-tripla-play-
14
1 =0
0 m3
3
3
scene
scese play--pppp--
5.11
12 14
51
5
19 113
38
64. L7
63 .s
vieual aztaazze medium narigate v0
11
0
91
55
95
visual-antaaze
visical-attin wawpaze--aagee--iiiiaao
4:
4
844
40
Vieual artuaze saazzz---iaat glant navigate-v)
0.40
-2
viecal-cuber www.-cbbgeee--pleeee... paagl-
5.
00 a5
30 a5
-45
31
89 44
63 sm
88
vi Vianal-cube
visual -ouull--ih----h-e---- hott---ooohee--ly-------
1:x
10
1
6 m3
39 57
28
40
15
14
15
16
17
18
20
viaual-acese
visual-scane-plaay-v
Table I: Evaluating SAW on state- and pixet-based offline goal-conditioned RL tasks. We compare our
method's 'ss average (hinary) success rate (%) against the numbers reported in Park et al. [37] across the five test-time goals for each environment, averaged deviations after = sign. Numbers within 5% of the hest value the row are bold. Results with an asterisk (*) use different value learning hy hyperparameters and are discussed further in Section 6.3.
8 seeds (4 seeds for pixel-based visual tasks) with standard
antmaze -plant
14-
25-
25
110L www.ttaageaaaloo up
02
Figure 3: Subgoal representations scale poorly to high-dimensional control in large state spaces. Using HIQL's subgoal representations (taken from an intermediate layer of the value function) for SAW's target subpolicy harms performance compared to training directly on observations. However, HIQL fails to learn meaningful behaviors when predicting subgoals directly in the raw observation space. RIS, which bootstraps on generated subgoals at every step, performs the worst of the three.
Pixel-based lucomotion: SAW maintains strong performance when given 64 x 64 x 3 visual obser- vations and scales much better to visual- -araazze-argg large than does its hierarchical counterpart. However, we do see a significant performance drop in the giant variant relative to the results in the state-based observation space. As possible explanation for this discrepancy, observed that value function training diverged for HIQL and SAW in sesuualanttazeegiiiie visual -usullhumano dmaze sizes (omitted since no method achieved non-trivial performance). This occurred even without shared policy gradients, suggesting that additional work is needed to scale offline value learning objectives very long-horizon tasks with high-dimensional visual observations.
as well as all
6.3 Manipulation results
State-based manipulation: SAW consistently matches sate-of-the-aa performance in cube environ- ments and significantly outperforms existing methods in the scene tasks, which require extended compositional reasoning. Interestingly, we found that methods which use expectile regression-basse based offline value learning methods (GCIVL, GCIQL, HIQL, and SAW) are highly sensitive to value learning hyperparameters in the cube-single single environment. Indeed, SAW performs more than twice as well on sube--inngleppayy performance (72% 45 vs. 32% with 0.7). While SAW is agnostic to the choice of value
with settings of T = 0.9 and 8 = 0.3, reaching state-of-the-an

---

learning objective, we make special mention of these changes since they depart from the OGBench convention of fixing value learning hyperparameters for each method across all datasets.
Pixel-based manipulation: contrast to the state-based environments, SAW and HIQI. achieve near-equivalent performance in visual manipulation. This suggests that representation learning, and not horgooor reasoning or goal stitching, is the primary bottleneck in the visual manipulation environments. While we do not claim any representation learning innovations for this paper, our results nonetheless demonstrate that SAW is able to utilize similar encoder-sharing tricks as HIQL to scale to high- -ighddimensiona observation spaces.
7 Limitations
A theoretical limitation of our approach, which is common to all hierarchical methods as well as RIS, occurs in our assumption that the optimal policy can be represented in the factored form ra (w 8, g). While this is true in theory (since we could trivially set ""(w 8, g) to point distribution at g). practical algorithms typically fix the distance of the subgoals to a shorter distance of & steps, where subgoals $t+k are sampled from the distribution Pern ($1+* st) induced by the behavior policy TT. This introduces bias by limiting the space of subgoals to k-step future occupancy distribution of the behavior policy Tz. rather than that of the current learned policy. However, discussed in Section 4.3, we find that subgoals sampled from the future state distribution work empirically well with respect to goals also sampled from the future state distribution, which is common in practice [17, 16, 57, 13, 36].
We suspect that methods which generate subgoals may perform better in datasets that require a high degree of stitching: they can synthesize "imagined" subgoals on which to bootstrap, while our approach may require alternative sampling strategies to reach the same level of performance. While SAW maintains strong performance compared to baselines when trained on suboptimal stitch datasets, its performance does degrade in the highly suboptimal explore datasets [Appendix J].
Discussion
We presented Subgoal Advantage- -eeighed Policy Bootstrapping (SAW), simple yet effective policy extraction objective that leverages the subgoal structure of goal-conditioned tasks to scale to long-horizon tasks, without learning generative subgoal models. SAW consistently matches or surpasses current state-of-the- methods across a wide variety of locomotion and manipulation tasks that require different timescales of control, whereas existing methods tend specialize in particular task categories. Our method especially distinguishes itself in long-horizon reasoning, excelling in the most diffieult locomotion tasks and scene-based manipulation.
In addition to providing a denser learning signal, one of the primary functions of value bootstrapping to reduce the variance of the target estimates. Does this also apply policy bootstrapping? We hypothesize that the answer is yes: when the value functions are noisy, as in long-horizon tasks, the advantage weight on given action, and therefore the gradient of the policy objective, can be extremely high-variance over the course of training. On the other hand, our bootstrapping objective benefits from stable action target provided by a subpolicy, which in turn enjoys clearer advantage signals [Section 4.3]. We believe that utilizing such target policies is a promising direction for improving the stability of oong-oorizon policy learning, especially when value functions noisy.
As brief addendum, we suggest that the family of policy bootstrapping algorithms can be seen as practical instantiation of the chunking theory of learning from neuroscience, which suggests that mastering complex, sequential skills initially involves breaking them down into a series of chunks [43, 6]. Individual chunks are blended into smoother and more efficient movements with repeated practice, providing a mechanism for acquiring progressively longer and more complex skills that can again be used as primitive chunks for planning, in a virtuous cycle of improvement over the horizon.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by the following awards JCK: National Institutes Health DP2NS and NSF CAREER 1943467.
References
[1] Abbas Abdolmalcki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a Posteriori Policy Optimisation. In International Conference Learning Representations, 2018.
[2] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight Experience Replay. In Neural Information Processing Systems, 2017.
[3] Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. Minimax Regret Bounds for Reinforcement Learning. In International Conference on Machine Learning, 2017.
[4] Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In AAAI Conference on Artificial Intelligence, 2017.
[5] Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. No: A Ssiion-Language-Aete Flow Model for General Robot Control. arXiv, 2410.24164 [cs], 2024.
[6] Rachel M. Brown and Iring Koch. Repetition costs in sequence chunking. 2023.
[7] Elliot Chane-Sane, Cordelia Schmid, and Ivan Laptev. Goal-Conditioned Reinforcement Learning with Imagined Subgoals. In International Conference on Machine Learning, 2021.
(8) Konrad Czechowski, Tomasz Odrzygóidz, Marek Zbysiński, Michal Zawalski, Krzysztof Olejnik, Yuhuai Wu, Lukasz Kucińiski. and Piotr Miloć. Subgoal Search For Complex Reasoning Tasks. In Neural Information Processing Systems, 2021.
[9] Peter Dayan and Geoffrey E Hinton. Feudal Reinforcement Learning. In Neural Information Processing Systems, 1992.
[10] Peter Dayan and Geoffrey E. Hinton. Using Enpectation-Maxiiizzztoi for Reinforcement Learning. Neural Computation, 9()):771-278, 1996.
[11] Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: Scalable Distributed Deep-RL. with Importance Weighted Actor-Learner Architectures. In International Conference Machine Learning, 2018.
[12] Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. Search on the Replay Buffer: Bridging Planning and Reinforcement Learning. In Neural Information Processing Systems, 2019.
[13] Benjamin Eysenbach, Tianjun Zhang, Sergey Levine, and Ruslan Salakhutdinov. Contrastive Learning as Cond-toondt Reinforcement Learning. In Neural Information Processing Systems, 2022.
[14] Amir-massoud Farahmand. Action-Gap Phenomenon in Reinforcement Learning, In Neural Information Processing Systems, 2011.
[15] Scott Fujimoto and Shixiang Gu. A Minimalist Approach to Offline Reinforcement Learning. In Neural Information Processing Systems, 2021.
[16] Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Manon Devin, Benjamin Eysenbach, and Sergey Levine. Learning to Reach Goals via Iterated Supervised Learning. In International Conference Learning Representations, 2020.

---

[17] Abhishek Gupta, Vikash Kumar, Corey Lynch. Sergey Levine, and Karol Hausman. Relay Policy Learning: Solving Hong-Horizon Tasks via Imitation and Reinforcement Learning. In Conference Robot Learning, 2020.
[18] Danijar Hafner, Kuang- Huei Lee, lan Fischer, and Pieter Abbeel. Deep Hierarchical Planning from Pixels. In Neural Information Processing Systems, 2022.
[19] Kyle Beltran Hatch, Ashwin Balakrishna, Oier Mees, Suraj Nair, Seohong Park, Blake Wulfe, Masha Itkina, Benjamin Eysenbach, Sergey Levine, Thomas Kollar, and Benjamin Burchfiel. GHIL-Glue: Hierarchical Control with Filtered Subgoal Images. In CoRL Workshop on Mastering Robot Manipulation a World of Abundant Data, 2024.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In IEEE Conference Computer Vision and Pattern Recognition, 2016.
[21] Zhiao Huang, Fangehen Liu, and Hao Su. Mapping State Space using Landmarks for Universal Goal Reaching, In Neural Information Processing Systems, 2019.
[22] Nicholas K. Jong, Todd Hester, and Peter Stone. The utility of temporal abstraction in rein- forcement learning. In International Joint Conference on Autonomous Agents and Multiagent Systems, 2008
[23] Leslie Pack Kaelbling. Learning to achieve goals. In International Joint Conference on Artificial Intelligence, 1993.
[24] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline Reinforcement Learning with Implicit Q-Learning. In International Conference on Learning Representations, 2021.
[25] Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch Reinforcement Learning. Springer, 2012.
[26] Sergey Levine. Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review. arXiv, 1805.00909 [cs], 2018.
[27] Sergey Levine and Vladlen Koltun. Variational Policy Search via Trajectory Optimization. In Neural Information Processing Systems, 2013.
[28] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu, Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems. arXiv, 2005.0164) 2020.
[29] Andrew Levy, George Konidaris, Robert Platt, and Kate Saenko. Learning Multi-Level Hierar- chies with Hindsight. In International Conference on Learning Representations, 2018.
[30] Yecheng Jason Ma, Jason Yan, Dinesh Jayaraman, and Osbert Bastani. Offline Goal-Conditions Reinforcement Learning via --Advantage Regression. In Advances in Neural Information Processing Systems, 2022.
[31] Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. VIP: Towards Universal Visual Reward and Representation via Value-Imple Implicit Pre-Training. In International Conference on Learning Representations, 2023.
[32] Vivek Myers, Catherine Ji, and Benjamin Eysenbach. Horizon Generalization in Reinforcement Learning. In International Conference on Learning Representations, 2024.
[33] Ofir Nachum, Shixiang (Shane) Gu, Honglak Lee, and Sergey Levine. cal Reinforcement Learning. In Neural Information Processing Systems, 2018.
Hierarchi-
[34] Ofir Nachum, Haoran Tang, Xingyu Lu, Shixiang Gu, Honglak Lee, and Sergey Levine. Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning? In NeurIPS DeepRL Workshop, 2019.
[35] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3M: A Universal Visual Representation for Robot Manipulation. In Conference on Robot Learning, 2023.
12

---

[36] Seohong Park, Dibya Ghosh, Benjamin Eysenbach, and Sergey Levine. HIQL: Offline Goal- Conditioned RL with Latent States as Actions. In Neural Information Processing Systems, 2023.
[37] Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. OGBench: Benchmark- ing Offline Coal-Conditioned RL. In International Conference on Leurning Representations, 2024.
[38] Seohong Park, Kevin Frans, Sergey Levine, and Aviral Kumar. Is Value Learning Really the Main Bottleneck in Offline RL? In Neural Information Processing Systems, 2024.
[39] Seohong Park, Tobias Kreiman, and Sergey Levine. Foundation Policies with Hilbert Represen- tations. In International Conference on Machine Learning, 2024.
[40] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-Weighted Regres- sion: Simple and Scalable Off-Policy Reinforcement Learning, arXiv. 1910.00177, 2019.
[41] Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Dibya Ghosh, Lachy Groom. Karol Hausman, Brian Ichter, Szymon Jakubezak, Tim Jones, Liyiming Ke. Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Allen Z. Ren, Lucy Xiaoyang Shi, Laura Smith, Jost Tobias Springenherg, Kyle Stachowicz, James Tanner, Quan Vuong. Homer Walke, Anna Walling, Haohuan Wang. Lili Yu, and Ury Zhilinsky. o..: 2504.16054 [cs], 2025.
Model with Open-World World Generalization. arXiv,
[42] Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley and Sons, 2005.
[43] Pavan Ramkumar, Daniel E. Acuna, Max Berniker, Scott T. Grafton, Robert S. Turner, and Konrad P. Kording. Chunking as the result of an efficiency computation trade-off. 7(1):12176, 2016.
[44] Konrad Rawlik, Mare Toussaint, and Sethu Vijayakumar. On Stochastic Optimal Control and Reinforcement Learning by Approximate Inference. In Robotics: Science and Systems, 2008.
[45] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust Region Policy Optimization. In International Conference on Machine Learning, 2015.
[46] John Schulman, Xi Chen, and Pieter Abbeel. Equivalence Between Policy Gradients and Soft Q-Learning. arXiv, 1704.06440 [cs], 2018.
[47] Younggyo Seo, Kimin Lee, Stephen L. James, and Pieter Abbeel. Reinforcement Learning with Action-Free Pre-Training from Videos. In International Conference on Machine Learning, 2022.
[48] Harshit Sikchi, Rohan Chitnis, Ahmed Touati, Alborz Geramifard, Amy Zhang, and Scott Nickum. Score Models for Offline Condditiondd Reinforcement Learning. In The Twelfth International Conference on Learning Representations, 2023.
[49] Alexander L. Strehl, Lihong Li, and Michael L. Littman. Reinforcement Learning in Finite MDPs: PAC Analysis. Journal of Machine Learning Research, 10(84):2413-2444, 2009.
[50] Richard S. Sutton, Doina Precup, and Satinder Singh. Between MDPs and semi-MDPs: A
framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1): 181-211, 1999.
[51] Stephen Tian, Suraj Nair, Frederik Ebert, Sudeep Dasari, Benjamin Eysenbach, Chelsea Finn, and Sergey Levine. Model-Based Visual Planning with Self- Supervisd Functional Distances. In International Conference on Learning Representations, 2020.
[52] Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. FeUdal Networks for Hierarchical Reinforcement Learning. In International Conference Machine Learning, 2017.
13

---

[53] Kevin Wang. Ishaan Javali, Michat Bortkiewicz, Tomasz Trzciński, and Benjamin Eysenbach. 1000 Layer Networks for Self Suff-Sueevised RL: Scaling Depth Can Enable New Goal- Roal-Reaching Capabilities. arXiv, 2503.14855 [es], 2025.
[54] Tongzhou Wang and Phillip Isola. Understanding Contrastive Representation Learning through Aligmment and Uniformity on the Hypersphere. In International Conference on Machine Learning, 2020.
[55] Tongzhou Wang and Phillip Isola. Improved Representation of Asymmetrical Distances with Interval Quasimetric Embeddings. In NeurIPS Workshop on Symmetry and Geometry in Neural Representations, 2022.
[56] Tongzhou Wang, Antonio Torralba, Phillip Isola, and Amy Zhang. Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learning. In International Conference on Machine Learning 2023.
[57] Rui Yang. Yiming Lu, Wenzhe Li, Hao Sun, Meng Fang, Yali Du, Xiu Lei Han, and Chongjie Zhang, Rethinking Goal-Conditioned Supervised Learning and Its Connection to Offline RL. In International Conference on Learning Representations, 2021.
[58] Rui Yang, Lin Yong, Xiaoteng Ma, Hao Hu, Chongjie Zhang, and Tong Zhang. What is Essential for Unseen Goal Generalization of Offline Goad-conditioned RL? In International Conference on Machine Learning, 2023.
[59] Michal Zawalski, Michat Tyrolski, Konrad Czechowski, Tomasz Odrzygóšdz, Damian Stachura, Piotr Piçkos, Yuhuai Wu, Lukasz Kuciński, and Piotr Miloš. Fast and Precise: Adjusting Planning Horizon with Adaptive Subgoal Search. In International Conference on Learning Representations, 2022.
[60] Zilai Zeng, Ce Zhang, Shijie Wang, and Chen Sun. Goal-Conditioned Predictive Coding for Offline Reinforcement Learning. In Neural Information Processing Systems, 2023.
[61] Lunjun Zhang, Ge Yang, and Bradly C. Stadie. World Model as Graph: Learning Latent Landmarks for Planning. In International Conference on Machine Learning, 2021.
[62] Tianren Zhang, Shangqi Guo, Tian Tan, Xiaolin Hu, and Feng Chen. Generating Adjacency- Constrained Subgoals in Hierarchical Reinforcement Learning. In Neural Information Process- ing Systems, 2020.
[63] Chongyi Zheng, Ruslan Salakhutdinov, and Benjamin Eysenbach. Contrastive Difference Predictive Coding. In International Conference on Learning Representations, 2023.
14

---

A Reinforcement Learning as Probabilistic Inference
In effort to make this work more self-contained, we provide short overview of the probabilistic inference perspective on reinforcement learning, which we use to derive the HIQL, RIS, and SAW objectives in Appendix B. The general structure of these derivations will be quite similar those discussed later on and follows closely from Abdolmaleki et al. [1], although we do deviate some places for clarity and correctness. For more thorough treatment, interested readers may refer the tutorial by Levine [26]. We break this introduction into two sections: (1) general introduction to the RL inference framework, (2) and discussion of the modifications and assumptions made apply these ideas to offline RL with a fixed stationary distribution p (s).
A.1 RL as inference
To view optimal control as probabilistic inference problem, we introduce optimality variable Oz. which takes the form of a binary random variable where O, = 1 the decision at time step t was optimal, and O = 0 if not. We define the joint distribution of optimality variables in trajectory T as
p(O = 1 x exp
/'r//a
where is a temperature hyperparameter and rr r(s,, a.). This has a natural interpretation: we assume deterministic dynamics, the probability of trajectory induced by the optimal policy is proportional to its exponentiated return. We can view this as a "soft" maximization over actions compared the hard arg max in greedy Q-learning [46]. A natural objective is then to optimize the marginal likelihood optimality for policy æ. Unfortunately, this a function of an expectation over the space of trajectories, which is intractable to compute. Instead, can form the evidence lower bound (ELBO) by using Jensen's 's inequality and introducing variational approximation g(T)
log Prr (0 1) log / Px (x(T)p(O
T)
2 / ((T)
logp(0 = 1) log (T( dr
q(T)
Eg
r're/a
- DKL (g(F))(p=(T)
11=0
= J(4)-
Under the assumption that (+( = (30) II: (8t41 st, as) v (at, st),, i.e., π is Markovian, we consider a family of variational distributions that factor in the same way as (T)
00
((T) = p (50) II (#t41 8t, ae) (at st).
t=
This choice of distribution allows to rewrite the divergence between trajectories as a factored distribution over individual timesteps, where all terms not depending on cancel out and we move the weighting parameter to the second term (which is equivalent for optimization purposes)
q(T))
[(4) = Eg(r)
7 rt
E ())
a log
(T(TT)
o
q(aa, (x))
= Bg(or
r'r
- Byttr)
Σ 1=
a log
T(a: 8t)
q(a; 2t) 8x))
= Eg(r)
7'r ar log
(10)
T(a:
Prior work [1] additionally discounts the per-timestep divergence term, that optimizing J with respect to is equivalent to solving the standard RL problem with an augmented reward f, = a log g(a+[s)). We follow this convention as well in the following section.
[
15

---

A.2 Offline policy optimization
To optimize the above objective, we define the state-action value associated with Equation 10 as
g(ar sc) ( 8x)
(s) = + =00
Tt log
(11)
We then perform partial maximization of J. The key difference between this and standard policy improvement, which updates (s) < arg max, r(s, a) + +BBp((r/ss.a)) [V (s')), .). that we select the
soft-optimal action instead of a hard arg max and use the value function V associated with the fixed policy æ instead of the current policy q.
We first expand Q using the regularized Bellman operator, which replaces r with
T By(als) r(s, a) a log g(a s)
+ >BB((')ss.) [V (6'))
T(a 8) By(als) (r(s,a) to [V" (s')1] aDkL (a(a s)ll s))
Bg(a(s) (s, a)] aDRL (g(a ()][=(a)) 8)).
This gives us "one-step" KI.-regularized objectie
max
max E (s) [ 90 (s, a)]
4
max E (x) [g(a)s) 1Q a)] - aDKL (4(a s)][=(a s))]
(12)
where (s) is the discounted stationary state distribution induced by π.
This does not fully optimize I since we ignore the dependence of the Q-value on policy However, we can also learn Q from off-policy data: in this work, we use a variant of implicit Q-learning [Equation 2]. which learns value function for improved (implicit) policy via value iteration. While the soft-optimal actor learned via this optimization procedure may differ from the implicit actor of IQL, such decoupled approaches work empirically well in practice.
For mathematical convenience, we can turn the KL regularization term into a hard constraint
max Eu(s) [Bg(als) (Q" a)]]
s.t. (u(a) DKL (g(a 8), (a 8))] <
(13)
and use Lagrangian duality to find a closed form for the optimal posterior
q(a 8) exp(Q(s, a)/o))a((a s).
(14)
This can be interpreted Bayesian update, where the prior me is updated by the likelihood of optimality (xp(Q(ss a)) Given tractable way to estimate the optimality-condiioone posterior distribution from samples, we can now turn policy optimization into a variational inference problem.
Expectation maximization for policy learning: Framing RL an inference problem allows to use tools from approximate variational inference to estimate the optimal state-conditional distribution over actions. The expectation maximization (EM) algorithm is one such tool that gives rise family of policy improvement al algorithms [10, 1, 40), and can be described as "solving sequence of probability matching problems, where is chosen at each step to match as best can a fictitious distribution that is determined by the average rewards experienced on the previous step" [10].
More concretely, EM algorithms for RL are based an iterative two-step process, consisting of:
Expectation (E) step: We construct a nonparametric, sample-based posterior q(a | s) x exp(M ((, a)/a)Tp(a s), where M(s, a) is some performance measure, e.g., the Q-function [1] the advantage function [40]. This exactly the "fictitious" distribution referenced above. As mentioned above, replace ne with the fixed behavior policy Ty in the offline setting.
16

---

2. Maximization (M) step: In this step, we update our parameterized policy towards this fictitious target by minimizing the KL divergence between the two. In practice, we are rarely able find the setting of 0 that globally minimizes the divergence-anc only noisy approximation of the true optimal posterior and can suffer from high sampling error. Instead, we simply take a few gradient steps on / towards q before resampling.
neither do we want to, since is
When q is non-paraan con-parametric as described above, we can directly substitute in our expression for [Equation 14] with normalizing constant into the KL divergence and simplify, turning both steps into the single objective of Maximum a Posteriori Policy Optimization [1, MPÓ]
IMPO E P (s),a)zz(aa4) [cxp(Q(s, a)/a) log TT(a s)]
(15)
similar procedure can be used on an objective that maximizes the expected improvement Ty( ((re) (TT over the behavior policy Te. giving rise to an advantage- weighted objective [40]
JAwR (0) P (s). [exp(A(s, a)/a) log Ta(a s)]
(16)
As point of clarification, in the online setting our replay buffer D is populated by fresh experience from which can be thought of as the prior distribution corresponding to the posterior However, in the offline setting, the prior is fixed to the offline dataset D, and ve instead takes the role of parametric estimate of the shape of that is repeatedly updated over sampled batches.
B Derivation and Unification of the HIQL, RIS, and SAW Objectives
As in the standard inference framework introduced in Appendix A, we the difnite-horizon counted GCRL formulation as an inference problem by constructing a probabilistic model. However, we introduce new variable corresponding to subgoal optimality, the likelihood function
dis-
p pU = 1 (w).9) x exp
3
7 (St, Wt, Wt,.9)
1=0
where B is an inverse temperature parameter and the binary variable U can be intuitively understood as the event of reaching the goal g as quickly as possible by passing through a subgoal wr, or passing through a subgoal which is on the shortest path between and Following Park et al. [36], we elect use an advantage function to reduce variance, which is especially important when considering multi-step advantage estimates as we do here.
We consider policies r/ and x² of a factored hierarchical form
00
(=(7 (g) = p(so) I St, .e) as(a st, we) (he st, g),
where the low-level policy over actions oh and high-level policy over subgoals Th are parameterized by and tb, respectively. However, for the sake of clarity, we drop the parameters from the notation in the following derivations.
B.1 HIQL derivation
For HIQL's hierarchical policy, we consider distributions of and g² of the same form
sc
q{r g) p(so) II p(sea st, at) o (at st, w.) g" (we 9).
To incorporate training the low-level policy, we also construct an additional probabilistic model for the optimality of primitive actions towards waypoint (note that this also be done to incorporate target policy training into the SAW objective, but we leave it out for brevity)
p0 = I,, (w+) x exp
r A A Gr, u.))
17

---

With these definitions, we define the evidence lower bound (ELBO) on the joint optimality likelihood P=(O U, 1) for policy æ and posterior q as
log (0 1,U - = = log / mo) 1, -,,,, [w) | g)d(w) dr dg
log / (g)a ( - 9)
dfwr dr dg
(((19)) |g
p00= 100,,,,U1,7
log ((rtrtoo
d'(r 9)4
p00 -,,,,,,,,,,,,
(eld log
q'(r ()
Expanding the fraction, moving the log inside, and dropping the start state distribution p(so) and transition distributions p(s++1 St, ar), which are fixed with respect and tb, gives
of (4w)))))
y' A(se, Ce. wx)
8
> 7' A(8e, W2,g)
t=
==
( 8e, w) x" 8 9) gfaal ( st.g) )
log
t=0
We rewrite the discounted sum over time as expectation over the (unnormalized) discounted stationary state distribution (s((s) = Σi= ? p(s = т) induced by policy æ. In practice, however, optimize approximation of [gg 0g vb) by sampling from the dataset distribution over states μrD, as justified in Appendix A. For brevity, we omit the conditionals in the expectations below, defining of (a) (a (, and q" (ww == g" (w s)
T (a s,w) ga s, w)
E
aA(s,a, log
(w" 8.9)
+ (
BA(s, w.g) log
(ww 8,9)
[aA(s, a, w)] ( [DEL [q s, w)ll"'(a s, w.]]] + ( [BA(s, w, g)] (e(e).p( [DKL (w (,)])l""(" [,,)]]] (17)
HIQL separately optimizes the two summation terms, which correspond to the low- and high-level policies, respectively. Following Abdolmaleki et al. [1] and Peng et al. [40], we turn each KL divergence penalty into hard constraint (xI(-Il-) S form the Lagrangian, and solve for the optimal low- and high-level policies to yield the sample-based gf and g" distributions
gf (a 8, s,g) x n (a 8, s, exp(A(s,,, (a s,9) & (w s, exp( A(s, w. w,g)).
Minimizing the KL divergence between these sample-based estimates of the optimal posteriors and their respective parametric policies v and z yields the bilevel AWR policy extraction objectives for HIQI
Tinqq. (a))) [exp(A(ss a, w)) log Tf(a s, s,w]] JHiQ(()) (u(s)) [exp(A(s, w, g)) log (w s.g)].
While our derivation produces an on-policy expectation over states, actions, and subgoals, we approximate this objective with samples drawn from the dataset distribution following the justification in Appendix A. The goal distribution is not known priori (see Appendix C of Park al. [37] for commonly used goal distributions).
18

---

B.2 RIS and SAW derivations
Unlike HIQL, RIS and SAW both seek learn unified flat policy, and therefore we choose a policy posterior qº (T) that factors
of (T g) p(so) II p( 88, (,) (at st:
For RIS, which learns a generative subgoal policy identical to that of HIQL, we also use the same distribution g which factors over sequence of waypoints {wo (Wo, W1,--..
((w) 9) = p(50)
p( st, ae) (az st, we) (w;; St, g).
t=
Rather than jointly learning low- and high-level policies, RIS bootstraps from target subpolicy x zuub which is treated fixed. Using these definitions, we define the evidence lower bound (ELBO) on the likelihood of subgoal optimality px = 1) for policy π
log px (U 1) log / p(g)p(U I, (w) 99)ddd(wdg
/ plo)a'( (9))
9) p(U = 1,T,
g) d(w) dr dg
log
q 9)
g)
plU = 1, {w)
9)
log (4
g)
1,7, [w] 9)
2 (1ol log
= m]
( g)
Expanding the fraction, moving the log inside, and dropping the start state distribution p(sa), transi- tion distributions p($ St; 0t), and target subpolicy nu (a: st, Wz), which are fixed with respect to the variationals, leaves us with
nt' (az St, we) T (wr st.g) t=0 q (a, 8e.g) (wv St, .)
(r g). ((
B
Σ t=0
r A(s,, We, g) +
log
Once again, we express the discounted sum over time as an expectation over the discounted stationary state distribution jus) and omit the conditionals in the expectation over qd (a) and qd (w) for brevity. Simplifying gives
<n s.9)
x aa 8, w)
]
E (o)))) BAAss, w.g) log
+ log
(tww s.g)
(a s.g)
Eu(s)) [BA(s, w, g)g - Eu(s). [DKL (q" (w s, g) (w s,9)) + (10).) [DRL. (a' (a 8, || (a w.)]
RIS partitions this objective into parts and optimizes them separately. The first line is identical to the HIQL high-level policy objective and yields the same AWR-like objective for parametric generative subgoal policy n.
RIS:
This (v)) = (10))1))) [exp(A(s, w, 9)) log ( s. s.
The remaining KL divergence term seeks minimize the divergence between the flat goal-conditioned policy posterior of and the subgoal-conditione target subpolicy nuub(a (a s, w) from earlier. Importantly, because nuu (a s, w) is (presumably) parametric distribution, we can directly optimize the KL divergence in the space of policies with another parametric policy To (,g).. yielding the final bootstrapping objective
policy <a s, w). which fit separately with the
This [u(()) (w)) v(a) DKL. (Te(a)) s, s (a s, w))]
19

---

This divergence is minimized in expectation over optimal subgoals qd s, which in turn is approximated by the parametric subgoal policy r" (we 8,9))-
SAW: Instead of fitting a generative model q" (w s. over the potentially sigh-dimensional space of subgoals, we can use simple application of Bayes' rule to directly approximate the expectation over the optimality-condiionee distribution of subgoals p(w 8,g. U = =1), where
p(w 8, U = 1) x pw 6)D(U = w,g)
x xp(w 8) exp((As, w, g)).
Although the proportionality constant in the first line the Pa (U 1), which is the subject of optimization, we note that approximating the expectation over subgoals w corresponds to the expectation step in a standard expectation- expectationmmxximizatooo (EM) procedure [1]. Because we are only seeking to fit the shape of the optimal posterior over subgoals for the purposes of approximating the expectation over g², and not maximizing Pr = 1) (the M step), we can treat p., (U = 1) as constant with respect to g² to get
JsAw (0)
ou p(a) [PK s, s,g) (a 8, w))]
a(s) (w | (T)RRI(TO(a s, s,g))l s, w.) dg ds
x p(s) D" (wl exp(A(s, ()x () s, w) dg ds
= P (wels))))) exp(A(s, (s, w. 9) (DLL s, | (a s, w))l,
giving us the final subgoal advintage-weighteed bootstrapping term in Equation 8.
C Offline GCRL Baseline Algorithms
In this section, we briefly review the baseline algorithms referenced in Table 1. For more thorough implementation details, well goal-sampling distributions, interested readers may refer to Appendix C of Park et al. [37] as well as the original works.
Goal-conditioned behavioral cloning (GCBC): GCBC is an imitation learning approach that clones behaviors using hindsight goal relabeling on future states in the same trajectory.
Goal-conditioned implicit (Q, VJ-learning (GCIQL & GCIVL): GCIQL is goal-conditioned
variant of implicit Q-learning [24), which performs policy iteration with expectile regression to avoid querying the learned Q-value function for outtof-distribution actions. Park et al. [36] introduced V-only variant that directly regresses towards high-value transitions [Equation 2], using r(s,g) g) + >V (s', g) as an estimator of (ss, a, g). Since it does not learn Q-values and therefore cannot marginalize over non-causal factors, it is optimistically biased in stochastic environments.
Although both baselines are value learning methods that can be used with multiple policy extraction objectives (including SAW, which extracts policy from value function learned with GCIVL), the OGBench implementations are paired with following objectives: Deep Deterministic Policy Gradient with behavior cloning penalty term [15, DDPG+BC] for GCIQL, and AWR [Equation 3] for GCIVL
Quasimetric RI (QRL): QRL [56] is non-traditional value learning algorithm that uses Interval Quasimetric Embeddings [55, IQE] to enforce quasimetrie propertics (namely, the triangle inequality and identity of indiscernibles) of the goal-conditioned value function. It relies on the Lagrangian dual form of a constrained "maximal spreading" objective to estimate the shortest paths between states, then learns one-step dynamics model combined with DDPG+BC to extract policy from the learned representations.
Contrastive RL (CRL): CRL [13] is a representation learning algorithm which uses contrastive learning to enforce that the inner product between the learned representations of a state-action pair and the goal state corresponds to the discounted future state occupancy measure of the goal state, which estimated directly from data using Monte Carlo sampling. CRI. then performs one-step policy improvement by choosing actions that maximize the future occupancy of the desired goal state.
20

---

Hierarchical implicit Q-learning (HIQL); HIQL [36] is policy extraction method that learns two levels of hierarchical policy from the same conditiodd value function. The low-level policy n' is trained using standard AWR, and the high-level policy x² is trained using an action-free, multi-step variant of AWR that treats (latent) subgoal states as "actions."
Reinforcement learning with imagined subgoals (RIS); RIS [7] is policy extraction method originally designed for the online GCRL setting, which learns a subgoal generator and a flat, goal- conditioned policy. Unlike SAW, RIS uses a fixed coefficient on the KL term, instead learning a subgoal generator and bootstrapping directly on target policy (parameterized by an exponential moving average of online policy parameters rather than separately learned subpolicy) conditioned on "imagined" subgoals. It also incorporates a value-based policy learning objective similar to our approach, but learns a Q-function and differentiates directly through the policy with DDPG.
To modify RIS for the offline setting in implementation for Figure fixed the coefficient on the KI. term to 3 = 100.0 for navigation and / = 3.0 for manipulation tasks, trained a subgoal generator identical the one in HIQL, and replaced the dataset subgoals SAW with "imagined" subgoals. Otherwise, for fairness of comparison, our offline RIS implementation SAW, including separate target subpolicy
Offiin RIS (RIS
used the same hyperparameters and architectures network instead of a soft copy of the online policy, subgoals at a fixed distance instead of at midpoints, and AWR instead of DDPG+BC for the policy extraction objective, which we found to perform better in locomotion environments.
D Links to Planning Invariance
As an aside, we note that the discussions in this paper are closely related to the recently introduced concept of planning invariance [32], which describes policy that takes similar actions when directed towards a goal as when directed towards an intermediate waypoint en route to that goal. In fact, we can say that subgoal-conditioned HRL methods achieve form of planning invariance by construction, since they simply use the actions yielded by seypoint-conditioa policies to reach further goals. By minimizing the divergence between the full conliconditioned policy and an associated subgoal- conditioned policy, both SAW and RIS can also be seen as implicitly enforcing planning invariance.
E
Computational Resources
All experiments were conducted on a cluster consisting of Nvidia GeForce RTX 3090 GPUs with 24 GB of VRAM and Nvidia GeForce RTX 3070 GPUs with 8 GB of VRAM. State-based experiments take around 4 hours to run for the largest environments (humanoi dmaze- -gee-ggie giant aat-navviaate) and visual experiments up to 12 hours.
F Implementation Details
F1 Simplified advantage estimates
GCRL typically uses a sparse reward formulation where the reward is always constant (typically -)): in other words, the coal-conditioned reward function is always r(s. g) = -1 unless the current state is within some small distance e from the goal g, where r(sg. g) = 0. While the full advantage estimate would be A (st, w, w.g) -V (st, g) + +>*V(W, g) --1 -r-r r (sr .g), the
over rewards should almost always be a constant when combined with hindsight goal relabeling, unless the goal was already reached at some point between time t and f + k 1. Finally, the discount 7 is always constant (since we fix as a hyperparameter), and can therefore be roughly folded into the inverse temperature parameter 3 in the subgoal advantage exp(3A(s, w.g)).
While this is not completely faithful approximation, we found in our experiments that it yielded similar results and even improved performance in some cases by reducing the absolute magnitude of the advantage term the exponent and therefore the "sharpness" of the resulting distribution, compared to the full expression that applies the discount factor / to V ((w.g).
21

---

F.2
Goal sampling
Following the standard hyperparameter settings in OGBench, we use separate sampling dis- tributions to train the value networks and actors. The value functions are trained with mixture distribution Puix (g s) where 20% of goals are simply set to the current state st, 50% are sampled from future states &t+k in the same trajectory, using geometric sampling where k ~ Geom(1 - 7), and 30% are sampled uniformly at random from the entire dataset D. On the other hand, the policy trained with future states sampled uniformly from the same trajectory.
F.3 Target policy
While Chane-Sam et al. [7] use an exponential moving average (EMA) of the online policy parameters 0 as the target policy prior Tg. we instead simply train a smaller policy network parameterized separately by &o on (sub)goals sampled from k steps into the future, where k is hyperparameter. We find that this leads to faster convergence, albeit with a small increase in computational complexity.
E4 Architectures
Apart from the goal encoder discussed below, all networks use 3-layer multilayer perceptrons (MLPs) of dimension 512 with layer normalization and Gaussian error linear units (gelus). For environments with pixel-based observations, we additionally use small Impala encoder [11] with a single Resnet block [20]. For value learning, we use an ensemble of two networks for both the online and target networks with double-Q learning as in the GCIVL implementation. Importantly, only the full policy and not the target subpolicy actor suu uses the goal encoder.
Goal encoder: During our experiments, we observed that the choice of network architecture for both the value function and policy networks had a significant impact on performance in several environments. Instead of taking in the raw concatenated state and goal inputs, HIQL processes the concatenated current observation and goal states with a representation module consisting of an additional three-layer MLP for state-based environments, followed by length-normalized linear projection to a 10-dimensional bottleneck representation. This is appended to another Impala encoder visual environments. The value and low-level policy networks receive this representation place of the goal information, as well as the raw (in state-based environments) or encoded (in pixel-based environments, using separate Impala encoder) state information. While SAW has no need for subgoal representation module, we found that simply adding these additional layers (separately) to the value and actor network encoders significantly boosted performance in state-based locomotion tasks, with modifications to the former improving training stability in pointmaze and modifications to the latter being critical for good performance in the antmaze and iamanoiddazze environments.
While we did not perform comprehensive architectural ablations due to computational limitations, we note that the desirable properties of the unit hypersphere as representation space well-studied in contrastive learning [54] and preliminary work by Wang et al. [53] has explored the benefits of scaling network depth for GCRL (albeit with negative results for the offline setting). Further studying the properties of representations emerging from these architectural choices may inform future work in representation learning for offline GCRL.
22

---

G Hyperparameters
We find that our method is robust to hyperparameter selection for different horizon lengths and environment types in locomotion tasks, but is more sensitive to choices of the value learning expectile parameter and the temperature parameter of the divergence term in manipulation tasks (see Appendix I.2 for training curves of different 3 settings). Unless otherwise stated in Table 2, all common hyperparameters are the same as specified in Park et al. [37] and state, subgoal, and goal-sampling distributions are identical to those for HIQL.
Emirrnment Type
Dutaset
Expectile
AWR
KLD
Subgoal steps 2
poincasze modiun narigate vo
0.7
3.3
3.0
25
pointaaze
pol poinraaze-- Jarge-------eee--- tarigggggg go
077
3.0
3.0
20
pol nowssee-eegggg antt-narr gate-so
07
3.0
3.0
25
unteace-eediiue www.eaa-seesuun--aiitteeee...
0.7
8.3
3.0
aetrade
Parggee-large--ee nart gate-co
3.0
anteazee-ee gtant---nttttttt gate-co
3.0
buzanoidaaze
busanoidaaze- large sarigatt-rga
buzanoidaaze giant navigate-
on-norigate---
-antzaoze-g" antt-neelgeea--
paaggelllyy-eeee 4ouble
0.3
scene- play-vl -V
0.7
1.0
Vieual cube-cueeeeeeee -iggl--------
0.7
3.0
3.0
Visual-cube
vieual -el-----ebbhleeeee e-play--0
07
3.0
3.0
10
viausl viausl-cnbe-tt -etl---ebb-tppii e-pla--1
0.7
3.0
3.0
Visuel-scese
(aaull-scusa-play-00
0.7
8.3
3.0
Table 2: SAW hyperparameters. Each cell indicates the hyperparameters for the corresponding environment and dataset. From left to right, these hyperparameters are: the expectile parameter T for GCIVL, the one-step AWR temperature (used for training both the target and policy networks), the temperature on the KI. divergence term 8, and the number of subgoal steps k:
23

---

H Waypoint Advantage Estimation
In this section, we report results for goal-conditioned waypoint advantage estimation (GCWAE), which uses subgoal generator to produce an advantage estimator. Our implementation uses the exact same value and high-level policy training objective and architecture as HIQL, but instead evaluates the advantage of dataset actions with respect to "imagined" subgoals en route to the goal to provide a clearer signal for policy learning [Equation 5J. We show that this approach is able to achieve significantly better performance than its one-step counterpart (GCIVL) in long-horizon locomotion, but continues to lag behind HIQL, and struggles in manipulation tasks. In our experiments, we found that all training statistics nearly identical to those of HIQL, except for the one-step action advantage evaluated with respect to the subgoals, as shown in the figures below.
revipats
part yo Man Mww.
Satatede Steae
Trakiigg
Sucre Pale
Tasto Steme
Toining Stere
Training curves for scene-play, -atmazo-large-navige and humanoid- giigg---nnnnnnaaeee the left, and the mean one-step advantage over dataset ac- tions with respect to subgoals on the right.
Figure 4:
24

---

I
Ablations
In this section, ablate various components of our objective and assess its sensitivity various hyperparameters.
1.1 One-step AWR ablation
We ablate the one-step AWR term in objective, which is akin to training purely bootstrapped policies from a target policy (which itself is trained with AWR). Note that ablating the bootstrapping term simply recovers the GCIVL baseline. We observe that ablations the one-step term primarily affect performance in short-horizon, stitching- heavy tasks such as the simpler manipulation envi- ronments. On the other hand, performance is largely unaffected in longer-horizon manipulation and locomotion tasks, confirming our initial hypotheses that the bulk of SAW's s performance in more complex tasks due to policy bootstrapping rather than one-step policy extraction.
10 boikdiaa Samuudy
naning Steze
Tuiiing Sope
Figure 5: Training curves for cube cube-single-ples and sate--doube-playy with one-step ablations.
MAN
Yalking staps
Figure Training curves for scene-ple plla and an antmaze- larzzz--iarg--naigatt with one-step ablations.
25

---

1.2 Hyperparameter sensitivity
Here, investigate SAW's sensitivity to the inverse temperature hyperparameter. We observe similar pattern to the one-step AWR ablation experiments, where the manipulation environments are much more sensitive to hyperparameter settings compared more complex, long-horizon tasks.
single-piy
Figure 7: Training curves for cube-single-play and cube- dooo-dooueelllleeeee play with different values of (where the default hyperparameters settings are | 0.3 and B 1, respectively).
Telding Steee
Figure 8: Training curves for scene-plas play and antmaze-large-nee -aaarggennvvigttt with different values of (where the default hyperparameters settings are 3 1 and / respectively).
Stitching Experiments
We report SAW and RISofl' performance on the OGBench stitch datasets below, which are de- signed to test an algorithm's 's ability stitch together actions from many short, suboptimal trajectories:
Eavironment
Datasel
GCBC GCIVL
GCIQL
QRL
CRL
HIQL
RIS
SAW
pointmaze medium stitch v0
23 11s
70
131
21
50 :12
0.13
74
78
87
pointmaze- stitch-rr
12
31
84
17
pointraze- wwwaaaee---aa--eeee giant-eti
animaze- netnaaeeneediun- atiich-v antnaze Large stitch- vg
29
18
7.
18
11 44
67 1.
=s
66 19
antoaze
antnaze giant stitch--ve vo
0.10
0.13
0.18
0.10
2
0 in
1
antnaze uninaze-nedieee meddnn------- explore-v0
2
19
13
1as
3
37
31
25
Large- explere-vo
wodiun- stitch-vo
draize
draaze- large-ati itii-th--
3
hunanoi husanoidnazee-e dnazze-giiiiie.o www.:a/eee.iittttttieeee stt--sit
0.
0.
0.
0.
4
Table 3: Evaluating SAW on stitch datasets state-based environments. We compare our method's average (binary) success rate (%) against the numbers reported in Park et al. [37] across the five test-time goals for each environment, averaged over 8 seeds (1 seeds for pixel-based visual tasks) with standard deviations after the + sign. Numbers within 5% of the best value in the row are in bold.
While SAW achieves competitive performance with HIQL pointmaze-stitch and antmaze-ste stitch, it lags behind in the explore datasets and humanoidmaze environments, sug- gesting that the generalization and stitching properties of generative models are increasingly important when data is more suboptimal or low-coverage (in high-dimensional spaces).
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We support our primary claims by providing a novel offline GCRL algorithm that does not subgoal generator, derivation of existing methods (HIQL and RIS) from the same inference- based framework, and experiments demonstrating that subgoals limit performance in long-horizon, high-dimensional control.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Limitations of this work are discussed in Section 7.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
27

---

Answer: [Yes]
Justification: This paper provides derivations of existing methods as well as the proposed algorithm, with all theorems and formulas referenced either in the main body in the full derivations included in Appendix B. Assumptions and limitations of those assumptions are discussed in Sections 5.1 and 7.
Guidelines:
The answer NA means that the paper does not include theoretical results. the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification; All algorithmic details and hyperparameter settings [Appendix G] are disclosed in the paper, and anonymized code is attached as supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
28

---

reproducible, they 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: Anonymized code attached in the supplementary materials, and a link to an open-source implementation will be added to the abstract after the peer review process.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /nips.
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All training and test details are either specified in the paper or follow the default conventions of open source benchmarks.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: seeds.
All experiments report the standard deviation of the results across random
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
29

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Compute resources are detailed in Appendix E.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: : /at::///neuuiiie /neuriss....p// cc cctppblicc/eeeeecceecceecceeeeee cawudeeeeelilleeeeeneneeenee
Answer: [Yes]
Justification:
The research conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper is primarily concerned with foundational reinforcement learning algorithms, with experiments performed in simulated robotic environments.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
30

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We utilize the open source benchmark OGBench [37] with appropriate citations and provide a link in the Experiments section.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
31

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithooe thcode. com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
32

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: https////eeurs..eeeeeeeee lbtte://herrig.cc/oofeeeeeee / for what should or should not be described.
33


Reviews:
[{"review_id": "vu4tUJ09Tb", "reviewer": "Reviewer_eAgf", "summary": "Some of the best approaches in long-horizon offline GCRL tasks (like HIQL) are hierarchical methods that generate subgoals with a high-level policy, and then use a low-level policy to reach subgoals. This paper presents a method (SAW) that performs well on long-horizon offline GCRL without the need for a subgoal generator.\n\nThe authors analyze a popular method for offline GCRL, HIQL, and identify benefits arising from a better signal-to-noise ratio by using waypoints, as well as from sampling nearby waypoints from the dataset. Instead of predicting subgoals at test time like HIQL, SAW distills or bootstraps waypoint policies into a flat policy at training time itself.\n\nExperiments show that SAW performs as well as or better than HIQL on locomotion and manipulation tasks considered from OGBench.", "questions": "My main concerns are expressed in “Strengths and Weaknesses”. Below are some additional questions -\n \n- It is unclear whether policy bootstrapping differs from the distillation of policies. If not, this has been done in other contexts, such as expert iteration (https://arxiv.org/pdf/1705.08439) and other works.\n- Where does Q come from in Equation (3), is it just $r + V(s’) - V(s)$?\n- According to Table 2, shouldn’t all results for cube (and scene) be asterixed as they use different values of $\\beta$ from the other tasks? Shouldn’t HIQL also be given a similar benefit?\n- Why does HIQL do better on the visual cube than the state-based version, similarly for the scene tasks?\n- Given that HIQL is the main baseline for the proposed approach, did the authors compare performance on other environments considered in the HIQL paper (like Roboverse)?\n- Would the authors expect scenarios where having an explicit subgoal generator would lead to better performance due to the generalization properties of the subgoal generator?", "limitations": "Discussed in Appendix A.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n- Offline GCRL is a crucial practical setting for problems like robotics. Improving and simplifying prominent methods, such as HIQL, would interest many researchers in the NeurIPS community.\n- The analysis to identify reasons for the success of hierarchical methods in offline GCRL (sections 4.2 and 4.3) helps understand the motivation and builds a coherent thread to support the paper. Figure 1 is also quite neat for building intuition about the considered methods.\n- The proposed method, SAW, demonstrates strong empirical performance. It performs as well as or better than HIQL (the current best) on locomotion and manipulation tasks considered from OGBench.\n\n### Weaknesses\n\n- The primary weakness of the current submission is that it is unclear whether SAW is significantly simpler than HIQL. One would expect that eliminating the hierarchical nature of HIQL and adopting a flat policy would simplify training, but this does not appear to be the case from a direct comparison of Algorithm 1 from this paper and the HIQL paper. Both methods require learning a value function and the sub-policies in the same manner. The difference is that SAW has a distillation loss instead of a subgoal generator loss. Both setups involve 3 sets of parameters and associated losses, so it is hard to compare the simplicity at training time, at least. Even though the subgoal generator from HIQL has been removed, there is still a notion of separate sub-policies, which constitutes a kind of hierarchy. I am curious to hear the authors’ thoughts regarding this issue.\n\n- Given that the paper is developed with the motivation of obtaining the benefits of HIQL without a hierarchical method, it requires more background information (in the main text) on the precise equations and updates used in HIQL. The paper provides an intuitive explanation of how HIQL works and includes some details; however, a comparison of HIQL and SAW with relevant equations would be helpful to eliminate confusion. For example, while Figure 1 states that HIQL imagines subgoals $w$, lines 163-164 indicate that they are sampled from the dataset, not imagined (unlike GCWAE).", "was_revised": true, "final_justification": "Thank you for the detailed response and answers to my questions. Thanks for the clarification regarding W1; I would suggest that the authors explicitly mention that the paper does not claim to reduce training complexity, as it may be a natural way to interpret the idea of \"flattening the hierarchy\" and removing the need for a sub-goal generator.\n\nThe response addresses my concerns and I have updated my score accordingly.", "rebuttal": "We are glad that the reviewer appreciated our motivating analyses of hierarchical methods and our painstaking work in Adobe Illustrator on Figure 1! In our response and revisions, we emphasize our core contribution: SAW is less about simplicity, and more about **resolving a fundamental tradeoff between tractable subgoal generation and asymptotic performance** in hierarchical methods, which allows it to scale to extremely long-horizon, high-dimensional tasks. Additionally, we have (**1**) summarized changes to the paper to directly and mathematically elucidate the relationship between SAW and HIQL, (**2**) performed additional hyperparameter tuning on HIQL for fairness of comparison in `cube-single-play`, and (**3**) addressed all reviewer questions, including a differentiation between distillation and bootstrapping.\n\n**Weaknesses**\n\n**W1(a)**: *The primary weakness of the current submission is that it is unclear whether SAW is significantly simpler than HIQL.*\n\n- We would like to clarify that we do not claim to reduce training complexity in this paper — instead, we argue that the key bottleneck of hierarchical GCRL methods is the use of **subgoals as an interface** between levels of the hierarchy. The issues associated with subgoal prediction lead to a number of additional challenges, including choosing a subgoal representation learning objective, verifying subgoals for reachability and realism [1, 2, 3, 4], and dealing with non-stationary optimization dynamics in the low-level actor [5, 6], which must condition on continually changing subgoal representations.\n- Our core contribution is an algorithm that can **bypass all of these problems** by removing subgoal generation and “flattening the hierarchy.” In our results [Figure 3], we demonstrate a fundamental tradeoff of hierarchical methods: low-dimensional subgoal representations are necessary for tractable subgoal prediction, but can limit the performance of the low-level policies tasked with reaching those subgoals. Without these constraints, SAW is able to surpass HIQL and achieve the first instance of non-trivial success in the extremely long-horizon, high-dimensional `humanoidmaze-giant` environment, among others.\n\n**W1(b)**: *Even though the subgoal generator from HIQL has been removed, there is still a notion of separate sub-policies, which constitutes a kind of hierarchy.*\n\n- While we do learn a separate “low-level” target policy, we emphasize that this is a design choice for training speed and stability rather than some hierarchical aspect of SAW  — we could instead acquire bootstrapping targets from the (same) flat SAW policy conditioned on the sampled subgoals. As for inference, this target policy is discarded and only the flat policy, conditioned directly on the final goal, is used.\n\n**W2(a)**: *Given that the paper is developed with the motivation of obtaining the benefits of HIQL without a hierarchical method, it requires more background information (in the main text) on the precise equations and updates used in HIQL. The paper provides an intuitive explanation of how HIQL works and includes some details; however, a comparison of HIQL and SAW with relevant equations would be helpful to eliminate confusion.*\n\n- We certainly agree that a simpler and more direct comparison in the main text would improve clarity. We have merged the explanations of RIS and HIQL into one section, with explicit comparisons between the relevant equations.\n\n**W2(b)**: *For example, while Figure 1 states that HIQL imagines subgoals, lines 163-164 indicate that they are sampled from the dataset, not imagined (unlike GCWAE).*\n\n- HIQL samples from the $k$-step future state distribution of the dataset $w \\sim p^{\\mathcal{D}}\\left(s_{t+k} \\mid s_t\\right)$ to train its subgoal generator $\\pi^h$, and then samples from $\\pi^h$ during inference to get subgoals for the low-level policy. In this context, we would say that HIQL is “imagining” subgoals during **inference**, while GCWAE and RIS imagine them during **training**. We agree that the language is indeed confusing, and have clarified this point accordingly.\n\n**Questions**\n\n**Q1**: *It is unclear whether policy bootstrapping differs from the distillation of policies. If not, this has been done in other contexts, such as expert iteration and other works.*\n\n- Policy bootstrapping differs from policy distillation in that bootstrapping targets are not conditioned on estimates of the same quantity — in the case of Expert Iteration (ExIt), both the expert MCTS player and the apprentice estimate the same action $a$ conditioned on the same state $s$, and one estimate of $\\pi(a\\mid s)$ is distilled into the other to leverage the complementary strengths of MBRL and MFRL. On the other hand, policy bootstrapping uses the subgoal structure of GCRL to update the policy $\\pi(a\\mid s,g)$ conditioned on the full goal $g$ using the action distribution of a policy $\\pi(a\\mid s, w)$ conditioned on a **different** subgoal $w$.\n- As an analogy, consider the classic temporal difference (TD) update for value learning: $V(s) \\leftarrow r(s,a)+\\gamma V(s')$. We would refer to this as bootstrapping because we use an estimate of a **different** state’s value $V(s')$ to update $V(s)$, whereas in distillation, we would simply imitate another agent’s estimate of the same state-value $V(s)$.\n\n**Q2**: *Where does Q come from in Equation (3), is it just $r+V\\left(s^{\\prime}\\right)-V(s)$?*\n\n- Yes, this is exactly correct — GCIVL uses a value-only estimator of the advantage $A(s,a,g) = Q(s,a,g) - V(s,g)$ where the $Q(s,a,g)$ is estimated as $r(s,g) + \\gamma \\overline V(s',g)$. We have clarified this in the Offline value learning section (lines 105-113) in our updated draft, thanks for pointing out this omission!\n\n**Q3**: *According to Table 2, shouldn’t all results for cube (and scene) be asterixed as they use different values of from the other tasks? Shouldn’t HIQL also be given a similar benefit?*\n\n- We note that all of the OGBench baseline algorithms, including HIQL, were individually tuned **per task category** (e.g., locomotion vs. cube manipulation vs. scene manipulation, state- vs. pixel-based observations, etc.) to achieve the reported results. Indeed, HIQL uses a different step size for manipulation vs. point and ant locomotion vs. humanoid locomotion tasks (10 vs. 25 vs. 100 steps), and adjusts the high- and low-level $\\alpha$ inverse temperature parameters for different datasets. Our method uses the same step sizes as HIQL for all environments and only adjusts the KLD $\\beta$ to a single value for all manipulation tasks (the only exception being the asterixed `cube-single` environment, as mentioned).\n- For fairness, we ran experiments with HIQL on `cube-single-play` using the expectile $\\tau=0.9$ and tuned the high-level inverse temperature $\\alpha$, reaching a success rate of $44\\tiny{\\pm9}$, compared to $72 \\tiny{\\pm5}$ for SAW and $15\\tiny{\\pm3}$ for the original HIQL. We will update Table 1 accordingly.\n\n**Q4**: *Why does HIQL do better on the visual cube than the state-based version, similarly for the scene tasks?*\n\n- One previous hypothesis suggests that it has to do with difficulty in learning good subgoal representations from state information (see Appendix C of OGBench), and suggest alternative subgoal representation learning objectives as a future research direction. However, our implementation of RIS, which also uses subgoals, performs very well on state-based `cube-single-play`, suggesting that the culprit may lie elsewhere — perhaps in the fixed timescale of the hierarchical policy? We do not claim to understand this performance disparity, but believe that it is certainly an interesting direction for future work.\n\n**Q5**: *Given that HIQL is the main baseline for the proposed approach, did the authors compare performance on other environments considered in the HIQL paper (like Roboverse)?*\n\n- While we did not evaluate our method on Roboverse, we note that OGBench manipulation tasks are significantly more complex by several metrics — for example, SAW achieves a 25% higher success rate on `scene-play` than HIQL ($63\\tiny{\\pm}6$ vs. $38\\tiny{\\pm3}$), which requires up 4x more steps (400 vs. 100) and contains far more subtasks (2-8 vs. 1-2) than the longest tasks in Roboverse, according to a direct comparison between the two benchmarks in the OGBench paper [7]. We argue that this better evaluates the compositional long-horizon reasoning abilities of different algorithms.\n\n**Q6**: *Would the authors expect scenarios where having an explicit subgoal generator would lead to better performance due to the generalization properties of the subgoal generator?*\n\n- There may certainly be scenarios where a subgoal generator may be able to produce useful subgoals for out-of-distribution (OOD) goals or initial states during online evaluation. While this may improve empirical performance in some cases, we note that it is **impossible** to provide performance guarantees for such OOD generalization in the offline setting without strong assumptions on dataset coverage w.r.t. the test-time state distribution [8]. As such, we believe these generalization properties are outside of the scope of offline GCRL.\n\n**References**\n\n[1] Czechowski, K., et al. Subgoal Search For Complex Reasoning Tasks (2024).\n\n[2] Hatch, K. B., et al. GHIL-Glue: Hierarchical Control with Filtered Subgoal Images (2024).\n\n[3] Zawalski, M., et al. Fast and Precise: Adjusting Planning Horizon with Adaptive Subgoal Search (2024).\n\n[4] Zhang, T., et al. Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learning (2020).\n\n[5] Levy, A., et al. Learning Multi-Level Hierarchies with Hindsight (2019).\n\n[6] Vezhnevets, A. S., et al. FeUdal Networks for Hierarchical Reinforcement Learning (2017).\n\n[7] Park, S., et al. OGBench: Benchmarking Offline Goal-Conditioned RL (2025).\n\n[8] Park, S., et al. Is Value Learning Really the Main Bottleneck in Offline RL? (2024)."}, {"review_id": "RV2nfLm1hH", "reviewer": "Reviewer_6htN", "summary": "This paper proposes a method to bring some of the benefits of hierarchical RL to offline goal-conditioned RL without the additional overhead of HRL, which the authors call Subgoal Advantage-Weighted Policy Bootstrapping (SAW). They build the description of their method by explaining two prior methods, RIS (a flat method) and HIQL (a hierarchical method), and showing how SAW is different and addresses limitations of these methods, deriving their optimization target based on an advantage function rather than a learned subgoal generator. The paper then presents results for SAW's effectiveness in several environments with a large number of offline datasets for each, showing favorable performance compared to other GCRL methods.", "questions": "* Can you expand upon the justification for simplication of the advantage function (line 208)? It is not clear to me why this should be an effective replacement.\n* Can you better explain the posterior over waypoints $q({w} | g)$ introduced in Line 216? What does this represent intuitively?\n* How do we transition to a fixed target subpolicy $\\pi^{sub}$ in Line 218?\n* SAW removes some models (e.g., the subgoal generator) and adds others (e.g, $\\pi^l(a | s, w)$ mentioned in line 232). Overall, how does this affect the total size of the models compared to other methods? The compute needed for a training or inference pass?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "# Strengths\n* The paper is generall well-written and organized. \n* The paper seems well-placed in the literature, with a thorough discussion of existing methods, some of their limitations, and how SAW improves on them.\n* The investigation into the benefits of HRL in GCRL is interesting and others may find it of use. In particular, the discussion of improved exploration and multi-step rewards, which are often cited as the reason for HRL's benefit in online RL, being not present or not used by some effective methods in offline RL, was good.\n* SAW seems to provide a noticeable benefit over existing methods in most of the locomotion and manipulation domains used in the experiments.\n\n# Weaknesses\n* There is little discussion of the efficiency of SAW in terms of compute or time or model size. Intuitively, removing the training and use of the generative subgoal model and replacing it with a simpler objective should reduce these things as well, but that is not clear from the text.\n* The derivations are not always easy to follow. Some of this is likely brevity needed for paper length, but some things are unclear to me. See Questions for some examples.", "was_revised": true, "final_justification": "The authors addressed my concerns regarding some of the derivations and have promised to adjust them in the final version of the paper.\nThey also addressed my concern about the lack of discussion of the efficiency of SAW (in the model size and computation senses).\n\nAs these were the factors that made my choose Borderline Accept, I have raised my rating to Accept.", "rebuttal": "We are glad that the reviewer found our paper well-written and organized and appreciated our investigation into the benefits of HRL. Their questions have helped us clarify key details in our derivation. In our response and revisions, we address the reviewer’s concerns about computational efficiency by clarifying our core contribution: while SAW does offer simplicity in some sense (e.g., end-to-end inference), its primary goal is to **resolve a fundamental tradeoff between tractable subgoal generation and asymptotic performance** in hierarchical methods. This is what allows SAW to scale to extremely long-horizon, high-dimensional tasks. However, we do provide a detailed analysis of SAW’s compute and model sizes relative to its hierarchical counterpart, and answer all derivation-related questions below.\n\n**Questions**\n\n**Q1**: *Can you expand upon the justification for simplification of the advantage function (line 208)? It is not clear to me why this should be an effective replacement.*\n\n- GCRL typically uses a sparse reward formulation where the reward is always constant (typically $-1$) unless at the goal — in other words, $r(s,g) = -1$ unless $s=g$, in which case $r(s,g)=0$. While the full advantage estimate would be $A\\left(s_t, w, g\\right)=-V\\left(s_t, g\\right)+\\gamma^k V(w, g)+\\sum_{t^{\\prime}=t}^{k-1} r\\left(s_{t^{\\prime}}, g\\right)$, the sum over rewards is almost always a constant unless the goal was reached between time $t$ and $t+k-1$. Additionally, it is standard to use a goal sampling distribution that samples goals from future states $s_{t'> t+k}$ more than $k$ steps into the future (of the same trajectory), so the probability of the same goal state appearing earlier in the trajectory is extremely low. Finally, the discount on $\\gamma^k$ is always a constant, and can be roughly folded into the inverse temperature parameter $\\beta$ in the subgoal advantage weight $\\exp(\\beta A(s,w,g))$.\n- While this is not a completely faithful approximation, the authors of HIQL found empirically that both choices yielded similar results. We even found in our experiments that it improved performance by reducing the absolute magnitude of the advantage term in the exponent, compared to applying the discount factor $\\gamma^k$ to $V(w,g)$. We will add this rationale in the appendix to justify these choices.\n\n**Q2**: *Can you better explain the posterior $q(w\\mid g)$ over waypoints introduced in Line 216? What does this represent intuitively?*\n\n- The variational distribution $q(\\\\{w\\\\}\\mid g)$, which factors into $q(w_t\\mid s_t, g)$, follows the definition in [1] and represents the optimal sample-based distribution over waypoints. For a more concrete explanation, consider the optimization of the high-level subgoal generator $\\pi^h$ in HIQL: we can think of $q$ as a non-parametric “target” distribution  $q(w\\mid s,g)\\propto \\pi^h(w\\mid s,g) \\exp(\\beta A(s,w,g))$ constructed from samples of $s,w,g \\sim \\mathcal D$, where $\\beta$ is an inverse temperature hyperparameter. Having this $q$ allows us to iteratively optimize our objective $\\mathcal J(\\pi, q)$ — the ELBO on the joint optimality likelihood — via alternative coordinate ascent in $q$ and $\\pi^h$. This is analogous to an expectation-maximization (EM) algorithm, where we alternate between estimating a non-parametric distribution $q$ from samples (the expectation step), then minimizing the KL divergence between our parametric policy $\\pi^h$ and the variational $q$ (the maximization step).\n- We have thoroughly revised our derivations in both Appendix C and Section 5 to clarify the roles of the non-parametric $q$ and parametric policy $\\pi$ and make our variable choices consistent with prior work [1, 2].\n\n**Q3**: *How do we transition to a fixed target subpolicy $\\pi^\\mathrm{sub}$ in Line 218?*\n\n- The target subpolicy $\\pi^\\mathrm{sub}$ is simply a part of the factored form of variational distribution $q$, which we design to factor in the same way as $p_\\pi(\\tau \\mid g)$ (see lines 210-212). This is the same form of variational distribution as used in our derivation of HIQL, but with a fixed subpolicy in place of the low-level policy $\\pi^\\ell$. This choice is both for correctness with respect to RIS, which uses a separate “prior” policy parameterized by an exponential moving average of online policy parameters; and for brevity, since it allows us to focus on the optimization of the waypoint optimality $p_\\pi(U=1)$ when deriving the RIS and SAW objectives. We use the separate notation of $\\pi^\\mathrm{sub}$ to emphasize that it is not part of the optimization, unlike $\\pi^\\ell$ in our HIQL derivation.\n\n**Q4**: *SAW removes some models (e.g., the subgoal generator) and adds others (e.g, $\\pi^\\ell(a\\mid s, w)$ mentioned in line 232). Overall, how does this affect the total size of the models compared to other methods? The compute needed for a training or inference pass?*\n\n- Similar to our response to reviewer eAgf, we would like to clarify that we do not claim to reduce training complexity in this paper — instead, we argue that the key bottleneck of hierarchical GCRL methods is the use of **subgoals as an interface** between levels of the hierarchy. The issues associated with subgoal prediction lead to a number of additional challenges, including choosing a subgoal representation learning objective, verifying subgoals for reachability and realism [3, 4, 5, 6], and dealing with non-stationary optimization dynamics in the low-level actor [7, 8], which must condition on continually changing subgoal representations.\n- Our core contribution is an algorithm that can **bypass all of these problems** by removing subgoal generation and “flattening the hierarchy.” In our results [Figure 3], we demonstrate a fundamental tradeoff of hierarchical methods: low-dimensional subgoal representations are necessary for tractable subgoal prediction, but can limit the performance of the low-level policies tasked with reaching those subgoals. Without these constraints, SAW is able to surpass HIQL and achieve the first instance of non-trivial success in the extremely long-horizon, high-dimensional `humanoidmaze-giant` environment, among others.\n- However, to address the reviewer’s original question, we provide an in-depth look at the complexity of each stage below:\n    - **Training**: the training procedure for value learning and the low-level/target subpolicy are exactly the same between SAW and HIQL. HIQL’s high-level policy simply performs another AWR training step to predict encoded subgoals, requiring forward passes through the goal representation encoder and the high-level policy. On the other hand, SAW computes a standard 1-step AWR loss along with the bootstrapping loss, which adds additional training complexity. However, we show in our ablations of this term (Appendix H.1) that this additional term is **critical for performance** in shorter-horizon, stitching-heavy environments such as `cube-single-play`, in which SAW shows large improvements over HIQL ($72 \\tiny{\\pm5}$ vs. $15\\tiny{\\pm3}$ success rate — although we were able to improve HIQL’s performance to $44\\tiny{\\pm9}$ through additional tuning, for fairness).\n    - **Inference**: for inference, the model size and architecture of HIQL’s low-level actor $\\pi^\\ell$ and SAW’s (flat) actor $\\pi$ are exactly the same. During inference, HIQL’s low-level actor does not need to use its goal representation encoder, since the high-level actor $\\pi^h$ outputs an already-encoded goal representation. Therefore, the main comparison in inference is the compute required by HIQL’s high-level actor compared to the goal encoder of SAW’s actor (see Appendix F for an analysis of the additional encoder). Both take in the raw observation and pass it through a two-layer MLP (with a prepended Impala encoder in visual environments), resulting in a low-dimensional representation — then, they both perform a length-normalization step. In fact, **there is actually no difference in inference-time computational complexity between the two**. In tests on an NVIDIA 3090 GPU, we found less than a 0.01 ms difference in the average per-step inference speed between the two methods.\n\n**References**\n\n[1] Abdolmaleki, A., et al. Maximum a Posteriori Policy Optimisation (2018).\n\n[2] Peng, X. B., et al. Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning (2019).\n\n[3] Czechowski, K., et al. Subgoal Search For Complex Reasoning Tasks (2024).\n\n[4] Hatch, K. B., et al. GHIL-Glue: Hierarchical Control with Filtered Subgoal Images (2024).\n\n[5] Zawalski, M., et al. Fast and Precise: Adjusting Planning Horizon with Adaptive Subgoal Search (2024).\n\n[6] Zhang, T., et al. Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learning (2020).\n\n[7] Levy, A., et al. Learning Multi-Level Hierarchies with Hindsight (2019).\n\n[8] Vezhnevets, A. S., et al. FeUdal Networks for Hierarchical Reinforcement Learning (2017)."}, {"review_id": "grslB9B288", "reviewer": "Reviewer_aGY8", "summary": "This paper proposes a novel method for offline goal-conditioned reinforcement learning (GCRL) called Subgoal Advantage-Weighted Policy Bootstrapping (SAW). It focuses on the challenge of scaling flat GCRL policies to long-horizon tasks without relying on hierarchical RL (HRL). SAW avoids learning generative models by bootstrapping on real, near subgoals from the dataset using advantage-weighted importance sampling. Across 20 diverse locomotion and manipulation tasks (from OGBench), SAW achieves state-of-the-art or near-SOTA performance and shows strong scalability to long horizons and high-dimensional visual observations.", "questions": "I'm curious about the potential challenges that may arise when applying this method in an online setting. Specifically, how would the reliance on real subgoals from dataset trajectories adapt to a setting where data is collected and updated continuously?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- I deeply appreciate the flow of the paper; the writing is clear, elegant and thoughtful.\n- The proposed method, SAW, is technically solid and shows consistent improvements over strong baselines, particularly in challenging long-horizon settings such as humanoidmaze-giant.\n- Removed computationally heavy subgoal generators, as well as complex hierachical architectures that requires intensive manual design and paramter tuning.\n\nWeaknesses:\n- I did not find any significant weaknesses. While some sensitivity to hyperparameters (e.g., τ and β) is observed and acknowledged by the authors, it appears manageable and does not detract from the overall contribution.", "was_revised": true, "final_justification": "As is stated in my initial review, there is no significant weakness that I observed and my question is well addressed in the rebuttal phase.", "rebuttal": "We thank the reviewer for their recognition of our method’s strengths and performance, and are glad that they appreciated the flow of the paper! We provide a (perhaps excessively) detailed answer to their question below.\n\n**Questions**\n\n**Q1**: *I'm curious about the potential challenges that may arise when applying this method in an online setting. Specifically, how would the reliance on real subgoals from dataset trajectories adapt to a setting where data is collected and updated continuously?*\n\nWhile we have not tested SAW in the online setting, we are optimistic that online extensions of our ideas have significant potential for improved training stability and scalability, and provide our rationale below:\n\n- **Extrapolation error**: While offline RL methods do not often transfer well to online RL due to over-conservatism leading to poor exploration, SAW is designed to mitigate extrapolation error on **subgoals**, which we argue is significantly **more harmful** than error on states and actions. States and actions are grounded in the observation and (bounded) action spaces of the environment, and the only harm is in overestimating the value of a given state-action pair. In the online setting, such optimism in the face of uncertainty [1] can actually **improve** exploration.\n    - On the other hand, approximation error in a subgoal generator may produce target states that are not reachable by the current policy or even within space of physically plausible states, leading to arbitrary behavior by the low-level policy. Indeed, [2] uses a form of AWR (commonly used in offline RL) to train a high-level policy in the **online** setting, which points to the importance of staying within the dataset distribution of realistic subgoals.\n    - We believe that the success of such “conservative” subgoal-based methods provides an optimistic outlook for extensions of SAW to the online setting, where we can completely avoid extrapolation error and use the subgoal advantage-weighting to selectively bootstrap on the best intermediate waypoints that become **progressively more optimal** as on-policy data is collected and updated.\n- **Subpolicy training stability**: Another well-known problem of hierarchical policies is the introduction of non-stationaries in the low-level actor [3, 4], which must condition on **continually changing subgoal representations** over the course of training. We believe that the stability benefits of SAW, which does not require such representations, would be especially beneficial to the online setting where the performance of the low-level subpolicy continuously affects the quality of the generated data.\n\n**References**\n\n[1] Brafman, R.I & Tennenholtz, M. R-Max — A General Polynomial Time Algorithm for Near-Optimal Reinforcement Learning (2002).\n\n[2] Chane-Sane, E., et al. Goal-Conditioned Reinforcement Learning with Imagined Subgoals (2021).\n\n[3] Levy, A., et al. Learning Multi-Level Hierarchies with Hindsight (2019).\n\n[4] Vezhnevets, A. S., et al. FeUdal Networks for Hierarchical Reinforcement Learning (2017)."}, {"review_id": "1IaQ1ucI88", "reviewer": "Reviewer_XN7s", "summary": "The authors propose SAW, a new offline RL algorithm to learn a goal-conditioned policy. SAW is inspired from HIQL and RIS, but manages to avoid sampling from a distributional goal generator. It manages to outperform several strong methods on the recently released OGBench benchmark.", "questions": "*Questions:*\n- Is it possible to use Hindsight Experience Replay on top of SAW? If not, why? If yes, why not considering it in this work?\n- would you agree that replacing the goal generator in RIS by sampling future states along trajectories and using it offline would exactly define SAW? \n- how does SAW perform on the Ogbench stitch and explore dataset? If your answer that SAW is only meant to address navigate-like datasets, then this has to be made more explicit in the paper.\n\n*Major issues:*\n\nA lot can be done to improve clarity:\n- the methods being sophisticated, a better way to provide both high-level intuitions and their rigorous mathematical treatment has to be found. Under the current form, the paper proposes an unstructured mix of both levels, which makes it hard to read.\n- the RIS and HIQL algorithms should be presented in some background section\n- The introductory paragraph of Section 5 concludes with the limitations of RIS, it should conclude with an overview of the SAW approach.\n- The beginning of 5.1 cannot be understood without first reading Appendix C. For instance, \"the KL bootstrapping term for RIS’s flat policy\" is not explained in the main paper. Again, a quick presentation of RIS and HIQL in the background could help a lot\n- it is not clear why RIS (or its offline variant) does not appear as a baseline in Table 1 nor in Appendix D\n- some important variables are not defined: what does $O_t$ stand for?\n- for policies, we have $\\pi^l$, $\\pi^h$, $\\pi^l_\\theta$, $\\pi^h_\\psi$, $\\pi_\\theta$, $\\pi_\\psi$, $\\pi^{sub}_\\psi$, $\\pi^{sub}$, do we really need so many notations?\n- to me a core insight of the paperis in lines 593 -> 594, but this is not enough put forward.\n\n*Minor issues:*\n- Appendix H (ablations) is not mentioned in the main text, though it may convey some important conclusions\n- In Equations (6) and (7), shouldn't there be an expectation over the joint distribution p(s,a,w,g) instead of p(s,a,w) p(g) ? Otherwise, you might sample a goal g that is not reachable from state s via subgoal w, meaning the goal g does not belong to the same trajectory as (s,a,w).\n- the notion of Q function is used throughout the paper, but not explained in the problem setting. \n- nothing is specified in terms of architectures of the V functions, Q functions, etc.\n\n*Typos and polishing:*\n- when an equation closes a sentence, it should finish with a dot (lines 533 and 576)\n- to refer to an equation, use \\ eqref instead of \\ ref\n- line 268: the Park et al. [31] -> remove \"the\"\n- line 553: the mathematical term is not an Equation (an equality or inequality is missing)\n- line 588: the $p...$ -> remove \"the\"", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- the approach is interesting, the underlying intuition is well depicted with Fig. 1\n- the method shows excellent results on OGBench navigate-family datasets\n\nWeaknesses:\n- I had a very hard time understanding the method, clarity could be much improved\n- the relationship to RIS is not sufficiently explored\n- it is unclear whether the method only works well in the OGBench navigate-family datasets, or could be used in the stitch-family datasets too\n- the paper requires significant polishing\n- the code is included in the submission, but the authors do not pretend to open-source it upon acceptance", "was_revised": true, "final_justification": "I'm just reapeating my rebuttal comment here.\n\nAfter reading all the reviews and the rebuttal from the authors on these reviews, I found that the authors have satisfactorily addressed all the points raised, including my points.\n\nThe authors have promised to make many changes to their paper to address these points, I regret that the NeurIPS policy does not make it possible to see the resulting revised version, that could have triggered further discussion. In the absence of such revised version, I have no further question to raise and I'm now ready to change my evaluation into an accept, in accordance with the feeling of all other reviewers.\n\nI just hope that the authors will indeed make all the changes that they have promised, to make their paper as good as it should be.", "rebuttal": "We thank the reviewer for their comments, which have significantly improved the clarity and presentation of our paper. To address their concerns, we have (**1**) elaborated on the relationship between SAW and RIS, performed additional experiments to (**2**) incorporate RIS as a baseline and (**3**) demonstrate SAW’s strong performance on additional datasets, (**4**) made extensive clarity-related changes, and (**5**) corrected all mentioned typos. Regarding open-source concerns, we will add a link to the implementation after the review process, as promised in the Paper Checklist. In fact, an open-source implementation of SAW is **already available** to the community, but we cannot reference it here due to anonymity.\n\n**Q1**: *Is it possible to use Hindsight Experience Replay (HER) on top of SAW?*\n\n- SAW and all OGBench baselines **already use** HER, where goals are sampled from a mixture of the current state’s (future) trajectory, the uniform dataset distribution, and or simply set to the current state $g=s$. We have added these details to Appendix D.\n\n**Q2**: *would you agree that replacing the goal generator in RIS by sampling future states along trajectories and using it offline would exactly define SAW?*\n\n- Replacing goal generation in RIS with sampled future states is a key advantage of SAW, but **does not** recover the full SAW objective. Rather than indiscriminately bootstrapping from policies conditioned on any sampled future states $w$, SAW uses the subgoal advantage weight $\\exp(\\beta A(s,w,g))$ to flexibly tune the strength of the bootstrapping term based the advantage of $w$ with respect to $g$. RIS instead uses a subgoal generator to produce “imagined” subgoals and distills from all subpolicies equally, using a fixed weight on the bootstrapping term. This can be suboptimal in cases where there are no good waypoints in the dataset distribution $p(s_{t+k}\\mid s_t)$, since RIS cannot dynamically reduce the influence of the bootstrapping term relative to the one-step policy extraction objective.\n- Indeed, we highlight that there are key commonalities between SAW, RIS, and HIQL — in fact, one of the primary contributions of our paper is a **unified theoretical framework** for inference over the optimal waypoint posterior. From this, all three methods can be derived as different forms of bootstrapping on subgoal-conditioned policies: HIQL at test-time with a subgoal generator, RIS at training time with a subgoal generator, and SAW at training time with an advantage weight on dataset subgoals. We have emphasized this contribution in Section 5.\n\n**Q3**: *how does SAW perform on the Ogbench stitch and explore dataset?*\n\n- As discussed in our Limitations section, we expect the performance of SAW to degrade when the goal distribution $p(g)$ contains many goals for which sampled waypoints are highly suboptimal — a tradeoff for improved scaling to high-dimensional control and a point we will make more explicit in our revisions. In the OGBench `stitch` and `explore` datasets, which contain highly suboptimal trajectories, the default is to sample 50 or 100% of goals from the dataset at random, which are unlikely to coincide with good waypoints. Indeed, how best to sample actor goals remains an open problem in the field. Nevertheless, we find that **SAW (without tuning) matches HIQL’s state-of-the-art performance** across all `antmaze-stitch` datasets (SAW vs. HIQL success rate over 8 seeds: $96\\tiny{\\pm 1}$ vs. $94\\tiny{\\pm 1}$ on `medium-stitch`, $65\\tiny{\\pm 9}$ vs. $67\\tiny{\\pm 5}$ on `large-stitch`, $1\\tiny{\\pm 2}$ vs. $2\\tiny{\\pm 2}$ on `giant-stitch`), and plan to conduct more comprehensive experiments (including on the `explore` datasets) for an updated draft. We believe SAW’s performance on highly suboptimal data can be improved by using more principled strategies, such as Prioritized Experience Replay for goal sampling.\n\n**Major issue 5**: *RIS as a baseline*\n\n- We would like to clarify that RIS **does** appear in Appendix D (lines 626-639), where we detail modifications made to the original algorithm for offline, long-horizon GCRL.\n- We initially avoided a direct comparison in Table 1 for two reasons: (1) We believe that the number of algorithmic modifications required deviates from a faithful reproduction of the original work: the initial design using DDPG and target policy networks failed to reach meaningful performance in most environments, which are far larger than the original evaluations. (2) The original RIS paper did not perform evaluations on pixel-based locomotion. We find that using shared representations between the actor and value functions, which is critical for HIQL and SAW’s performance in these tasks, leads to frequent **training collapse in all `visual-antmaze` tasks** for RIS that we were unable to resolve.\n- To allow for explicit comparisons, we have run additional experiments with RIS, including hyperparameter tuning on the KL weight $\\beta$ which improves results on `antmaze-giant` ($\\beta=100$ below vs. $\\beta=3$ in Figure 3):\n$$\n\\\\begin{array}{lll}\n\\\\hline\n\\\\textbf{Dataset}&\\\\textbf{RIS}&\\\\textbf{SAW}\\\\\\\\\n\\\\hline\n\\\\texttt{pointmaze-medium}&88\\\\tiny{\\\\pm6}&\\\\mathbf{97}\\\\tiny{\\\\pm2}\\\\\\\\\n\\\\texttt{pointmaze-large}&63\\\\tiny{\\\\pm13}&\\\\mathbf{85}\\\\tiny{\\\\pm10}\\\\\\\\\n\\\\texttt{pointmaze-giant}&57\\\\tiny{\\\\pm12}&\\\\mathbf{68}\\\\tiny{\\\\pm8}\\\\\\\\\n\\\\texttt{antmaze-medium}&\\\\mathbf{96}\\\\tiny{\\\\pm1}&\\\\mathbf{97}\\\\tiny{\\\\pm1}\\\\\\\\\n\\\\texttt{antmaze-large}&\\\\mathbf{89}\\\\tiny{\\\\pm3}&\\\\mathbf{90}\\\\tiny{\\\\pm3}\\\\\\\\\n\\\\texttt{antmaze-giant}&65\\\\tiny{\\\\pm4}&\\\\mathbf{73}\\\\tiny{\\\\pm4}\\\\\\\\\n\\\\texttt{humanoidmaze-medium}&73\\\\tiny{\\\\pm5}&\\\\mathbf{88}\\\\tiny{\\\\pm3}\\\\\\\\\n\\\\texttt{humanoidmaze-large}&21\\\\tiny{\\\\pm7}&46\\\\tiny{\\\\pm4}\\\\\\\\\n\\\\texttt{humanoidmaze-giant}&3\\\\tiny{\\\\pm2}&\\\\mathbf{35}\\\\tiny{\\\\pm4}\\\\\\\\\n\\\\texttt{cube-single}&\\\\mathbf{81}^* \\\\tiny{\\\\pm6}&72^* \\\\tiny{\\\\pm5}\\\\\\\\\n\\\\texttt{cube-double}&36\\\\tiny{\\\\pm4}&\\\\mathbf{40}\\\\tiny{\\\\pm7}\\\\\\\\\n\\\\texttt{cube-triple}&3\\\\tiny{\\\\pm2}&\\\\mathbf{4}\\\\tiny{\\\\pm2}\\\\\\\\\n\\\\texttt{scene}&\\\\mathbf{64}\\\\tiny{\\\\pm7}&\\\\mathbf{63}\\\\tiny{\\\\pm6}\\\\\\\\\n\\\\texttt{visual-antmaze-medium}&55\\\\tiny{\\\\pm47}&\\\\mathbf{95}\\\\tiny{\\\\pm0}\\\\\\\\\n\\\\texttt{visual-antmaze-large}&43\\\\tiny{\\\\pm44}&\\\\mathbf{82}\\\\tiny{\\\\pm4}\\\\\\\\\n\\\\texttt{visual-antmaze-giant}&4\\\\tiny{\\\\pm1}&10\\\\tiny{\\\\pm2}\\\\\\\\\n\\\\texttt{visual-cube-single}&63\\\\tiny{\\\\pm37}&\\\\mathbf{88}\\\\tiny{\\\\pm3}\\\\\\\\\n\\\\texttt{visual-cube-double}&28\\\\tiny{\\\\pm6}&\\\\mathbf{40}\\\\tiny{\\\\pm3}\\\\\\\\\n\\\\texttt{visual-cube-triple}&18\\\\tiny{\\\\pm1}&\\\\mathbf{20}\\\\tiny{\\\\pm1}\\\\\\\\\n\\\\texttt{visual-scene}&38\\\\tiny{\\\\pm3}&\\\\mathbf{47}\\\\tiny{\\\\pm6}\\\\\\\\\n\\\\hline\n\\\\end{array}\n$$\n- We find that RIS is able to match HIQL in `antmaze` but suffers from increased reliance on subgoal representations in high-dimensional environments such as `humanoidmaze` and all `visual` variants, leading to significantly worse performance than either HIQL or SAW.\n\n**Clarity Issues**\n\n- **Improving flow (Major issues 1, 3, & 8)**:\n    - *Mixing intuition & mathematical rigor & Section 5 introduction*: We have rewritten the relevant sections to improve the logical flow as follows: (1) provide high-level intuition on hierarchies as policy bootstrapping in Section 4.4, (2) use this to reframe RIS as one instantiation in a new Section 4.5, (3) provide high-level introduction to SAW in the opening of Section 5, and (4) proceed with the theoretical analysis and derivation of our generator-free SAW objective. Please let us know if you have any additional suggestions!\n    - *Core insight in lines 593-594*: While this part of the derivation appears in lines 230-231, we agree that it was not sufficiently emphasized as the key insight for avoiding generative subgoal models. We have rewritten the surrounding text to remedy this.\n- **RIS and HIQL background (Major issues 2 & 4)**: We have replaced our informal background of RIS in the Related Works and HIQL in Section 4.1 with precise equations, referencing Figure 1 for intuition and making explicit comparisons.\n- **Notations (Major issues 6 & 7; Minor issues 2 & 3)**\n    - *What $O_t$ does stand for?* $O_t$ is a binary *optimality* variable commonly used in the RL as inference framework, defined as $p\\left({O}_t=1 \\mid s_t, a_t, g\\right)\\propto\\exp \\left(\\beta A\\left(s_t, a_t, g\\right)\\right)$. This definition allows us to reframe optimal control as inference over the posterior distribution of actions, conditioned on optimality ($O_t=1$) . We have added this definition to Appendix C.1.\n    - *Reducing the number of policy notations*: We have removed the parameter subscripts $\\theta$ and $\\psi$ for clarity. However, we retain the superscripts as they are important distinctions in our derivations (e.g., while $\\pi^\\ell$ is part of the optimization process for HIQL,  $\\pi^\\mathrm{sub}$ is a fixed target for RIS/SAW).\n    - *Expectation over* $p(g)$: We separate the goal distribution $p(g)$ since it may be independent of the current state, e.g., sampling from the uniform dataset distribution. While some goals may not be reachable from the current state, such goal sampling strategies are generally accepted practice, given that we don’t know *a priori* the full set of reachable goals per state.\n    - *Defining the $Q$ function*: We use a value-only estimator where $Q(s,a,g)$ is estimated as $r(s,g) + \\gamma \\overline V(s',g)$. We have clarified this in the Offline value learning section (lines 105-113).\n- **Ablations (Minor issue 1)**: We **do** reference Appendix H in Section 5.3 (line 247) to justify the addition of the one-step policy extraction objective, and have now added an additional reference to the $\\beta$ tuning curves in the Experimental section.\n- **Architectures (Minor issue 4)**: We have added architectural details to the Appendix. For fairness, all components of SAW are exactly the same as those of HIQL — the only difference is that we prepend HIQL’s goal representation module to SAW’s flat actor instead of its target subpolicy, as discussed in Appendix F."}]

Author rebuttal:

Meta-review: The paper considers goal-conditioned reinforcement learning (GCRL) and proposes a variant of the method HIQL which they call SAW which is significantly simpler because it avoids to define subgoals.  The paper is well written and explains well the underlying ideas.  Experiments show that SAW better than previous methods.  

The discussion between the reviewers and the authors clarified several important points, which lead to the rating "5 accept" for all four reviewers.
Decision: Accept (spotlight)