Horizon Reduction Makes RL Scalable
Authors: Seohong Park
Kevin Frans
Deepinder Mann
Benjamin Eysenbach
Aviral Kumar
Sergey Levine
Venue: NeurIPS 2025 spotlight ()
Keywords: reinforcement learning
offline reinforcement learning
Primary area: reinforcement_learning

Abstract: In this work, we study the scalability of offline reinforcement learning (RL) algorithms. In principle, a truly scalable offline RL algorithm should be able to solve any given problem, regardless of its complexity, given sufficient data, compute, and model capacity. We investigate if and how current offline RL algorithms match up to this promise on diverse, challenging, previously unsolved tasks, using datasets up to 1000× larger than typical offline RL datasets. We observe that despite scaling up data, many existing offline RL algorithms exhibit poor scaling behavior, saturating well below the maximum performance. We hypothesize that the horizon is the main cause behind the poor scaling of offline RL. We empirically verify this hypothesis through several analysis experiments, showing that long horizons indeed present a fundamental barrier to scaling up offline RL. We then show that various horizon reduction techniques substantially enhance scalability on challenging tasks. Based on our insights, we also introduce a minimal yet scalable method named SHARSA that effectively reduces the horizon. SHARSA achieves the best asymptotic performance and scaling behavior among our evaluation methods, showing that explicitly reducing the horizon unlocks the scalability of offline RL.

Horizon Reduction Makes RL Scalable
Seohong Park'
Kevin Frans'
Deepinder Mann
Benjamin Eysenbach² Aviral Kumar" Prrnceeton University
Sergey Levine'
University of California, Berkeley
CCannegie Mellon University
sechong@berkeley. edu
Abstract
In this work, we study the scalability of offline reinforcement learning (RI.) algo- rithms. principle, truly scalable offline RL algorithm should be able to solve any given problem, regardless of its complexity, given sufficient data, compute, and model capacity. We investigate and how current offline RL algorithms match up to this promise diverse, challenging, previously unsolved tasks, using datasets up to 1000x larger than typical offline RL datasets. We observe that despite scal- ing up data, many existing offline RL algorithms exhibit poor scaling behavior, saturating well below the maximum performance. We hypothesize that the horizon is the main cause behind the poor scaling of offline RL. We empirically verify this hypothesis through several analysis experiments, showing that long horizons in- deed present fundamental barrier to scaling up offline RL. We then show that var- ious horizon reduction' techniques substantially enhance scalability on challenging tasks. Based
insights, also introduce minimal yet scalable method named SHARSA that effectively reduces the horizon. SHARSA achieves the best asymptotic performance and scaling behavior among our evaluation methods, show- ing that explicitly reducing the horizon unlocks the scalability of offline RL.
Code: https:/ http:/giittpt com/ sechongaark/hhoiiooorrrrreeee
Introduction
Scalability, the ability to consistently improve performance with more data and compute, is the core of the success of modern machine learning algorithms, across natural language processing (NLP), computer vision (CV), and robotics. In this work, we are interested in the scalability of offline reinforcement learning (RL), a framework that can leverage large-scale offline datasets to learn performant policies. While prior works have shown that current offline RL methods scale to more (but not necessarily harder) tasks with larger models and datasets [49, 86), it remains unclear how RL scales with data to more challenging tasks, especially those that require more complex, longer- horizon sequential decision making.
Our main question, posed informally is:
To what extent can current offline RL algorithms solve complex tasks simply by scaling up data and compute?
In principle, a truly scalable offline RL algorithm should be able to master any given task, no matter how complex and long-horizon it is, given sufficient amount of data (of sufficient coverage), compute, and model capacity. Studying how current offline RI. algorithms live up to this promise is important, because it will tell us whether we are ready to scale existing offline RL methods, or if we must further improve offline RL algorithms before scaling them.
Throughout this work, we use the term "horizon reduction" to refer to techniques that reduce the effective decision horizon, such as st-tep returns and hierarchical policies.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Standard RL (best method)
RL with horizon reduction (best method)
cube-octuple
puzzle-4i 4x5
puzzle e4x6
hummaze-gil -giant
Success Rate
50-
100
100
100
01
0
1K 10M 100M 1B
1M 10M 100K
10M 100M 1B
1M 10M 100M 1B
Dataset Size
Dataset Size
Dataset Size
Dataset Size
Figure 1: Horizon reduction makes RL scalable. Standard offline RI. methods struggle scale on highly challenging tasks, not improving performance with more data. We show that this is mainly because the long horizon can fundamentally inhibit scaling, and that horizon reduction techniques unlock the scaling of offline RL.
To answer this question, we generate large-scale datasets for tasks that require highly complex, long- horizon reasoning, and study how current offline RL algorithms scale with data. Specifically, on complex simulated robotics tasks across diverse domains in OGBench [75], we collect dataset with up to one billion transitions for each environment, which is x000 larger than standard 1M-sized offline RL datasets used in prior work 75]. To isolate the fundamental sequential capabilities of RL algorithms, we also idealize environments by removing other potential confounding factors, such as visual representation learning. In these controlled yet challenging environments, we evaluate the performance of state-of-the-arr offline RI algorithms while varying the amount of data.
peerision-makinng
We observe that many existing offline RL algorithms struggle to scale, even with orders of magnitude more data in these idealized environments. Specifically, we show algorithms such as IQL [47], SAC+BC (Appendix E.I,, CRL [18], and FQL [76] often either completely fail to solve complex tasks, or require an excessive amount of compute and model capacity to reach even moderate performance. Their performance often saturates far below the maximum possible performance (Figure 1), especially on complex, long- horizon tasks, suggesting that there exist scalability challenges in offline RL.
We hypothesize that the reason behind this poor scaling is due to the curse of horizon in both value learning and policy learning. In value learning, we argue that the temporal difference (TD) learning objective used in many offline RL. algorithms has a fundamental limitation that inhibits scaling to longer horizons: biases (errors) in the target Q values accumulate over the horizon. Through controlled analysis experiments, we show that this bias accumulation is strongly correlated with poor performance. Moreover, we show that increasing the model size or adjusting other hyperparameters. does not effectively mitigate this issue, suggesting that the horizon fundamentally hinders scaling. In policy learning, we argue that the complexity of the mapping between states and optimal actions in long-horizon tasks poses a major challenge, and support this claim with experiments.
We then demonstrate that methods that explicitly reduce the value or policy horizonsexhibit substan- tially better scaling (Figure 1). For example, we show that even simple techniques to reduce the value horizon, such as n--tepp returns, can substantially improve scaling curves and even asymptotic perfor- mance. Based on insights, we also propose minimal yet scalable RL method called SHARSA that reduces both the value and policy horizons. Our method relies only on simple objectives that do not require excessive hyperparameter tuning, such as SARSA and behavioral cloning, while effec- tively reducing the horizon. Despite the simplicity, we show that SHARSA generally exhibits the best scaling behavior and asymptotic performance among our evaluation methods.
Contributions. Our main contributions are threefold. First, through our 1B-scale data-scaling analy- sis, empirically demonstrate that many standard offline RL algorithms scale poorly complex, long-horizon tasks. Second, we identify the horizon as a main obstacle to RL scaling, and empirically show that horizon reduction techniques can effectively address this challenge. Third, we propose simple method, SHARSA, that exhibits strong asymptotic performance and scaling behavior.
2 Related work
Offline RL. Offline RL aims to train a wwward-maximizing policy from static dataset without online interactions [55]. The main challenge in offline RL is to maximize rewards while staying close to the dataset distribution to avoid distributional shift. Previous works have proposed a number of techniques to address this challenge based on behavioral regularization [22, 76, 91, 99], conservatism [48].

---

weighted regression [77, 78, 97], in- in-mle maximization [25, 47, 100], uncertainty minimization [3, 71], one-step RL [7. 18], model-based RL [43, 102, 103], and more [10, 31, 39, 40, 53, 84, 96]. Among these methods, we mainly consider three distinct representative model-free algorithms that have been reported to achieve state-of-the-ar performance on standard benchmarks [75, 76, 92], [47], SAC+BC (Appendix E) and CRL [18], as the main subject of our scaling analysis. We leave the scaling study of offline model-based RL for future work.
Scaling RL. Prior work has studied the scalability of RL algorithms in various aspects. Many previous works focus on scaling online RL to solve more diverse tasks with larger models [29, 70], more compute [8 and parallel simulation [16, 24, 58, 85]. More recently, several works have also explored the scaling of online, on on-policy RL on language tasks [38, 94]. Unlike these works that study online RL, we focus on the scalability of offtine RL algorithms.
Many prior works on scaling offline RL focus on scalability more tasks by training large, multi- task agent that is capable of solving more diverse (but not necessarily harder) tasks [8, 11, 49, 54, 80, 86]. Unlike these works, we focus the ability to solve more challenging tasks that require highly complex sequential decision making, given more data and compute. This is analogous to scalability along the "depth" axis, as opposed to the "width" axis that the prior works have explored. This is an important, complementary axis to study, as it will let us know whether offline RL is currently bottlenecked by the amount of data and compute, or the fundamental learning capabilities of algorithms. A closely related work is Park et al. [73]. which shows that poor policy extraction and generalization can bottleneck the scaling of offline RL. We study the scalability of offline RL to more complex (in particular, longer-horizon) tasks when these bottlenecks are removed, with the use of more expressive policy classes and with datasets 100x as large as this prior work. This makes our study distinct from and complementary to the challenges that Park et al. [73] highlight.
Horizon reduction and hierarchical RL. In this work, we identify the horizon length as one of the main factors that inhibit the scaling of RL. Prior works have developed diverse techniques to reduce the effective horizon with multi-step hierarchical value functions [1, 5, 14, 56, 66, 87, 90, 106), hierarchical policy extraction [26, 62, 72], or high-level planning [17, 19, 35, 36, 44, 45, 57, 69, 74, 82]. While these works in hierarchical RL have mainly focused on exploration [68], representation learning [67, 72], and planning [17]. we focus on scalability, showing that horizon reduction mitigates bias accumulation and unlocks the scaling of offline RI. In this work, we also propose a new, minimal method (SHARSA) to reduce the horizon. SHARSA is related to previous hierarchical methods that use rejection sampling for subgoal selection [1, 33, 64]. Inspired by these works, SHARSA uses a minimal set of techniques (e.g., flow behavioral cloning and SARSA) that address the horizon issue in a scalable manner (see Section 6.1 for further discussions).
3 Experimental setup
Problem setting. We aim to understand the degree to which current offline RL methods can solve complex tasks simply by scaling data and compute. In particular, we are interested in the capabilities of offline RL algorithms to solve challenging tasks that require complex. long- long-hoziron sequential decision-maker given enough data. To this end, we focus on the offline goal-conditioned RI. setting [75], where we want to train agents that are able to reach any goal state from any other initial state the fewest number of steps, from a static, pre-collected dataset of behaviors. This problem poses substantial learning challenge, behaviors purely from binary sparse rewards and an unlabeled (reward-free) dataset. We note that although we mainly focus on goal-conditioned tasks in this work, our claims are not limited to goal- conditioned settings (Appendix D).
the agent must learn complex, long- long-horizon, multi-task
Formally, we consider controlled Markov process defined as M = (S.A, μu p). where S is the state space, .A is the action space, pas)) E A(S) is the initial state distribution and p(s' | s, s.a): S x A + A(S) is the transition dynamics kernel. Here, ((x) denotes the set of probability distributions on space X, and we denote placeholder variables in gray. We denote the discount factor as 7. The dataset D= = (-(")
] Jn (1. consists of N length-// state-action trajectories, (N0, 0n, 81, 01y..., 8H)). We assume that these trajectories are collected in an unsupervised, task- agnostic manner.
3

---

Environments and datasets. We employ four highly chal- lenging offline goal-conditioned RL tasks in robotics from the OGBench task suite [75]. Among these tasks, cube in- volves sequential pick-and-place manipulation of multiple cube objects, puzzle involves solving a combinatorial puz- zle called "Lights Out" with a robot arm, and humanoidmaze involves whole-body control of humanoid agent navi- gate a given maze. These OGBench tasks provide multi- ple variants with varying levels of difficulty, and we employ the hardest tasks (cube-octuple, puzzle-(4x) numanoidmaze- dwwze-aaaaiaee gi ant) to maximally challenge offline RL al- gorithms. To our knowledge, no current offline RI. algorithm has been reported to achieve non-trivial performance (i.e., non- zero performance on most evaluation goals) on these hardest tasks with the original OGBench datasets [75].
cube-octuple
puzzle-4x6
4x6), and
humanoi
humano dmaze -giant
On these environments, we generate up to 1B transitions using the scripted policies provided by OGBench. These datasets consist of "play" [62] task-agnostic trajectories to ensure sufficient coverage and diversity (see also the discussion about dataset coverage in Section 4). Specifically, they contain trajectories that randomly navigate the maze mimaze) sequentially perform random pick-and -lacce (cube), or press random buttons (puzzle). These task task-agnos unsupervised datasets conceptually resemble unlabeled Internet-scale data used to train vision and language foundation models. We also note that our 1B-sized datasets contain about 1M trajectories and 10M atomic behaviors in manipulation environments, which is similar or even larger than of the largest robotics datasets to date [13].
Idealization. To isolate the core sequential decision making capabilities of RL from other confound- ing factors, such as challenges with visual representation learning, distributional shift, and data cov- erage, we idealize environments and tasks in our analysis experiments. While these challenges are certainly important in practice, our rationale is to first understand how current offline RL algorithms solve highly challenging tasks in an idealized, controlled setting with near-infinite data.
Specifically, we employ low-dimensional state-based observations with oracle goal representations to alleviate challenges in visual representation learning. We also remove some evaluation goals that may require ouitt--diitriuutinn generalization to ensure all tasks remain in-distribution. Finally, we ensure that the datasets have sufficient coverage and optimality, by verifying that our data-collecting script enables achieving near-perfect performance on the same environment with fewer objects (see Section 4 for the full discussion). We refer to Appendix F for the details.
we evaluate. In this work, we mainly consider three performant, widely-used offline model- free RL algorithms across different categories: IQL, CRL, and SAC+BC. IQL [47] is based on in-sample maximization, CRL [18] based on contrastive learning and one-step RL [7], and SAC+BC (Appendix E. 1) is based on behavioral regularization [22, 99]. Additionally, we employ flow behavioral cloning (flow BC) [6, 12] to understand the scalability of behavioral cloning as well. Due to high computational costs, we use four random seeds in our scaling experiments (unless otherwise noted), and report 95% confidence intervals with shaded areas in the plots. A full description and implementation details of the algorithms are provided in Appendices E and F.
Methods
4 Standard offline RL methods struggle to scale
We now evaluate the degree to which four standard offline RL methods (flow BC, IQL, CRL, and SAC+BC) can solve the four challenging tasks by simply scaling up data. Figure shows the scaling plots of the four methods with 1M, 10M, 100M, and 1B iB-sized datasets (see Figure 13 for the full training curves). These methods are trained for 5M steps with [1024, 1024, 1024, 1024] 1024]-sized -ized multi- layer perceptrons (MIPss..
In aggregate, our results show that none of these four standard offline RL methods able solve all four tasks, even with the largest 1B-sized datasets. Notably, all of them completely fail on the hardest cube-octuple task. Moreover, their performance often quickly plateaus well below the
htttp://enn htt///e..ikipiiia..ao wtt.://pdid...o//lala/aeeeeeeeee
(game).

---

Flow BC
IQL
CRL
SAC+BC
Success Rate Success Rate
100
100
100
100
cube
octuple
1M
10M 100M 18
1M
10M 100M
18
1M
10M
100N
13
1M
10M
100N
18
100
puzzle
4x5
1H
10M 100M 18
1M
10M 100M 19
10M 100N
10W 100N
Success Rate
100
100
puzzle
4x6
1H
10M
100%
18
1M
10M 100M
18
10M 100N
10X 100N
Success Rate
100
100
100
100
hummaze
giant
1H
100% 18
1M
100M 12
10M 10ON
10M 100M Dataset Size
Dataset Size
Dataset Size
Dataset Size
RL with horizon reduction (best method in Section 6)
Figure 3: Standard offline RL methods struggle scale on challenging tasks. train four offline RL methods with 1M, 10M, 100M, and 1B data on four complex, long- long-horizon tasks. However, even with 1B data, their performance often saturates far below the maximum performance (100%).
optimal success rate (i.e., 100%), despite scaling up data. In other words, these standard offline RL methods struggle to scale on these tasks.
A keen reader may already have several questions about this result. Before proceeding further, we first address those potential questions.
Q: How do you know these tasks are solvable with the given datasets?
A: We will see in Section 6 that it is indeed possible to solve these tasks, at least achieve non- trivial performance (denoted blue in Figure 3). In Appendix A, we also show that these algorithms can solve the same tasks with fewer objects, using datasets collected by the same scripted policy, This verifies that the dataset distribution (induced by the scripted policy) provides sufficient coverage to learn near-optimal policy.
Q: Have you tried further increasing the model size?
A: A natural confounder in the results above is the model size. To understand how this affects performance, train SAC+BC, the best method cube-double and puzzle-4x4 (Figure 10), using up to 35x larger models with 591M parameters, on the largest 1B datasets. Figure shows the training curves. The results suggest that while using larger networks on some tasks to some degree, this alone is not sufficient to master the tasks, especially the hardest cube-octuple task. Moreover, the performance often saturates (or sometimes degrades) despite using larger models. Appendix B, we provide more ablations with different architectures (residual MLPs and Transformers), which show similar trends.
improve performance
While an even larger network with a smaller learning rate might further improve performance (which unfortunately could not afford, 591M models already require 8 days of training), are interested in performance within a reasonably bounded total compute budget. If an algorithm is unable to achieve good performance within practical amount of compute, deem it challenge in scalability. In contrast, in Section 6, we will show that horizon reduction techniques enable achieving
We already decreased learning rate for the largest 591M model; Appendix B for the details.
5

---

cuube-octuple
puzzle-4xx
puzzle-4x6
answer hummaze -glant
Sucoees Bats
100
100
100
100
-o- KP 100003 (scae)
KP [1054]eis (W0P)
ICP [1024]v) (179)
0
0
0
BM
0
BK
SN
BM
Steps
Steps
Steps
Staps
3L with borizor reduction (best nethod in Section
XLP [1024]-1)
Figure 4: Increasing model capacity alone is not sufficient to master the tasks.
significantly better scaling behavior and asymptotic performance (denoted in blue in the above figure) even with the original [1024) 4-sized models.
Q: Are you sure this isn't just hy perparammeee or design choice issue?
A: While there always a possibility of achieving better performance with better hyperparameters, despite our extensive efforts in adjusting hyperparameters and design choices, were unable to achieve promising scaling results with these methods. In Appendix B, we present 9 ablation studies on policy classes (Gaussian and flow policies), network architectures (MLPs and Transformers), value ensembles, regularization techniques, learning rates, target network update rates, batch sizes, and gradient steps, showing that none of these changes substantially improves scalability or asymptotic performance across the board.
5 The curse of horizon
Why do current offline RL methods exhibit poor scaling behavior on these challenging tasks? In the previous section, we observed that adjusting model sizes or other hyperparameters does nor effectively improve scaling on complex, long-horizar
tasks, even though they scale on simpler tasks (see Figure 10). This suggests that there may exist a fundamental obstacle that inhibits the scaling of offline RL. We hypothesize that this obstacle is the horizon. In this section, we discuss and analyze the curse of horizon along two orthogonal axes: value and policy.
5.1 The curse horizon in value learning
Many offline RL algorithms train Q functions via temporal difference (TD) learning. Unfortunately, the TD learning objective has a fundamental limitation: at any gradient step. the prediction target that the algorithm chases biased [89], and these biases accumulate over the horizon. Such biases do not exist (or at least they do not accumulate) in many scalable supervised and unsupervised learning objectives, such as next-token prediction. As such, we hypothesize that the presence of bias accumulation in TD learning is one of the fundamental causes behind the poor scaling result in Section 4. This hypothesis partly explains why CRL, which is not based on TD learning, achieves significantly better asympiotic performance on humanoi dmaze- in Figure 3.
Didactic task setup. We empirically validate this hypothesis by performing an analysis on a didactic task named combination-oo lock (Figure 5). This en- viromment has H states and two discrete actions. The states are linearly ordered, and each state has an "an- swer" action. The state order and answer actions are randomly chosen (and kept fixed) when instantiating the environment. The agent starts from the first state, and whenever selects the correct action, it moves forward by one step; otherwise, is sent back to the first state. The agent always gets reward of 1 at each step, except at the final (goal) state, where gets a reward of 0 and the episode terminates. Hence, the agent must memorize actions to reach the goal.
combination lock
|0
0
2
246
190
23
491
350
and
512 states
Figure 5: combination- with H 512.
To understand the effect of bias accumulation in deep TD learning, we evaluate two offline Q learning algorithms with different TD horizons: standard (1-step) DQN [65] and n-step DQN (see Appendix F.1 for details).* Note that the optimal Q functions for both algorithms are the same
Although these are online RL algorithms, we can also use them for offline RL without modification, as our datasets have uniform coverage and thus do not require conservatism.

---

(under the optimal, full-coverage datasets) and thus have the same learning complexity, but the latter involves n times fewer TD recursions. To compare the maximum possible performance of these two algorithms in fair way, we employ two types of datasets that have uniform coverage of length-(1,, (1, n) trajectory segments, evaluate each method on both datasets, and select the best one for each method. In this experiment, we do not use a discount factor (i.e., 1) as the task has finite horizon. We refer the reader to Appendix F.1 for the full experimental details.
Success Rate (t)
TD Error (1)
Q Error (1)
Success Rate
100
n-step
2.5
66
100
50
0.0
0
258 512 1024 2048 4096 Horizon Length
256 512 1024 2048 4096
256 512 1024 2048 4096 Horizon Length
Horizon Length
Figure 6: 1-step TD learning suffers bias accumulation (i.e., high Q errors).
We train 1-step and 64-step DQN on combination- lock with different horizon lengths, ranging from H- = 256 to H = 4096. First, we measure their performance. The first plot in Figure 6 shows that the performance of 1-step DQN drops faster than that of 64-step DQN as the horizon increases.
Next, we measure two metries: the TD error and the Q error. The TD error measures the difference against the TD target y, and the Q error measures against the ground-truth truth Q value Q". The results are presented in the second and third plots in Figure 6. They show that 1-step DQN has significantly larger Q errors than 64-step DQN, even though they have similar TD errors. Since the Q error corresponds to compounded error in the learned Q function, this strongly suggests that bias accumulation happens in practice, and that it can substantially affect performance on long-horizon tasks. We further corroborate this point by measuring how Q errors vary across state positions in a single episode. Figure 7 shows that Q errors indeed become larger as the distance from the end increases.
Q Error (1)
D-stop
200
64
0 000000000000000000
4096
Distance from 80
Figure 7: Biases accumulate.
Then, it possible to fix error accumulation in 1-step TD learning by tuning hyperparameters, or is a fundamental limitation of deep TD learning? As in Section 4, we adjust diverse hyperparameters, such as the model size, learning rate (LR), and target network update rate (TUR), and present the ablation results in Figure 8. The results suggest that simply increasing the model size decreasing LR or TUR provides limited or no improvement in both performance and bias accumulation (measured by Q errors). This matches the observation in Section 4. In contrast, 64-step DQN achieves significantly better performance and Q errors, even with the default-sized network. This suggests that error accumulation over the horizon may be fundamental factor that obstructs scaling up TD learning.
Of course, there is a possibility that under certain hyperparameter settings, such as with a very low learning rate and much larger network, 1-step DQN might be able to converge to the optimal policy on hong-horizon tasks. While it is impossible experimentally eliminate this possibility entirely, our results do suggest 1-step TD learning scales poorly in horizon, in the sense that it may require an excessive amount compute, model capacity, and the practitioner's time to achieve good performance. In contrast, our results show that techniques that explicitly reduce the effective horizon, such as step returns (as one example), can potentially be much more effective in addressing this issue.
5.2 The curse of horizon in policy learning
Orthogonal the bias accumulation issue in value learning discussed in the previous section, the policy may also independently suffer from the curse of horizon. This is because, even when the value function is perfect, the policy still needs to fit the mapping between states and optimal actions prescribed by the Q function, where this mapping can be increasingly complex as the horizon becomes longer. For example, the goal-conditioned setting, the mapping between the optimal actions and distant goals can be highly complex [75], as it depends the entire topology of the state space.
7

---

Succeas Rate (1)
Buccana Rate (1)
Baccaaa Rate (1)
Succaan Babe
PLP Bize
Succesn Rate
Succode Rabe
TiEL
(0hndtep))
(00ntrey)
0,0003
0,003
[1-atep)
[1-stap)
(1-Stap)
Borizon Length
2048 6096
612
Leugth
2048 6096
0.03020
512 1924 2049 6096
Length
[1-9tap)
Q Eeror ($)
Q Erzoz (U)
(t-atep)
0 Erzoz (L)
(1-Stepp
Q1
10
109-
512 1004 SOAN c095 Horizon Length
256 517 1024 R0AK 4095 Eorizon Length
255 517 1006 SOAR 4090
Borizon Length
Figure 8: Regardless of hyperparameters, 1-step TD learning struggles to handle long horizon.
Analogous to n-step returns in value learning, we can mitigate the curse of horizon in policy learning by reducing the effective horizon using a hierarchical policy [26, 62, 72, 75]. For example, we can decompose a goal-conditioned policy [a s, g) into high-level policy r"(w | g) that outputs subgoal w. and low-level policy (a 8, w) that outputs actions given the subgoal. Since the complexity of the individual hierarchical policies is often (much) lower than that of the flat (i.e., non- hierarchical) policy [72], this can lead a policy that both performs and generalizes better [75]. This is akin to how chain-of-thought reasoning [98] improves the performance of language models, which shows that decomposing a problem into multiple simpler subtasks more effective than producing an answer directly. While we do not perform a separate didactic experiment for this point (as it is relatively well studied and analyzed in prior work [26, 72, 75]), we will empirically demonstrate how hierarchical policies can substantially improve the scalability of offline RL in challenging environments in the next section.
Horizon reduction makes RL scale better
Based on the insights in Section 5, we now apply value and policy horizon reduction techniques to our four challenging benchmark tasks, and evaluate how they improve the scalability of offline RL.
6.1 Horizon reduction techniques
As discussed in the previous section, there are two orthogonal axes of horizons in RL: the value horizon (Section 5.1) and policy horizon (Section 5.2). In experiments, consider four representative techniques that reduce one or both types of horizons. We refer to Appendix E.2 for the full details.
Value horizon reduction. For value horizon reduction, we consider n-step SAC+BC, variant of SAC+BC with n-step TD updates, analogous to n-step DQN in Section 5.1. This method reduces the value horizon, but not the policy horizon, it learns flat policy.
Policy horizon reduction. We consider two techniques that reduce the policy horizon, but not the value horizon. Hierarchical flow BC (hierarchical FBC) trains hierarchical policy (x* and x²) with flow behavioral cloning, without performing RL. HIQL [72] trains flat value function with coal-conditioned IQL, but extract hierarchical policy from it. These methods will tell us the degree to which having a hierarchical policy alone can improve performance.
Value and policy horizon reduction. We can reduce both the value and policy horizons with full- fledged hierarchical RI.. While there are several approaches that perform full hierarchical offline RL. with a (potentially complex) high-level planner (Section 2), there exist only handful of planning- free methods that reduce both the value and policy horizons [1, 33, 64]. Since these methods are either based on (less scalable) variational autoencoders or recurrent networks [1, 64], or only applicable to language- based tasks [33], we propose a new method called SHARSA in the following section.
6.2 SHARSA: minimal, scalable offline RL method for horizon reduction
We propose simple, scalable offline RL. method that reduces both the value and policy horizons for continuous control. Our main goal here is, rather than designing completely novel technique
8

---

that achieves -fate-of-the-ee art performance, to empirically demonstrate how reducing both types of horizons improves scalability, even with otherwise simple ingredients.
The main challenge with full hierarchical offline RL (ie., value and policy horizon reduction) is high-level policy extraction: learning subgoal policy = 8,9) that maximizes values while not deviating too much from the data distribution. For low-level or flat policies (whose output space is A), policy extraction is typically best done by reparameterized gradients in the action space [73] (e.g., DDPG+BC [22, 73). However, the same technique does not necessarily work for high-level policies (whose output space S), since first-order gradient information the state space may not be semantically meaningful (e.g,, the button states of puzzle are discrete, and thus first-order gradients in the state space are not even well-defined).
To address this challenge, we employ rejection sampling [9, 31, 33, 64] with an expressive flow policy [2, 6, 59, 61, 76] for high-level policy extraction: first sample N subgoals from high-level flow BC policy z and pick the best one based on a high-level (goal-conditioned) value function Qh:
T" (s.
arg max ME M cayy mom (wllsg)
Q" (4s.)).
(1)
where v denotes equality in distribution. This is beneficial because it does not use first-order information (unlike reparameterized gradients) while leveraging the expressivity of a flow policy [73, 76]. For the value function Q" in Equation (1), we employ high-level SARSA [89] for simplicity, which trains behavioral value functions with the following losses:
L (V*)) = E D(V (sV (sn,
Q"(Ss. (15h,8htn1110 (h.nn [))]
(2)
1. 10(00)) = E (sh, Shtn: 9), Σ ((r((n- + "
where 4 is high-level state value function, Q* is the target network [65], D is a loss function (regression or binary cross- cross-entropy: we use the latter), and the expectations are taken over length-n trajectories (Sh, at..... .... 8h+n) and goals g sampled from the dataset. We refer to Appendix E.3 for the full details. We note that one can use any decoupled value learning methods (i.e., those that do not involve policy learning, such as IQL. [47]) in place of SARSA. For the low-level policy, we can either simply employ goal-conditioned flow BC, or do another round of similar rejection sampling based on a low-level behavioral (SARSA) value function. We call the former variant SHARS! and the latter variant double SHARSA. We provide the pseudocode for SHARSA and double SHARSA and 2.
in Algorithms
SHARSA is appealing for two reasons. First, it is simple and easy to use. SHARSA is only based on behavioral cloning and SARSA. both of which do not require extensive hyperparameter tuning, unlike typical offline RL algorithms [73, 92]. Second, it reduces both the value and policy horizon lengths with an expressive flow policy. This mitigates the curse of horizon in a scalable way.
6.3 Results
We now evaluate the performance of various horizon reduction techniques on the main benchmark tasks. We present the data-scaling curves in Figure 9 (see Figure 14 for the training curves). The results show that horizon reduction techniques can indeed unlock the scalability of offline RL on these challenging tasks. We highlight three particularly informative comparisons:
Value horizon reduction: SAC+BC vs. n-step SAC+BC shows that simply reducing the value horizon with n-step returns (n-step SAC+BC) substantially improves scalability and even asymptotic performance on many tasks. This matches our didactic experiments in Section 5. We note that their network sizes and training objectives are identical, except for the use of n-step returns.
Policy horizon reduction: Flow BC hierarchical flow BC shows that reducing the policy horizon also significantly improves performance, but on different set of tasks. In particular, it shows that, on some tasks (e.g., octeloct is challenging achieve even non-zero performance without reducing the policy horizon.
TThis acronym stands for (state) (high-leel action) (reward) (state) (state)-(high--eew (high-level action).

---

Flow BC
SAC-BC Policy Bussoo
a-step 9AC-3C Hiraacciial FUC
BEQL
BHARSA
Doublo SHARSA
Boziecn
Polcey Bazizo
Poliyy Marizo
Polssy Harazon
Polsty Hacaoon
Succrss Sacccnen laes Success Itss Uiccns Itte
cibaa
octuple
110 13M 14
10 1994
54 100 1204
HM 1008
18 530 1204
53 1ree
1N 520 1se
puzzle
1x25
no 13M
16 1291
16
54 06. 1004
MM 1508
18
531
1106
538 1068
100
160
16C
100
100
100
500
puzzle
4x6
16e 3PM
10t 199
106 18tH
196 1891
199
331 1008
1086
102
100
10C
100
100
100
500
giant
1008
1008
Tatsses
Size
Bassses Size
Bessent S1zw
Dassewt S1oe
Dathaet Sime
Datanet. 51.0
Datatet: $1.00
Figure 9: Horizon reduction makes RL scalable. Value and policy horizon reduction techniques often result substantially better scaling and asymptotic performance.
Value and policy horizon reduction: SHARSA vs. the others shows that reducing both value and policy horizons leads to the best of both worlds. In particular, (double) SHARSA is the only method that achieves non-trivial performance on all four tasks our experiments. In Appendix C, we present several ablation studies on SHARSA, discussing the relative importance of various design choices (e.g., alternative policy extraction strategies and value learning objectives).
7 Call for research: offline RL algorithms should be evaluated for scalability
In this work, we empirically showed that standard, non-hierarchical offline RL methods struggle to scale on complex tasks. We hypothesized that this is due to the curse of horizon, and demonstrated that techniques that explicitly reduce the horizon length, including SHARSA, can unlock scalability.
However, this is far from the end of the story. Empirically, still none of these techniques enable mastering all four tasks (i.e., achieving a 100% performance), even with 1B data. Methodologically, these hierarchical methods only mitigate the етгor accumulation issue in TD learning with two-level hierarchies, rather than fundamentally solving it. Moreover, SHARSA and other n-step return- based methods implicitly assume that dataset trajectories are near-optimal within short segments (although double SHARSA relaxes this assumption some extent). Finally, our results still indicate room for improvement over SHARSA, as in some cases the performance does not always scale monotonically with increasing dataset sizes (Figure 9). These limitations current approaches open up a number of fruitful research questions in scalable reinforcement learning:
Can we completely avoid TD learning while performing RL (e.g., potentially with model-based RL [29], linear programming [79, 96], shortest path algorithms [15, ))?
Can we find simple, scalable way to extend beyond two-level hierarchies to deal with horizons of arbitrary length?
Is the curse of horizon fundamentally impossible to solve? The RL theory community suggests otherwise [104, 105], and can we instantiate such a principle within deep RL?
We conclude this paper by calling for research on scalable offline RL algorithms, that is, algorithmic research done on large-scale datasets and complex tasks. Currently, offline RL research is often mainly conducted on standard datasets (e.g., D4RL [21], OGBench [75], etc.) with 1M-5M transitions. However, success on small-scale tasks and datasets does not necessarily guarantee success on datasets that are 1000 larger, not every algorithm scales equally [88, 93]. Hence, to assess their potential at scale, it is important to directly evaluate new methods on substantially more challenging tasks and larger datasets and measure sealing trends. To facilitate this, we open-source our tasks, datasets, and implementations (link), where we have made them as easy to use possible. We hope that our insights in this work, as well as our open-source implementation, serve as a foundation for the development of scalable offline RL objectives that unlock the full potential of data-driven RL.
10

---

Acknowledgments and Disclosure of Funding
We thank Oleh Rybkin for helpful discussions. This work was partly supported by the Korea Foun- dation for Advanced Studies (KFAS), National Science Foundation Graduate Research Fellowship Program under Grant No. DGE 2146752, ONR N00014-22-1-2775. and NSF IIS-2150826. This re- search used the Savio computational cluster resource provided by the Berkeley Research Computing program at UC Berkeley.
References
[1] Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. Opal: Offline primitive discovery for accelerating offline reinforcement learning. In International Conference on Learning Representations (ICLR), 2021.
[2] Michael S Albergo and Eric Vanden- Eijuden, Building normalizing flows with stochastic interpolants. In Interaatinnal Conference on Learning Representations (ICLR), 2023.
[3] Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-baser offline reinforce- ment learning with diversified q-ensemble. In Neural Information Processing Systems (NeurlPS), 2021.
[4] Jimmy Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. ArXiv, abs/ 06450 2016.
[5] Pierre-Lue Bacon, Jean Harb, and Doina Precup. The option-critic architecture. AAAI Conference on Artificial Intelligence (AAAI), 2017.
[6] Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, al. A vision-language-actionn flow model for general robot control. ArXiv, abs/2410.24164, 2024.
[7] David Brandfonhrenner, William Whitney, Rajesh Ranganath, and Joan Bruna. Offline without policy evaluation. In Neural Information Processing Systems (NeurIPS), 2021.
[8] Yevgen Chebotar, Quan Ho Vuong, Alex Irpan, Karol Hausman, F. Xia, Yao Lu, Aviral Kumar, Tianhe Yu, Alexander Herzog, Karl Pertsch, Keerthana Gopalakrishnan, Julian Ibarz, Ofir Nachum, Sumedh Anand Sontakke, Grecia Salazar, Huong Tran, Jodilyn Peralta, Clayton Tan, Deeksha Manjunath, Jaspiar Singht, Brianna Zitkovich, Tomas Jackson, Kanishka Rao, Chelsea Finn, and Sergey Levine. Q-transformer: Scalable offline reinforcement learning via autoregressive q functions. In Conference on Robot Learning (CoRI.), 2023.
[9] Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling, International Conference on Learning Representations (ICLR), 2023.
[10] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, P. Abbeel, A. Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Neural Information Processing Systems (NeurIPS), 2021.
[11] Jie Cheng. Ruixi Qiao, Gang Xiong, Qinghai Miao, Yingwei Ma. Binhua Li, Yongbin Li, and Yisheng Lv. Scaling offline model-based rl via Conference on Learning Representutions (ICLR), 2025.
world-action model pretraining. In International
[12] Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Robotics: Science and Systems (RSS), 2023.
[13] Open X- Emboddment Collaboration, Abby O'Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, et al. Open x-embodiment: Robotic learning datasets and rt-x models. In IEEE International Conference on Robotics and Automation (ICRA), 2024.
[14] Peter Dayan and Geoffrey Hinton. Feudal reinforcement learning. Neural Information Processing Systems (NeurIPS), 1992.
[15] Vikas Dhiman. Shurjo Banerjee, Jeffrey M Siskind, and Jason J Corso. Flrshawarshall reinforcement learning: Learning from past experiences to reach new goals. ArXiv, abs/1809.0993 2018.
11

---

[16] Lasse Espeholt, Hubert Soyer. Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward. Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukcuoglu. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference on Machine Learning (ICML), 2018.
[17] Benjamin Eysenbach. Ruslan Salakhutdinov, and Sergey Levine. Search on the replay buffer: Bridging planning and reinforcement lcarning. Neural Information Processing Systems (NeurlPS), 2019.
[18] Benjamin Eysenbach, Tianjun Zhang, Ruslan Salakhutdinov, and Sergey Levine. Contrastive learning as goal condiiondd reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2022.
[19] Kuan Fang, Patrick Yin, Ashvin Nair, and Sergey Levine. Planning to practice: Efficient online fine- tuning by composing goals latent space. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2022.
[20] Jesse Farebrother, Jordi Orbay, Quan Vuong, Adrien Ali Taiga, Yevgen Chebotar, Ted Xiao, Alex Irpan, Sergey Levine, Pablo Samuel Castro, Aleksandra Faust, et al. Stop regressing: Training value functions via classification for scalable deep rl. In International Conference on Machine Learning (ICMI.), 2024,
[21] Justin Fu, Aviral Kumar, Ofir Nachum, G. Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. ArXiv, abs/2004.07219, 2020.
[22] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2021.
[233] Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference on Machine Learning (ICML), 2018.
[24] Matteo Gallici, Mattie Fellows, Benjamin Ellis, Bartomeu Pou, Ivan Masmitja, Jakob Nicolaus Foenster, and Mario Martin. Simplifying deep temporal difference learning. In International Conference on Learning Representations (ICLR), 2025.
[25] Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent rl without entropy. In International Conference on Learning Representations (ICLR), 2023.
[26] Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horin horizon tasks via imitation and reinforcement learning. In Conference Robot Learning (CoRI.), 2019.
[27] Tuomas Haarnoja, Aurick Zhou, Pieter Abbecl, and Sergey Levine. Soft actor-critic: Off oof-cooicy maximum entropy deep reinforcement learning with a stochastic actor. International Conference on Machine Learning (ICML), 2018
[28] Tuomas Haamoja, Aurick Zhou, Kristian Hartikainen, G. Tucker, Schoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishck Gupta, Picter Abbeel, and Sergey Levine. Soft actor-critic algorithms and applica- tions. ArXiv, abs/ 111.0595,, 2018.
[29] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse control tasks through world models. Nature, 640:647-653, 2025.
[30] Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpe2: Scalable, robust world models for continuous control. In International Conference on Learning Representations (ICLR), 2024.
[31] Philippe -stsuuch,, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies. ArXiv, abb//2304.1053, 2023.
[32] H. V. Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In AAAI Conference on Artificial Intelligence (AAAI). 2016.
[33] Kyle B. Hatch, Ashwin Balakrishna, Oier Mees, Suraj Nair, Seohong Park, Blake Wulfe, Masha Itkina, Benjamin Eysenbach, Sergey Levine, Thomas Kollar, and Benjamin Burchfiel. Ghil-glue: Hierarchical control with filtered subgoal images. In IEEE International Conference on Robotics and Automation (ICRA), 2025.
[34] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). ArXiv, abs/1606.08415, 2016.
[35] Christopher Hoang, Sungryull Sohn, Jongwook Choi, Wilka Carvalho, and Honglak Lee. Successor feature landmarks for long horizon goal goal-conditioned reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2021.
12

---

[36] Zhiao Huang, Fangchen Liu, and Hao Su. Mapping state space using landmarks for universal goal reaching. In Neural Information Processing Systems (NeurIPS), 2019.
[37] Ehsan Imani and Martha White. Improving regression performance with distributional losses. In International Conference Machine Learning (ICML), 2018.
[38] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. ArXiv, abs/2412.167200 2024.
[39] Michael Janner, Qiyang Li, and Sergey Levine. Reinforcement learning one big sequence modeling problem. In Neural Information Processing Systems (NeurIPS), 2021.
[40] Michael Janner, Yilun Du, Joshua B. Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning (ICML.), 2022.
[41] Leslie Pack Kaclbling. Learning to achieve goals. In International Joint Conference on Artificial Intelligence (ICAAI) 1993.
[42] Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Tharz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. Conference on Robot Learning (CoRL.), 2018.
[43] Rahul Kidambi, Aravind Rajeswaran, Pranceth Netrapalli, and Thorsten Joachims. Morel: : Model-based offline reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2020.
[44] Junsu Kim, Younggyo Sco, and Jinwoo Shin. handmark-guidde subgoal generation in hicrarchical reinforcement leamning In Neural Information Processing Systems (NeurIPS), 2021.
[45] Junsu Kim, Younggyo Seo, Sungsoo Ahn, Kyunghwan Son, and Jinwoo Shin. Imitating graph-based planning with goal-conditioned policies. In International Conference on Learning Representations (ICIR), 2023.
[46] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
[47] Ilya Kostrikov, Ashvin Nair, and Sergey Levine Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations (ICLR), 2022.
[48] Aviral Kumar, Aurick Zhou, G. Tucker, and Sergey Levine. Conservative q g-learning for offline reinforce- ment learning. In Neural Information Processing Systems (NeurlPS), 2020.
[49] Aviral Kumar, Rishabh Agarwal, Xinyang Geng, George Tucker, and Sergey Levine. Offfine q-lcarning diverse multi-task data both scales and generalizes. In International Conference Learning Repre- sentations (ICLR), 2023.
[50] Cassidy Laidlaw, Stuart J. Russell, and Anca D. Dragan, Bridging rl theory and practice with the effective horizon. In Neural Information Processing Systems (NeurIPS), 2023.
[51] Hojoon Lee, Dongyoon Hwang, Donghu Kim, Hyunseung Kim. Jun Jet Tai, Kaushik Subramanian, Peter R Wurman, Jacgul Choo, Peter Stone, and Takuma Seno. Simba: Simplicity bias for scaling up parameters in deep reinforcement learning. International Conference on Learning Representations (ICLR), 2025.
[52] John M Lec. Introduction to Smooth Manifolds. Springer, 2012.
[53] Jongmin Lee, Wonseok Jeon. Byung-Jun Lee, Joëlle Pineau, and Kee- BeneEang& Kim. Optidice: Offline policy optimization via stationary distribution correction estimation. In International Conference on Machine Learning (ICML), 2021.
[54] Kuang- Huei Lee, Ofir Nachum, Mengjiao Yang, L. Y. Lee, Danicl Freeman, Winnie Xu, Sergio Guadar- rama, Ian S. Fischer, Eric Jang, Henryk Michalewski, and Igor Mordatch. Multi-game decision transform- ers. In Neural Information Processing Systems (NeurIPS), 2022.
[55] Sergey Levine, Aviral Kumar, G. Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives open problems. ArXiv, abs/2005.01643, 2020.
[56] Andrew Levy, George Dimitri Konidaris, Robert W. Platt. and Kate Saenko. Learning multi-level hierarchies with hindsight. In International Conference on Learning Representations (ICLR), 2019.
13

---

[57] Jinning Li, Chen Tang, Masayoshi Tomizuka. and Wei Zhan. Hierarchical planning through goal- conditioned offline reinforcement learning. IEEE Robotics and Automation Letters (RA-L). 7(4):10216- 10223, 2022.
[58] Zechu Li. Tao Chen, Zhang- Thang-Weil Hong. Anurag Ajay, and Pulkit Agrawal. Parallel q- g-learning: Scaling off-policy reinforcement learning under massively parallel simulation. In International Conference on Machine Learning (ICML). 2023.
[59] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. In International Conference on Learning Representations (ICLR), 2023.
[60] Yaron Lipman. Marton Havasi, Peter Holderrieth, Neta Shaul, Maut Le, Brian Karrer, Ricky Q. Chen, David Lopez-Paz, Heli Ben-Hamu, and Itai Gat. Flow matching guide and code. ArXiv, abs/2412.06264, 2024.
[61] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In International Conference on Learning Representations (ICLR), 2023.
[62] Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference on Robot Learning (CoRL), 2019.
[63] Zhuang Ma and Michael Collins. Noise contrastive estimation and negative sampling for conditional models: Consistency and statistical efficiency. In Conference on Empirical Methods in Natural Lunguage Processing (EMNLP), 2018
[64] Ajay Mandlekar, Fabio Ramos, Byron Boots, Li Fei-Fei, Animesh Garg, and Dieter Fox. Iris: Implicit reinforcement without interaction scale for learning control from offline robot manipulation data. In IEEE International Conference on Robotics and Automation (ICRA), 2020.
[65] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. ArXiv, abs/1312.5602, 2013.
[66] Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforce- ment learning. Neural Information Processing Systems (NeurIPS), 2018.
[67] Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Near-optimal representation learning for hierarchical reinforcement learning. In International Conference on Learning Representations (ICLR), 2019.
[68] Ofir Nachum, Haoran Tang, Xingyu Lu, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Why does hierarchy (sometimes) work so well in reinforcement learning? ArXiv, abs/1909.10611 2019.
[69] Soroush Nasiriany, Vitchyr H. Pong, Steven Lin, and Sergey Levine. Planning with goal-conditioned policies. In Neural Information Processing Systems (NeurIPS), 2019.
[70] Michal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski, Piotr Miloć, and Marek Cygan. Bigger, reg- ularized, optimistic: scaling for compute and sample-efficient continuous control. In Neural Information Processing Systems (NeurIPS), 2024.
[71] Alexander Nikulin, Vladislav Kurenkov, Denis Tarasov, and Sergey Kolesnikov. Anti-cxploration by random network distillation. In International Conference on Machine Learning (ICML), 2023.
[72] Seohong Park, Dibya Ghosh, Benjamin Eysenbach, and Sergey Levine. Hiql: with latent states actions. In Neural Information Processing Systems (NeurIPS), 2023.
Offline goal-conditioned
[73] Scohong Park, Kevin Frans, Sergey Levine, and Aviral Kumar. Is value learning really the main bottleneck in offline rl? In Neural Information Processing Systems (NeurIPS), 2024.
[74] Scohong Park, Tobias Kreiman, and Sergey Levine. Foundation policies with hilhert representations. In International Conference on Machine Learning (ICMI.,, 2024.
[75] Scohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking offline goal-conditioned rl. In International Conference on Learning Representations (ICLR), 2025.
[76] Scohong Park, Qiyang Li, and Sergey Levine. Flow q-lcarning. In International Conference on Machine Learning (ICML), 2025.
[77] Xue Bin Peng. Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off policy reinforcement learning. ArXiv, abs/1910.00177, 2019.
14

---

[78] Jan Peters and Stefan Schaal. Reinforcement learning by reward-weighted regression for operational space control. In International Conference on Machine Learning (ICML), 2007.
[79] Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
[80] Scott Reed, Konrad Zoina, Emilio Parisotto, Sergio Gómez Colmenarejo, Alexander Novikow, Gabriel Barth-maron Mai Giménez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. Transactions on Machine Learning Research (TMLR), 2022.
[81] Oleh Rybkin, Michal Nauman, Preston Fu, Charlie Snell. Pieter Abbeel, Sergey Levine, and Aviral Kumar. Value-based deep rl scales predictably. In International Conference on Machine Learning (ICML), 2025.
[82] Nikolay Savinov, Alexey Dosovitskiy, and Vladlen Koltun. remi-parametric topological memory for navigation. International Conference on Learning Representations (ICLR), 2018.
[83] Paul J Schweitzer and Abraham Seidmann. Generalized polynomial approximations in markovian decision processes, Journal of mathematical analysis and applications. 110(2):688558)) 582, 1985.
(84] Harshit S. Sikehi, Qinqing Zheng, Amy Zhang, and Scott Niekum. Dual rl: Unification and new methods for reinforeement and imitation learning. In International Conference on Learning Representations (ICLR), 2024.
[85] Jayesh Singla, Ananye Agarwal, and Deepak Pathak. Sapg: Split and aggregate policy gradients. In International Conference on Machine Learning (ICMI.,, 2024.
[86] Jost Tobias Springenberg. Abbas Abdolmaleki, Jingwei Zhang, Oliver Groth, Michael Bloesch, Thomas Lampe, Philemon Brakel, Sarah Bechtle, Steven Kapturowski, Roland Hafner, Nicolas Manfred Otto Heess, and Martin A. Riedmiller. Offline actor-critic reinforcement learning scales to large models. In International Conference on Machine Learning (ICML), 2024.
[87] Martin Stolle and Doina Precup. Learning options in reinforcement learning. In Symposium on Abstruction, Reformulation and Appreximationn 2002.
[88] Richard Sutton. The bitter lesson, 2019. URL BitterLesson html.
[89] Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. IEEE Transactions on Neural Networks, 16:285-288 286, 2005.
[90] Richard S Sutton, Doina Precup, and Satinder Singh.
Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181-2 1999.
[91] Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist approach to offline reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2023.
[92] Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. Corl: Research-oriented deep offline reinforcement learning library. In Neural Information Processing Systems (NeurlPS), 2023.
[93] Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Won Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Q Tran, Dani Yogatama, and Donald Metzler. Scaling laws vs model architectures: How does inductive bias influence scaling? In Findings of the Association for Computational Linguistics: EMNLP 2023, 2023.
[94] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. ArXiv, abs/2550111254 2025.
[95] Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Neural Information Processing Systems (NeurlPS), 2017.
[96] Tongzhou Wang, Antonio Torralba, Phillip Isola, and Amy Zhang. Optimal goal-reaching reinforcement learning via quasimetric learning. In International Conference on Machine Learning (ICML). 2023.
[97] Ziyun Wang. Alexander Novikov, Konrad ZoIna, Jost Tobias Springenberg, Scott E. Reed, Bobak Shahriari, Noah Siegel, Josh Merel, Caglar Gulcehre, Nicolas Manfred Otto Heess, and Nando de Freitas. Critic regularized regression. In Neural Information Processing Systems (NeurIPS), 2020.
15

---

[98] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, al. Chain-of- thought prompting elicits reasoning in large language models. In Neural Information Processing Systems (NeurIPS). 2022.
[99] Yifan Wu, G. Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. ArXiv. abs/1911. 1361,, 2019.
[100] Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang. Victor Chan, and Xianyuan Zhan. Offiine with no ood actions: In sampa learning via implicit value regularization. International Conference on Learning Representations (ICLR), 2023.
[101] Greg Yang, Edward Hu. Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks via zcro-shot hyperparameter transfer. In Neural Information Processing Systems (NeurIPS), 2021.
[102] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y. Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. In Neural Information Processing Systems (NeurlPS), 2020.
[103] Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. In Neural Information Processing Systems (NeurlPS), 2021.
[104] Zihan Zhang, Xiangyang Ji, and Simon Du. Is reinforcement learning more difficult than bandits? near-optimal algorithm escaping the curse of horizon. Conference Learning Theory (COLT), 2021.
[105] Zihan Zhang, Yuxin Chen, Jason D Lee, and Simon S Du. Settling the sample complexity of online reinforcement learning. Conference on Learning Theory (COLT), 2024.
[106] Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. Archer: Training language model agents via hierarchical multi-turn rl. International Conference on Machine Learning (ICML), 2024.
16

---

A Offline RL scales well on short-horizon tasks
Flow BC
IQL
CRL
SAC+BC
Success Rate
100
100
100
100
cube
double
"earteen
0-00
2.50
SM
Success Rate
100
100
100
100
puzzle
4x4
"easter"
0
2.50
2.5H
Steps
Steps
Steps
Steps
Dataset Size
100X
1M
10M
100M
(a) Training curves.
Flow BC
IQL
CRL
SAC+BC
Success Rate
100
100
100
100
cube
double
"easser"
100K
1M
100M
1M
100%
Suc cess Rata
100
100
100
100
puzzle
4x4
"casser"
1M
100M
10M Dataset Size
1M 100%
Dataset
Size
Dataset Size
Dataset Size
(b) Data-scaling curves.
Figure 10: Offline RL scales well easier, shorter-horizon tasks. We evaluate flow BC, IQL, CRL, and SAC+BC on the same tasks with fewer objects, and show that they generally scale well on these simpler tasks.
To further verify the validity of our benchmark tasks as well as the offline RL algorithms considered in Section evaluate these methods on the same tasks with fewer objects: cube-double with cubes (as opposed to cube-octuple with 8 cubes) and puzzle-42h with 16 buttons (as opposed to puzzle-422 with 20 buttons). Figure 10 shows the training and data-scaling curves of flow BC, IQL, CRL, and SAC+BC on the two tasks. The results suggest that current offline RL algorithms generally scale well on these easier, shorter-horizon tasks. This confirms that our dataset distribution provides sufficient coverage to learn near-optimal policy, and serves sanity check for our implementations of these offline RI. algorithms
B Other attempts to fix the scalability of offline RL
In the main paper, showed that standard (flat) offline RL methods struggle scale complex, long-horizon tasks, and that horizon reduction techniques can effectively address this scalability issue. Are there other solutions to fix scalability without reducing the horizon? We were unable to find any techniques that are as effective as horizon reduction, and we describe failed attempts this section. Unless otherwise mentioned, below. We note that SAC+BC is the best method in Figure 10, and that behavior- rehavior-regularized methods of this sort achieve state-of-the-art performance standard benchmarks [91].
employ SAC+BC and the largest 1B datasets in the experiments
Larger networks. The first row of Figure shows the results with larger networks with ML.ss and residual MLPs (ResMLPs) [51, 70], up to 591M-sized models. To stabilize training, reduce the
17

---

cube-octuple
puzzle-4x5
puzzle-4x6
bumnaze -giiant
Succeas Rate
200
100
100
100
[0092)
loved (1roe)
MP
Drosi
(1499)
Rxs
[100] +17 (807)
0-4
0.
SM
SW
(0700
(Caduzit)
Shockea Rate
Teraort Type Ponan))
Traostormer (108)
(Cadazic)
Succeas Rata
100
100
100
100
(dstacis)
0-1
0-
Stccees Rate
:00
100
1.00-
1.00
Fenctions
-
5M
5M
5M
Succedo Rate
100
100
Pegatt Docar
Trme
Sucoess Rata
100
100
100
100
(dntactt)
0 0003
0 00023
0-
0-
SM
5M
BLcoRss Rate
TII
0.000 Coutiast)
Sicceas Rate
100
100
100
100
Sarih
2004 (Seduutt)
681
Stecress Rate
100
1.00
1.00-
1.00
Grudasn: Stapo
-
2511 (0n1an113
SM 10M 15M R0W 25M
SM 10M 15M 22N 75M
SM 10M 15M 20N P5K
0
BM 1OM 15M 30M 25M
Stops
Stopp
Stops
with reduction (bet aethod Secticn defaul: hyperparantters)
Figure 11: Nine different attempts to fix the scalability of offline RL without horizon reductions. The results show that of these fixes effective horizon reduction (denoted by the blue line) in general.
learning rate of the largest 591M model from 0.0003 to 0.0001, conceptually following the suggestion by Yang al. [101]. The results suggest that while larger networks can improve performance to some degree, simply increasing the capacity is not sufficient to master the tasks. On the other hand, horizon reduction enables significantly better asymptotic performance (denoted in blue) even with the default-sized models. We refer to the main paper (Section 4) for further discussion.
18

---

Transformers. We investigate whether replacing MLPs with Transformers [95] can improve perfor- mance. To handle vector-valued imputs with a Transformer, we first map the imput a T, -.-dimmnnio vector using dense layer, reshape it into length-7: sequence of Tx-dimensional vectors, pass through Tn self-attention blocks (with Tm ML. units) with four independent heads, and con- catenate the outputs for the final dense layer. We employ Transformers of two different sizes with Te, Tu, Tn, Im) = (2048, 16, 128, 4, 128) and (2048, 8, 256, 10, 1024). The former network has 3M total parameters and the latter has 41M total parameters. Due to the significantly higher computa- tional cost, we use a smaller batch size (256 instead of 1024) for runs with the larger Transformer, so that each run completes within three days. The second row of Figure 11 shows the results with Transformers. These results suggest that while using Transformers improves performance on some tasks, it still often falls significantly short of horizon reduction techniques.
More expressive policies. To understand whether a more expressive policy can improve performance, we train (goal-conditioned) FQL. [76], one of the closest methods to SAC+BC that use expressive flow policies [2, 59, 61]. The third row of Figure 11 presents the results, which suggest that simply changing the policy class does not improve performance the four benchmark tasks.
Larger Q ensembles. The fourth row of Figure 11 compares the results with 2 (default) and 10 Q networks. The results show that their performances are nearly identical.
Regularization. To understand whether additional regularization sue, we evaluate performance with weight decay (with a coefficient of 0.01, selected from {0.0001, 0.001, 0.01, 0.1)). We note that we use layer normalization [4] by default for all networks. The fifth row of Figure 11 shows the results. While weight decay yields a non-trivial improvement on one task (humano i dmaze- dmaze-giant), it does not improve performance the other three, more chal- lenging tasks.
address the scalability is-
Smaller learning rates (LRs) and target network update rates (TURs). The sixth and seventh rows of Figure 11 show the results with different learning rates and target network update rates. These results indicate that simply adjusting these hyperparameters does not substantially improve performance on the benchmark tasks.
Larger batch sizes. The eighth row of Figure 11 shows the results with larger batch sizes. While larger batches help on humanoidmaze- tasks.
they do not improve performance on the other three
Longer training. The ninth row of Figure 11 shows the results with 5x longer training (25M gradient steps in total). While extended training improves performance on humano iinaanoddazze-ieo -wwaz--gaaaaaaanee -iintt it does not yield significant improvements on the other three tasks.
Other attempts. In the earlier stages of this research, we tried a classification-bast loss with HL- Gauss [20, 37], but it did not lead to significant improvement in performance. We also tried residual TD error minimization [83] (i.e., removing the stop-gradient in the TD target), but we were unable to achieve non-trivial performance with the residual loss.
C Ablation studies of SHARSA
In this section, we present three ablation studies on the design choices of SHARSA. All results are evaluated on the largest 1B datasets.
Value learning methods. While SHARSA uses SARSA for the value learning algorithm, we can in principle use any decoupled value learning algorithm (i.e., one that does not involve policy learning) in place of SARSA, such as IQL [47] or its variants [25, 100]. The first row of Figure 12 compares the
performance of three different value learning methods within the SHARSA framework: SARSA, IQL with x = 0.7, and IQL with k = 0.9, where k is the expectile hyperparameter in IQL (Appendix E. The results suggest that the simplest SARSA algorithm is sufficient to achieve the best performance on our benchmark tasks, which partly aligns with recent findings [7, 18, 50].
Policy extraction methods. SHARSA uses rejection sampling for high-level policy extraction. In the main paper, we discussed how reparameterized gradient-based approaches may not be suitable for high-level policy extraction, due to potentially ill-defined first-order gradient information in the state space. To empirically confirm this, we replace rejection sampling in SHARSA with two alternative policy extraction methods based on reparameterized gradients: gradients: DDPG+BC [22, 73] and FQL [76].
19

---

cube-octuple
puzzle-4x5
puzzle-4x6
hummaze hummaze-giant giant
Surceea Rate
100
100
100
100
Leassing fistaed
O0L MIEE --00
Sucoses Bate
100
100
100
100
Rejection axppling
-om PQL
DarCHac
5M
SM
SN
Success Bate
100
100
100
100-
Valoo Loes
Begowaszon
0
0-
0-4
0
GM
SN
Steps
Stepe
Steps
Stepe
Figure 12: Ablation studies of SHARSA
The former extracts (high-level) Gaussian policy and the latter extracts (high-level) flow policy. We recall that SHARSA uses goal-conditioned BC for the low-level policy. The second row of Figure 12 presents the results. As expected, the results show that these reparameterized gradient- based methods perform worse than rejection sampling, especially on the puzzle e tasks, which contain discrete information (e.g., button states) in the state space.
Value losses. As explained Appendix E.3, we employ the binary cross-entropy (BCE) loss (instead of the more commonly used regression loss) for the value losses in SHARSA (Equations (25) and (26)). The third row of Figure 12 compares these two choices, showing that the BCE loss leads to better performance and faster convergence. While we do not provide separate plots, found that the BCE loss generally results in better performance, regardless the underlying algorithms.
D Additional results
Table 1: Horizon reduction improves performance in reward-based (non-goal-conndttonee)) RL too.
Tack
SARSA
IqL
SAC BC
D-SAC-BC
SEARSM (x = 0.5)
SHARSA (x = 0.7)
Harizon Reduction Type
Value policy
Value
cube-quadruple caattuuppp---l-eeeeee laly--eingletade taski-v0
0.
=s
D.
D:a0
50 2.
60
puzzle dzb play- pllay-singless siagletask taskki-rd
ia
24
0 0.40
0:40
94 43
94 a2
puzzle-4x6 plppppllayaaalaae saayy-atagietatttaaaaa
14
hunanoidmaze dmaze toee--gaan giant wtw.gggtttttgiitaa---eooe.e
0.
Results reward-based tasks. While focus goal-conditioned tasks in this work, the benefits of horizon reduction are not limited to goal-conditioned RL. To empirically demonstrate this, we additionally evaluate two horizon reduction techniques, n-step SAC+BC (which reduces the value horizon) and SHARSA (which reduces both the value and policy horizons), on four reward-based singletask tasks from OGBench [75]. We employ 100M-sized (cube) and IB-sized (others) datasets.
On these tasks, we evaluate SARSA, IQL (with 0.7), SAC+BC, restep SAC+BC, and SHARSA Additionally. we consider an IQL variant of SHARSA (with x = 0.7, Appendix C), which can be helpful as these singletask tasks have higher suboptimality due to the absence of hindsight relabeling. We use AWR with a = 10 for SARSA and IQL, and BC regularization with a = 0.01 (humanoidmaze)) or 0.1 (others) for SAC+BC and n-step SAC+BC.
Table shows the performance measured at the 1M epoch. The results suggest that these horizon reduction techniques significantly improve performance in reward-basse based offline RL as well.
20

---

Flow BC
IQL
CRL
SAC+BC
Saccess Rate
100
100
100
100
cube
octuple
0-9
0
0-000-0-0--0
0-0-9
0.49-
SM
SM
5M
5K
Success Rate
100
100
100
100
puzzle 4x5
SM
GM
6M
6X
Success Rate
100
100
100
100
puzzle
4x6
0-10-0
SM
SM
5M
5M
Success Rate
hummaze
giant
200-0000000--0000-0mm
10=0-00000
SM
5%
Steps
Steps
Steps
Steps
Dataset Size
1M
10M
100M
1B
with horizon reduction (best method Section
Figure 13: Training curves of standard offline RI. methods.
Flow BC
SAC-BC
nestap SAD+RC Rlararchical FRC
REQU
SHARSA
Double SHARBA
Value B0c3000
Paloe Recssos
Valle Recsoss
Feloe Rocuess
Pallve Roc
Valus Borizen
Sicccesn laste Staccmss late Surceus late Saucees lats
octuple
puzzle
4x5
10
pussle
1x8
Stopa
Stapa
Stupa
Stops
Stepo
Stepo
Dacapes Bi20
LM
JON
100M
1B
Figure 14: Training curves of horizon reduction techniques.
Full training curves. Figures 13 and 14 provide the full training curves of the methods considered in Figures and 9, respectively.
21

---

E Offline RL algorithms
In this section, we describe the offline (goal-conditooned) RL algorithms considered in this work. In the below. 7 E [0, 1J denotes the discount factor and & denotes the goal space, which is the domain of a goal specification function (*)))) : S + G. For example, in humano dmaae, ip is function that outputs only the z-y coordinates of the state. We also assume that the action space is d-dimensional Euclidean space (i.e., A = Rd), unless otherwise mentioned. We denote network parameters as (with corresponding subscripts when there are multiple networks). The goal-conditioned reward function r(s, g) S x G + R is given as either [s = g] (the 0-1 sparse reward function) or [s = gl - (the - 1-0 sparse reward function), where [:] is the Iverson bracket (iee., the indicator function for propositions). We use the former for classification -ased methods and the latter for regression-basee methods.
We denote the state-action-goal sampling distribution as yD. In general, p² a) is the uniform distribution over the dataset state-action pairs and pa s, is a mixture of the four distributions: the Dirac delta distribution at the current state (perr) geometric distribution over the future states (p (, the uniform distribution over the future states (Pimj). and the uniform distribution over the dataset states (P We refer Park et [75] for the full details. The ratios of these four distributions are tunable hyperparameters, which we specify in Table 3.
E.1 Flat offline RL algorithms
Flow behavioral cloning (flow BC) [6, 12, 76J. Goal-conditioned flow behavioral cloning trains a vector field (t s, 2,g) [0, 1] x S x RR G + Rd that generates behavioral action distributions.
minimizes the following objective:
L(O)= = E s.aep (olsa)) (U,L)) (ii.
[1,oa(4,,,'' g) (a -aa-- =)
a'=(1-)a+tt
where (n, 1J) denotes the uniform distribution over the interval [0, 1].
After training the vector field vo, actions are obtained by solving the ordinary differential equation (ODE) corresponding to the flow [52] generated by the vector field. We use the Euler method with step count of 10, following prior work [76]. See Lipman al. [60], Park et al. [76] for further discussions about flow matching and flow policies.
Implicit Q-learning (IQL) [47, 72]. Coal-conditioned IQL trains state value function Vo, () SxG + R and a state-action value function Qo (s, a) S xxxSxxo Axg g- + E with the following losses:
(0v) (xa) (n,o), gndp (slora)
[ Vor (8, Qba (s, ()))
(5)
(0())) ((a.as'')pp (saa))) g~ g~p (als,a)
(Qna (s, (s. r(s,g) Voo (s', .0))))
(6)
where ea denotes the expectile loss, (x) = k - 1x O] and 0g denotes the parameters of the target Q network [65].
From the learned Q function, extracts a (Gaussian) policy no, (a s,9) S g - AAA) by maximizing the following DDPG+BC objective [73]:
(=x) (ps (s,0), (gle,o) [Qe (s, no, ().g).g) g) + a a Te, (a s, s.g)]
(7)
where μ denotes the mean of the Gaussian policy Te, and denotes the hyperparameter that con- trols the strength of the BC regularizer. While the original IQL method uses the AWR objective [77], use DDPG+BC as Park et al. [73] found it to scale better than AWR.
Contrastive reinforcement learning (CRL) [18]. CRL trains a logarithmic goal-conditioned value function fo (s, 0,9) : S A R with the following binary noise contrastive estimation objective [63]:
J (00) = (sa))pppplp g (p
[log (fe, (s, g)) + log(1 - o (fo, a, )))) (8)
22

---

where Ø: IR - (0) is the sigmoid function, P. is geometric future goal sampling distribution, and p2 is the uniform goal sampling distribution. Eysenbach et al. [18] show that the optimal solution f* to the above objective is given by (s, a, g) log (s, a. + C(g), where QMC is the Monte- Carlo value function and C' is a function that does not depend on s and a. As in Eysenbach et al. [18], we employ an inner product parameterization model fo, as f(s.a.g) g) t (s, a) (2) (where we omit the parameter dependencies for simplicity) with k-dimensional representations,
4/1 8 x A + R* and 1/2 9 R*
Similarly to IQL., CRL extracts a policy with the following DDPG+BC objective:
J (0x)
L (s.a)))PP(P (10), (slsp())s.a) [fe, (s, Me, (). + log me, (a s.
(9)
Soft actor-critic + behavioral cloning (SAC+BC). SAC+BC the SAC [27] variant of TD3+BC [22, 91]. We found SAC+BC to be generally better than both TD3+BC [22] and its successor Re- BRAC due to the use of stochastic actions in the actor loss (note that TD3 [23] uses deterministic actions in the actor objective), which serves as regularizer in the offline RL setting. Goal-conditioned SAC+BC minimizes L7 and maximizes J below to train a Q function Q9a and a policy TT,
(00)) = (s,,,,") (soa)") gmppp g [ (8, a, - -((,,) -Q000 (s',a" a, .9)) 21 a m (ola' s)
) (10)
J" (0x) = (aaapppp (s.a), MTee (013.0)
Qr0 (8, a", .) - alla" - all2 A log me. (a |.
(11)
where a is the hyperparameter for the BC strength and À is the entropy regularizer (which is often automatically adjusted with dual gradient descent to match a target entropy value [28]).
On goal-conditioned tasks with 0-1 sparse rewards, since we know that the optimal Q values are always in between O and 1, we can instead use the following binary cross-entropy (BCE) variant for the Q loss [42]:
LICE = E >'p p (s,a,)) (sls,)) a ""w (ala'
(o, [BCE (Rog a,ls,,,g)) r(.r) + 18 (s', as',a", a.,,a)
.9)]]
(12)
where BCE y) = log (1 y) log(1 - 2). We found this variant to be generally better than the original regression objective, as it focuses better on small differences in low Q values, which is crucial for extracting policies on long-horizon tasks. When using the binary cross-entropy variant,
we model the logits of values (instead of the raw Q values) with neural network, and use the logit values in place of the Q values in the SAC+BC actor objective as follows:
Jic (0,) ) = E (s.a)=pp (s.a). a" rotT, (4)x.g)
[logit Qo (s, a" .o..g) - alla" x all2 Alog To, (a" ss g)]
(13)
We found the use of logits in the actor loss to be crucial long-horizon tasks, it applies more uniform behavioral constraints across the state space.
Flow Q-learning (FQL) [76]. FQL is a behavior-regularizz offline RL algorithm that trains flow policy with one-step distillation. Goal-conditioned FQL trains a Q function Qag (8, a,9) S x A x G + R, a BC vector field vo, (1, 8, 2. [o. 1] x S x Rd x g - Rd that generates a noise- conditioned flow BC policy no (s, (5)2 S x Rd x G + A, and a noise-conditioned one-step policy
me, (s, (,)) : S Idd x G + A, with the following losses:
(+*) = (e.,))x (s,a), (oloa)) (o) (. [1=(1_])tta
[1000 (t, s,a.s - (a ((a-2)) 2) 2)122]
(14)
- (1, - r((,g) - -Q000 (s',a" a, .g)) 27 3g)²²
(9())) = (E jop (aap)) g- ((la))a)) (a), ="= (0(12.)
(15)
(O))0)) (((a)) ((.a), (s,a),
--Q ua_ (8, 2, + allpo (8, pa, (s, 2
(16)
p~p (glssa),
(0,,a)
where a is the BC coefficient. The output of FQL is the one-step policy 140 In our experiments, we use the binary cross-entropy variant of FQL in our experiments, which replaces the regression loss in Equation (15) with the corresponding binary cross-entropy loss, as in Equation (12).
23

---

E.2 Hierarchical offline RL algorithms
n-step soft actor-critic behavioral cloning (n-step SAC+BC), n-step SAC+BC is variant of SAC+BC that employs n-step returns. The only difference from SAC+BC is that it minimizes the following value loss:
D ((000 an. g),
(0)) L (on on on Dapp, gr (glen.as)
y Fr(ssts, + (9) a² (.9)
==0
a" Nxo, (alph+a.9)
(17)
where omit the arguments in pd (sh, 0h........................ ........ 8h+4m and D is either the regression loss Reg(r, (x - y)² or the binary cross-entropy loss BCE(x, y) -y log æ - (1 - y) log(1 x). We found that the BCE loss performs and scales better in our experiments. In practice, we also need to handle several edge cases involving truncated trajectories and goals in the above loss; we refer the reader to implementation for further details.
Hierarchical flow BC (hierarchical FBC). Hierarchical flow BC trains two policies: a high-level
policy v (w (. S x G - A(g) and a low-level policy n (a s, w) S x g - A(A), where we denote subgoals by W. The high-level policy is trained to predict subgoals that are steps away from the current state, and the low-level policy is trained to predict actions to reach the given subgoal.
Both policies are modeled by flows, with vector fields vi, (4, 5, R, () : [0, 1] x S x R x + R
and v (t, s, 2, w) [0, 1] x S x IRd x g + RR where we assume that the goal space is G = R These vector fields are trained with the following flow-matching losses:
I." (Oh)) = E (sn,
youp, grup" (slen.a)), (0, (nUnl((())))). w²
[1rbb, (f, 8h, w², .9) (8((t)))
(18)
(024h)
I. (0t) E (s n >mp so.Ta) (- a o1=
[1 (t, sn,a", sh4n) (an - 2)1(2]
(19)
Hierarchical implicit Q-learning (HIQL) [72]. HIQL trains a single goal-conditioned value function with implicit V-leaming (IVL) [75], and extract hierarchical policies (T]), and Th with AWR-like objectives [77]. It trains value function Ver S x G + IR that is parameterized as Ve, gs. Ver (s, vor (8,g)) with representation function they (s,g) S G + R$ and remainder network Vey (s, =) : x R* TR, where do not distinguish the parameters for v/, V. and V to emphasize that they are part of the value network. The IVL value loss is as follows:
L (0v) =
(slP())).a) [ (Vay (8,9) - r(s,g) 7V,, (8' (6).))].
(20)
where e is the expectile loss described in Appendix E.1. From the value function, it extracts two policies by maximizing the following AWR objectives:
Jh (0i, = E ab (opp))))) gmop" " (ol)),an)
(V log Tb. (vov (sh, sh+w) I 1hh 9))
(21)
Jd ((()))
( (etViontiionnn log T6, (an shi vor (sh, snanl)
(22)
where a is the inverse temperature hyperparameter for AWR. Similar to n-step SAC+BC, there are several edge cases with truncated trajectories and goals, and we refer to our implementation for the full details.
E.3 SHARSA
SHARSA. SHARSA is our newly proposed offline RI. algorithm based on hierarchical flow BC and n-step SARSA. It has the following components:
High-level BC flow policy T²3,0, (w () S x x A(9),
Low-level BC flow policy =5.0, (a s, S A(A),
24

---

ststep Q function Qo0 (s, S x g x g + R
St-tep V function Vor (5,g) : S x 9 - IR.
As in hierarchical FBC, the policies are modeled by vector fields, 22 (t, (,,8,22 2, 9) [0, x S Rm x
g + Ih and og (t, s, 2, w): [0, 1] x S IR x - IRd, where we recall that G mm and A = Rd. They are trained via the following flow behavioral cloning losses:
L (Bn)) E
[lod), (t, sn. w., (89((14))=211p (23)
(sh Shoa japp grup" (alsmaan) 20, fm) (n w w'=(1- (1 (81+n)
(0:) E (s jup (0. (-.1). df=(1-t)++on)
[1 (1, 8h, 8h4n) (an - =)]12]
(24)
where we recall that Py is the goal specification function defined in the first paragraph of Appendix The value functions are trained with the following SARSA losses:
LV (0v) (sn, sn pmmpp grop" (aN
D Vbl (sns 9). Qbg (shs Shan.gg)]
(25)
D @ (8h, 8h g):
LZ(00) = -E (eennon on.... an) admup? grop" ponp((llu (nl)
y' ('r(sn+isg) + V (sh g)
1=
(26)
where D is either the regression loss Reg(z, 8) (x y)² the binary cross-entropy loss BCE(I, y) = yylog.z - (1 - y) log(1 - z). As before, we found that the BCE variant works better on long-horizon tasks.
At test time, we employ rejection sampling for the high-level policy. Specifically, it defines the distribution of the high-level policy o0 (w 8,9) 8 (g) as follows:
h (8,g)
arg max
S (s, wi,g),
(27)
O 9. (w|s.g)
where N is the number of samples. SHARSA simply uses the behavioral low-level policy; i.e., Té, (s, w)
d T/3.0, (s, w). We provide the pseudocode in Algorithm 1.
Double SHARSA. Double SHARSA employs an additional round of rejection sampling in the low- level policy further enhance optimality. To do this, defines additional low-level value networks:
Low-level Q function Q (s, a, w) S x A x G + R.
Low-level V function Vgg (s, w)
R.
They are trained with the following SARSA losses:
T."(O.) E "i...a 8h hupp Dop
D Vol (sh, Shta), L6, (8h, (Sh,
an, (htt))
(28)
[,,)))) (sn Sh1 )mpp
D @
(h, (Sn, an, 8h+m), r(hh, 8h+n) + VV,, (8/41: (hen))s (29)
where 5 is the low-level discount factor defined as 1-1/n. Ar time, double SHARSA defines the distribution of the low-level policy The (a s,w) S x g + A(A) with rejection sampling:
T (s, w)
arg max
2b, (8, az, w).
(30)
01 No (als,w)
We provide the pseudocode in Algorithm 2.
25

---

Algorithm SHARSA
> Training loop
while not converged do
Sample batch {(N,, ......., Sh 9)} from D
Hicrarchical flow BC
Update high-level flow BC policy T (8h-n I 8h, 9) with flo--machhing loss (Equation (23))
Update low-level flow BC policy a2(aa, sh, sh+n) with flow-matoe fow-matching loss (Equation (24))
High-level (re-step) SARSA value learning Update V to minimize (D (V* (sh, g), (sh sh+n:9)) 9))
Update Q" to minimize [D (Q" (sh,,,, 8h+n; 9), 2is0 Y ('r(sh+i,)) 9) + + ( 9))
return (3, g) defined below
Resulting policy
function m(s,9) g)
High-level: rejection sampling
Sample wi...., Wi ~ (5.9)
Set + arg max Q* wx. 9)
LLow-eevel: behavioral cloning
Sample ())
return
Algorithm 2 Double SHARSA
Training loop
while not converged do
Sample batch [(S), ak. ak...., Sh.mn 9)} from D
>- Hierarchical flow BC
Update high-level flow BC policy T/ (sh-n (h. g) with flow-matching loss (Equation (23))
Update low-level flow BC policy (a 18h, 8h+n) with flow-matching loss (Equation (24))
High-level (n--step) SARSA value learning
Update V to minimize D (V* g), Q'h Sh+-: sh+n;g))]
Update Q" minimize [D (Q* (sh, 8h+ns 9), En-1 7 ('r(8h+))g) 9) (sntan g))]
Low-level SARSA value learning
Update V to minimize E (8h. 8htn), ('(n, an, Sntn))]
Update Qf minimize E DD (Qh, ab, S14a), r(sh, sh+n) + V shtn))]
return (s, defined below
Resulting policy
function ms,
> High-level: rejection sampling Sample wi...., WN ~ (s.
Set w <- arg max w.......... (x,
Loowllevel: rejection sampling Sample 01,...,aN T w)
Set a arg mag........y Q ww)
return
26

---

F
Experimental details
We implement all methods used in this work on top of the reference implementations of OGBench [75]. Each run in this work takes more than three days a single A5000 GPU, We provide our implementations and datasets at https::// //ttts://gitubb github. com http://oooaprtt//tooooooorrrraee
F. Didactic experiments
In this section, we describe additional experimental details for the experiments in Section 5.1.
Task. The combination- lock task consists of states numbered from to H and two discrete actions. Each state is represented by flog2 H -ddmennioonal binary vector. The ordering of the states randomly determined by a fixed random seed, which ensures that all runs in our experiments share the same environment dynamics.
Algorithms. We consider 1-step DQN and n-step DQN (with n 64) in Section 5.1. The value losses for 1-step DQN and n-step DQN are as follows:
L ss-step(() (0) = E
(o(Ss, an) r(sh, an)
max assIEd
(0(s))1,0++1)
(31)
(outenn (h))shaan) an)
n-1
Qo(sh (h )
Ln-step (0) E
Σ ((sn+i)
max anen EA
(32)
where the expectations are taken over consecutive state-action trajectories uniformly sampled from the dataset. We also employ double Q-Jearning [32] to stabilize training.
Datasets. We generate two types of datasets. The first a 1- step uniform-coverage coverrag dataset, collected by sampling state- action pairs uniformly from all possible 2H tuples. The second is a 64-step uniform-coveraae the following procedure: first sample a state uniformly from H states, and then perform either 64 consecutive correct actions (with probability 0.5) or 64 consecutive incorrect actions (with probability 0.5). Note that the former provides Figure 15: 1-step DQN on two datasets.
Succesa
Succeee Babe
100
Dusezet: Type
- t-rnag instors 6t-rzop ustfors
dataset, collected by
266 512 1004 2048
Langth
uniform state-action coverage for 1-step DQN, and the latter provides uniform state-action coverage for 64-step DQN. We use the 1-step uniform dataset for 1-step DQN and the 64-step uniform dataset for 64-step DQN. Since 1-step DQN works worse on the 64-step uniform dataset (Figure 15) and 64- step DQN incompatible with the 1-step uniform dataset, this setup provides a fair comparison of the maximum possible performance of the two algorithms, with the dataset factor marginalized out.
Metrics. We train each agent for 5M gradient steps and evaluate every 100K steps. We measure three metrics: success rate, TD error, and Q error. The success rate is measured by rolling out the deterministic policy induced by the learned Q function, averaged over all evaluation epochs. The TD error is measured by the critic loss (Equations (31) and (32)), averaged over steps on and after 4M. The error is measured by the difference between the predicted Q values and the ground-truth Q values (i.e., the negative of the remaining steps to the goal), evaluated at the final epoch.
We provide the full list of hyperparameters in Table 2.
F.2 OGBench experiments
Tasks. We use three existing tasks and one new task from OGBench [75]: humanoidmaze wwwiddddaeeeeaaaiieeeee giant, puzzle-4x5, puzzle-4x6, and cube-octuple, In the cube domain, we extend the most challeng- ing existing task, quad-uuar (with cubes), to create new task, cube-octuple (with cubes), to further challenge the agents. All these tasks are state-based and goal-conditioned. We employ the oraclerep variants from OGBench, which provide ground-truth goal representations (e.g., in cube, the goal is defined only by the cube positions, not including the agent's 's propriocep- tive states). This helps eliminate confounding factors related to goal representation learning. For the cube-d double task used in Figure 10, we exclude the swapping task (task4) from the evaluation
27

---

goals (Figure 19), as we found that this task requires non-trivial degree of distributional generaliza- tion. We refer Figures to 21 for illustrations of the evaluation goals, where the goal images for existing tasks are adopted from Park et al. [75].
Datasets. On each of these tasks, generate 1B-sized dataset using the original data- sata-generation script provided by OGBench. The cube and puzzle datasets consist of length- angtth11000 trajectories, and the humanoi dmaze dataset consists of length-4000 trajectories, as in the original datasets. These datasets are collected by scripted policies that perform random tasks with a certain degree of noise. In humanoi dmaze, the agent repeatedly reaches random positions using (noisy) expert low-level controller; in cube, the agent repeatedly picks a random cube and places it in random position; in puzzle, the agent repeatedly presses buttons in an arbitrary order. Notably, these datasets are collected in an unsupervised, task-agnostic manner (i.e., in the "play"-styl [62]). In other words, the data-collection scripts are not aware of the evaluation goals.
Methods and hyperparameters. We generally follow the original implementations, hyperparameters, and evaluation protocols Park al. [75]. We train each offline RL algorithm for 5M gradient steps (2.5M steps for simpler tasks in Figure 10) and evaluate every 250K steps. At each evaluation epoch, measure the success rate of the agent using 15 rollouts on each of the 5 (4 for cube-doube e) evaluation goals. For data-scaling plots, we compute the average success rate over the last three evaluation epochs (i.e., 4.5M, 4.75M, and 5M steps), following Park al. [75].
The hyperparameters (in particular, the degree of behavioral regularization) of each algorithm individually tuned each task based the largest 1B datasets. We provide the full list of hyperparameters in Tables and 4, where we abbreviate n-step SAC+BC as n-SAC+BC and double SHARSA DSHARSA.
G Result tables
We provide result tables in Tables 5 and 6, where standard deviations are denoted by the "*" sign. In the tables, we abbreviate flow BC as FBC, hierarchical flow BC HFBC, n-step SAC+BC as m-SAC+BC, and double SHARSA as DSHARSA. We highlight values at or above 95% of the best performance in bold, following Park et al. [75].
28

---

Table 2: Hyperparameters for didactic experiments.
Hyperparameter
Value
Gradient steps
5M
Optimizer
Adam [46]
Leurning rate
0.0003
Batch size
MLP size
[512.512.512
Nonlinearity
GEILG [34]
Layer normalization
True
Target network update rate
(1.055
Discount factor ?
Horizon reduction factor
64
Table 3: Common hyperparameters for OGBench experiments.
Hyperparameter
Value
Grudient steps
5M (default), 2.5M (cube-double, puzzle-4x4)
Optimizer
Adam [46]
Leurning rate
0.0003
Batch size
1024
[1024, 1024, 1024,
Nonlinearity
GELU [34]
Layer normalization
Target network update
0.005
Discount factor
0.999 (default), 0.99 (cube-double, puzzle-4x4)
Flow steps
Horizon reduction factor
(cube, humanoidmazan 25 (puzzle)
Expectile (IQL.)
0.9
Expectile (HIQL)
(cube, humanoidnaze), 0.7 (puzzle)
Value representation dimensionality & (CRL)
1024
Goal representation dimensionality & (HIQL)
128
Double Q aggregation (SAC+BC. SHARSA, FQL)
min(Q1. Q2) (cube, puzzle), (Q1 + Q2) (bumanoidmaze)
Value loss type (SAC+BC, SHARSA, FQL)
Binary cross entropy
Actor (rean H
goou PgoonrPeaa P Paaga
ratio (BC)
(0, 0,
Actor (Rean (Pean Peeareoonnr PegcarPega Paod ratio (others)
(0, 1, (cube). (0, U,,, 0.55) (puzzle), 0, 1,0) (humanoi 0,
Value (pear
Pegarroea-PPuu (ogaj-P Panal ratio (CRL)
Value (pem (Pariggomm Perar-gromooiiee Panaa) ratio (others)
(0.2, 0.0.5.033
Policy extraction hyperparumeters
Table
Table 4: Policy extraction hyperparameters for OGBench experiments.
SAC+BC
FQL
S-C+AC
SEARSA
DSHARSA N
cube-octuple
10
0.1
32
puzzle 4x5
0.3
3
0.1
32
32
puzale-4x6
0.3
0.1
32
32
humanoidnaze-git dwdddmazzeggannt.
0.3
0.1
0.03
32
32
cube double
10
1
puzzle-4x4
0.3
29

---

task1
task2
task3
task4
task5
Figure 16: Evaluation goals for puzzle-4x4.
task1
task2
task3
task4
task5
Figure 17: Evaluation goals for puzzle-4xx
mazzle-4x5..
t
task1
task2
task3
task4
task5
Figure 18: Evaluation goals for puzzle-4x6.
30

---

task1 single- pup
task2 double-papi
task3 double-pap
task4
task5 stack
svap
Figure 19: Evaluation goals for double As stated in Appendix evaluation.
F2, task4 is omitted from our
task1 quadrople-pap
task2 eetuple-ppp1
task3 octuple-pap2
task4 stack!
task5 stack2
Figure 20: Evaluation goals sube-octupoe
PEAG
task1
task2
task3
task4
task5
Figure 21: Evaluation goals for humano..mazz--ii.. giant.....g.....................
31

---

Table 5: Full results (at 1M epoch).
tuck:
4o
0-
0o
0.
0
ta
Bize
0:30
0:21
0:
11
taaich
0:
g0
taak
0.2
0.aa
0:sn
217
3.27
R5 2011
55
0.2a
g0
2
tasks
cuba-actuple
0962633
13
188
0:
0.30
100M
Tosks
0:m
0
0
0:
19 m4
19
10
10284
0:m
0.
0
mm
0::e
0.
0.
0
overa33
0.ux
0:p0
20 Lm
20
took2
1M
tankc
cm
0.00
0.54
0
0:00
0.2
0.49
0.0
Daa
63
09
taxk2
0.s
0.s0
13
S.s
95
10K
tack4
0.
Uax
0.s
Uaa
0:m0
0:00
0.0
0.29
T5 137
0.
pozele-4r5- www.aaaggggaooaergggo.o lerep--o
tank:
98
0.s
75
25:
100
100
0.50
15:
2.55
2:0
100
100
100M
0.u
Uax
Maa
D.sa
0:m
87
02
72
0.29
taak:
0.zx
100 o4
7as
95
70
35 2:4
100
100
tack2
0.
Uas
Maa
25 km
08
100
taaks
0:
98
tarks
0.29
0762611
20.40
1.
19
20
4.42
00
90-
tock1
0ax
Dan
20 am
tark4
0
tuaks
0-
0.
0
0.
15
3:
0.2
0.2
0.g
taak2
Dasa
S
100
100
Das
45
taces
0.30
0.2
0.2
0:20
puzzle- 408- play-orocheet ococlerpp--
overall
19
Das
14-3
G-2
1:t
2-2
47 m2
62
tark?
0.29
40.00
100M
teaks
0.29
Das
0:an
0.29
2:
0.2
05
n2
05
00
tooks
0-e
0.
0.4
Daa
2.2
67.00
08
17
Tas
37
100
1000
teak2
Das
02
00
0.z
0:30
Bam
0.23
0.29
65
70
t03b4
Bas
Das
tooks
25
0-
2-
38
62
3=
5.3
30-17
0.2
Tax
13:an
ST
1M
tosks
0-x
Q.
Bas
15 m0
Sar
10 km
8
T
17
37
tank1
0.2
2.3
22.2
3.24
55 15
10:24
30
t02k2
2-3
Gas
87 413
90 cm
45 asa
33. aus
taaks
S
85
75
37
task5
23
008781
88
4.
50
19
22
51
tooks
0..
07 km
58
13 a5
10 412
22 4..
3
19
100M
tosld
82
tazi0
3.x4
8.x
75
0mm0
98
45
0987611
4.2
70
7.44
82
21.
29
43
37
task?
0.25
0.a
2M4
5.5
68
10:27
13
36 2m
tosk
77
20
taxi4
2:3
87
2.3
77
10
0..
3.4
06
08
45
35
43
07
overa]]
0.x
314
02 u4z
5:50
82
19 m5
25 am
45
30
32

---

Table 6: Full results (averaged over 4.5M, 4.75M, and 5M epochs).
taski
0-
0.s
0
tark2
0:2
0:2
0:24
gn
1M
tarks
t02k4
0.2
0=2
g0
(1:2
cm
tank1
0.29
U.a
0.an
on
06
79
tark2
0g2
0.2
an
taok3
Dau
10
tark1
0.2
0.59
Bann
an
87.21
task5
0am
Dau
Gas
1.
0787612
so
25 m4
30
22
task3
29
ta184
0am
Dau
taxks
0.m
sm
4:14
3.
0.sx
overa1l
0aa
27 a4
21
tosk2
1M
tasks
tarks
0.m
1:3
0.00
tacks
0.2
0.e9
8.42
100
taxk2
0.m
55
100
10C
tasks
tarks
0.m
D.s
Daa
mm
H5
21 ssa
Daa
0:
cm
98
516
0..9
1:
task]
100
26
46
100
100
task2
1.x1
R1 si
99
100
100M
talk3
taxks
0.m
0.s
65 am
cm
80
94
00
overa1l
71
tank1
1.21
ST
96
79
96
26 sk
100
100
talk2
1.1
11a
74
aaz
as
100
100 a
tasks
0.m
0.s
74
08
tarks
61
5M
08
0rerall
19
71
D-42
10..
09
tark1
2.
44
Maa
Uax
task2
tark
tasks
0.s
o0ara11
0.2
0an
24
task]
64
3.
100
100
task4
tanks
0-
Dis
60
09
tarks
0:3
0.2
puzzle And play-pay--aa oroclezep-v0
ovsrall
0.
18
Das
14..
2:
01..
tark?
35
77
100M
tarks
tasks
0.>
19
1:l
73
02
91
tooks
0-
0-
0-m
0as
0
0-
overall
2
tark1
57
665
100
task2
Bas
76
83
74
10
tanks
taok4
0.2
Dan
0.n
59
Bai
mm
91
US
task1
0.e
30
2
3:
12
38
tank?
1:1
2.>
39
12 2m
16
55 2.
1N
took3
1at
Das
29
S:
21.4
36
11.4
30
24.44
G8
tark1
2,>
7.55
K1.
16
71.27
11 27
51.14
35.25
71 235
taok2
13
25
S1 -15
40 a13
13
58.44
87
5a5
78
36
27
12
Frmancidaazs oraciaergg----
12
84
22
85
taok:
4.
6.43
TL -1:
50 a1
70
18
50 013
06
75
14.00
30
56
tasks
17
100
12
91.11
95
09
0vera1l
0.2
17:b0
70
36
01
27
76
59
05
task1
Sas
8.3
[3
30
10
tasks
tosk3
7L
02
52
22
2.
73
50..
47
61
tast5
12:
16.
77
14 am
08
30
59
94
00
overa11
6.2
14
00
use
91
35
72.45
51.44
74.0
33

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
We support the claims via empirical experiments (Sections to 6).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See the second paragraph of Section 7.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
34

---

Justification:
We do not have theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide the full implementation details and code.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
35

---

Answer: [Yes]
Justification:
We provide the code and data-generation script.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
//nps.ccc/ httpp/ubllccceeee cubbic/guides. / wwabbbmmssionnPoo ssionnolicyy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperpa- rameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide the full implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report 95% confidence intervals in all plots.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
36

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //tt/rnrrss...// cc httppp/lccttthhccciiceleeeeeesse
Answer:
[Yes]
Justification:
We conform the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This is purely algorithmic research.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
37

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not pose such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly acknowledged the code and datasets used the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
38

---

Answer: [Yes]
Justification:
We provide instructions to reproduce datasets and experiments.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non- standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
39

---

The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to our LLM policy (https: / wtw.urpp..c/ooooeeeeeeeeeeeeeeee e for what should or should be described.
40


Reviews:
[{"review_id": "o40pyHaGoe", "reviewer": "Reviewer_2ypq", "summary": "This paper studies the scalability of standard MLP-based offline RL algorithms. Ideally, provided with a huge expert dataset that widely covers the state and action spaces, a good offline RL algorithm should be able to achieve perfect performance regardless of task difficulty. However, by increasing the size of the dataset from 1M to 1B, the author shows that current offline RL algorithms fail to achieve non-trivial performance on four challenging long-horizon OGBench tasks. Then, by demonstrating with a toy environment, Combination-Lock, the author hypothesizes that (1) the bias accumulation in TD value learning over lone horizon, and (2) the inability of direct mapping between optimal actions and distant goals in policy learning, orthogonally hinders the scalability of offline RL algorithms.  Therefore, the author discusses several horizon reduction methods that can mitigate the bias accumulation in value learning, e.g., n-step returns, or decompose the long-horizon policy extraction into a hierarchy. Moreover, the author proposes (Double) SHARSA that combines horizon reduction in both value and policy learning in a minimalist way. Experiments show that horizon reduction can effectively improve the offline RL performance as the data size scales up.", "questions": "1. In Line 233-235, the author describes two types of datasets used for the Combination-Lock environment, evaluates 1-step DQN and 64-step DQN on both datasets, and reports the best performance for each method. How do you perform the 64-step DQN on the dataset of uniform coverage of length-1 trajectory segments?\n2. This work mainly focuses on the *MLP-based* offline RL methods. Could the author comment on the scalability of *Transformer-based* offline RL or imitation learning methods, e.g., Behavior Transformer (BeT), Vector-Quantized Behavior Transformer (VQ-BeT), or Q-Transformer? Does transformer architecture bring any advantage of scalability to those methods naturally? Will the curse of horizon hamper their scalability to the same extent as it hinders the MLP-based methods?", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "### Strengths:\n- The paper is coherent, well-organized, and easy to follow. The figures are clean, informative, and contribute effectively to the overall presentation.\n- The research question is clearly defined and appropriately scoped, with no overclaims. The hypotheses are supported by enough empirical results. A thorough ablation studies are provided in the main body and in the appendix, which addresses many potential concerns regarding the paper’s arguments. And the experimental details are well-documented in the appendix for reproduction.\n- The research question should be of interest to the RL community, as RL scalability remains underexplored compared to fields like NLP and CV. The paper touches on several types of offline RL algorithms and evaluates them across four meaningful control tasks. The insight on horizon reduction is actionable, and the proposed methods and experimental protocol have the potential to inspire future work in improving offline RL scalability.\n---\n### Weaknesses:\nI do not identify major weaknesses, and a few comments are listed below:\n- Regarding the novelty, while the curse of horizon has been discussed in the RL literature, and the proposed method (SHARSA) builds upon existing techniques, the paper's central contribution—the connection between the curse of horizon and offline RL scalability—is both insightful and actionable. The paper has the potential to serve as a benchmark and reference point for future research in this area.\n- In Line 276-277, the author borrows an analogy to Chain-of-Thought reasoning. Although I understand what it conveys, it appears a bit irrelevant and implies little to the hierarchical policy learning discussed in this section.\n- In Line 321-331, the author describes the SHARSA value learning process but does not mention how horizon reduction is done here (although details are included in the appendix). This part would be more self-contained if the author briefly mentions that n-step return is used for high-level SHARSA value learning, while 1-step return is used for low-level (double) SHARSA learning.", "was_revised": true, "final_justification": "I did not identify the major weakness of this paper, and I list its positive aspects below:\n\n- This paper raises and answers a research question that should be of interest to the RL community, as RL scalability remains underexplored compared to fields like NLP and CV. \n\n- The paper touches on several types of goal-conditioned offline RL algorithms and extensively evaluates them across four meaningful control tasks with data scaling. \n\n- The insight on horizon reduction is actionable, and the proposed methods and experimental protocol have the potential to inspire future work in improving offline RL scalability. \n\n- During the rebuttal, the authors have addressed my questions accordingly. \n\nTherefore, I recommend acceptance.", "rebuttal": "Thank you for the detailed review and constructive feedback about this work. Please find our response below.\n\n* **Could the author comment on the scalability of Transformer-based offline RL methods?**\n\nThanks for this question! We also had a similar question, and have already investigated whether using Transformers instead of MLPs can improve the scalability of offline RL in Appendix B (L652-663). The results in Figure 11 show that, although replacing MLPs with Transformers improves performance to some degree on two tasks, it does *not* necessarily substantially improve scalability in general, and does not match the performance of horizon reduction techniques. This further supports our hypothesis that the horizon is a fundamental bottleneck in offline RL scaling.\n\nThat being said, we believe recent advancements in policy architectures (e.g., BeT, VQ-BeT, etc.) might independently contribute to performance improvements, orthogonal to horizon reduction. Finally, while Q-Transformer shows promising initial results, our analysis suggests that the way Q-Transformer deals with the horizon (i.e., treating individual actions as separate steps) might lead to suboptimal scalability, as it effectively *expands* the horizon rather than reduces it.\n\n\n* **How do you perform the 64-step DQN on the dataset of uniform coverage of length-1 trajectory segments?**\n\nAs the reviewer correctly pointed out, 64-step DQN is not applicable to the 1-step uniform dataset, so we evaluated 64-step DQN only on the 64-step uniform dataset (Appendix F.1, L857-858). On the other hand, we evaluated 1-step DQN on both the 1-step and 64-step uniform datasets and selected the best-performing one. Even though this selection procedure favors 1-step DQN, we show that 64-step DQN outperforms 1-step DQN. We will further clarify this point in the final version of the paper.\n\n* **Further details about SHARSA and double SHARSA (L321-331)**\n\nThanks for the feedback. The difference between SHARSA and double SHARSA lies in the low-level policy: SHARSA uses behavioral cloning for the low-level policy, whereas double SHARSA uses an additional round of rejection sampling based on another (1-step, short-horizon) SARSA value function (Appendix E.3). As the camera-ready version allows one additional page, we will include detailed explanations of SHARSA and double SHARSA in the main paper of the final version. \n\n* **Analogy to chain-of-thought reasoning is less clear (L276-277)**\n\nThanks for the feedback. We wanted to mention that hierarchical policies and chain-of-thought reasoning have a common ground in the sense that decomposing a problem into subtasks can improve performance. We will further clarify this analogy in the final version of the paper!\n\n---\n\nWe would like to thank you again for raising important questions about Transformer results and other clarification concerns, and we believe the additional clarifications have strengthened the paper. Please let us know if you have any additional concerns or questions."}, {"review_id": "tBcRmg8UHD", "reviewer": "Reviewer_mzHq", "summary": "This paper presents experiments on how value horizon and policy horizon affect the ability of offline RL algorithms to meaningfully improve from additional data. They show that, depending on the task, large value horizons and/or large policy horizons essentially prevent learning, even when given access to very large data sets. Finally, a simple method that combines existing techniques for reducing these horizons is shown to dramatically improve the ability of offline RL to take advantage of additional data. The paper also promises to release a large dataset for further investigating such large-horizon effects.", "questions": "- How/why were the tasks chosen? Is it because they are easy to access, easy to generate data for, have an appropriate “challenge”? Some other reason?\n- How sensitive do you expect your results to be w.r.t. task choice, assuming tasks are still complex enough to have large horizons?\n- n-step SAC+BC seems best on maze. Do you suspect this is due to some quirk of how the hierarchies are implemented?\n- SHARSA seems to beat double SHARSA under some specific conditions, and not all methods increase performance with more data monotonically. Is this within expectations given the challenges of implementing SHARSA?", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n- The paper is well-written and easy to read.\n- Claims are justified and/or explained and the logical argument from hypothesis to experiment is clear.\n- The paper presents generalizable knowledge and is thus likely to have significant impact.\n- The experiments are simple, convincing, and explained and justified clearly.\n- The paper offers a potentially very useful dataset.\n\nWeaknesses:\n\n- There are no major weaknesses in my opinion, other than a few additional questions I have about task choice, which I understand may be hard to address given the space.", "was_revised": true, "final_justification": "I stand by my original statement. I think this is a very solid paper.", "rebuttal": "Thank you for the highly detailed review and positive feedback about this work! Please find our response below.\n\n* **How/why were the tasks chosen? Is it because they are easy to access, easy to generate data for, have an appropriate “challenge”? Some other reason?**\n\nOur goal was to perform a controlled scaling analysis of offline RL, and thus we sought a benchmark that provides highly challenging tasks with controllable data-generation scripts. OGBench was a great example for this, as it provides both diverse, challenging (unsolved) tasks and scripted expert policies. Among OGBench tasks, we chose the most difficult task in each category (`cube-octuple`, `puzzle-{4x5, 4x6}`, and `humanoidmaze-giant`) to cover various types of challenges in offline RL.\n\n* **How sensitive do you expect your results to be w.r.t. task choice, assuming tasks are still complex enough to have large horizons?**\n\nThanks for the question. We believe the horizon is a fundamental challenge in (offline) RL, as shown in our analysis (Section 5.1), and expect that the results generally apply to other long-horizon tasks too. In our experiments, we empirically support this claim on tasks across different domains (navigation, game-like puzzles, and manipulation).\n\n* **n-step SAC+BC seems best on maze. Do you suspect this is due to some quirk of how the hierarchies are implemented?**\n\nAs the reviewer points out, n-step SAC+BC generally led to the best performance on `humanoidmaze-giant`, being slightly better than SHARSA. We believe this (relatively minor) performance difference mainly stems from the difference in policy extraction. SAC+BC uses reparameterized gradients (as in DDPG), whereas SHARSA uses rejection sampling. A prior work [1] shows that reparameterized gradients generally lead to better performance than rejection sampling, as the former utilizes first-order gradient information. While we chose to use rejection sampling in SHARSA for simplicity (as our goal with SHARSA was to provide a simple, scalable recipe, rather than achieving the best possible performance), we believe it is possible to further improve the performance of SHARSA with more advanced low-level policy extraction methods like reparameterized gradients.\n\n* **SHARSA seems to beat double SHARSA under some specific conditions, and not all methods increase performance with more data monotonically. Is this within expectations given the challenges of implementing SHARSA?**\n\nWhile SHARSA generally leads to the best performance among the methods we evaluated, as the reviewer points out, it does not always monotonically improve performance with more data in our experiments. We believe this is mainly due to random variation in optimality in generated datasets. For example, a 100M-sized dataset might have a larger *ratio* of relatively more optimal transitions for a specific evaluation task than a 1B-sized one, which may potentially explain the reversed trend in `cube-octuple`. However, we believe an ideal algorithm should not lead to worse performance with more data (despite the change in state-action distributions), and we will acknowledge this point as a limitation in Section 7 in the final version of the paper.\n\n---\n\nWe would like to thank you again for asking important questions about our work. We believe the additional clarifications have strengthened the paper. Please let us know if you have any additional concerns or questions.\n\n[1] Park et al., Is Value Learning Really the Main Bottleneck in Offline RL?, NeurIPS 2024."}, {"review_id": "ISVDRmNRL3", "reviewer": "Reviewer_qjCd", "summary": "This paper investigates an important issue in offline reinforcement learning (RL): the failure of current algorithms to scale effectively to long-horizon, complex tasks, even when provided with massive datasets (up to 1 billion transitions). The authors' central contribution is to identify and empirically validate the \"curse of horizon\" as a primary problem. They decompose this curse into two key issues: (1) the accumulation of value-estimation errors in temporal difference (TD) learning, and (2) the explosion in policy complexity for tasks with distant goals. As a solution, the paper systematically demonstrates that methods incorporating \"horizon reduction\" can effectively mitigate these issues and unlock scalability.", "questions": "See weakness.", "limitations": "Yes. They are upfront about the dependence on data quality (Section 7), the idealized nature of the state-based environments, and the fact that even their proposed method does not achieve perfect performance.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\nThe empirical work in this paper is a major strength. The use of datasets up to 1 billion samples is rare in previous research and provides evidence for their claims about performance saturation. The design of the \"combination-lock\" toy problem is an excellent example of a controlled experiment that provides clear evidence for the \"bias accumulation\" hypothesis in value learning. The choice of difficult, long-horizon tasks from OGBench demonstrates the practical relevance of the problem.\n\n### Weaknesses\n\nThe primary weakness of this paper lies in its limited technical novelty. The proposed \"Horizon Reduction\" technique for value learning is essentially the well-established n-step target method. Similarly, the approach for policy learning introduces a standard hierarchical framework. Both are familiar techniques that have been widely used in the field.\n\nThe term \"Horizon Reduction\" is potentially misleading. In the context of Markov Decision Processes (MDPs), the \"horizon\" typically refers to the length of a trajectory (H). The use of n-step returns for the value function target does not actually shorten this trajectory horizon but rather adjusts the Bellman backup mechanism. A more precise term would be \"n-step value bootstrapping\" or similar.\n\nMoreover, the empirical results suggest that the performance gains stem predominantly from the application of n-step targets. As seen in Figure 9, a simpler SAC+BC baseline already performs quite well, which questions the severity of the \"curse of horizon\" in the policy learning. This raises doubts about the necessity of introducing a hierarchical policy, a modification that adds significant complexity and may hinder the method's scalability.\n\n### Overall Assessment\nIn summary, while the technical contributions are incremental, the paper's large-scale experiments are valuable and offer insights for future research in offline reinforcement learning. For this reason, my overall assessment is positive.", "was_revised": true, "final_justification": "The authors have addressed my concern about the limited effectiveness of reducing the policy horizon and have promised to clarify the potentially ambiguous term “horizon reduction.” I am therefore raising my score to 5.", "rebuttal": "Thank you for the highly detailed review and constructive feedback about this work. Please find our response below.\n\n* **Limited technical novelty**\n\nAs the reviewer points out, the main goal of this work is not to propose an entirely novel technique (L306-308). Instead, we aim to provide *new insights* based on our systematic scaling analysis. Through our scaling study, we identified the horizon as a fundamental bottleneck in offline RL scaling, and proposed a simple, scalable recipe that leads to better asymptotic performance.\n\nTo the best of our knowledge, this paper is the first to perform a 1 billion-scale data-scaling analysis in offline RL (with orders of magnitude larger datasets than the closest prior analysis work [1]), showing that horizon reduction leads to better *scalability*. We would like to note that most previous works that study horizon reduction (n-step returns, hierarchical RL, etc.) have mainly focused on sample efficiency, performance, or exploration, rather than scalability (L96-102 in Section 2).\n\nWe believe our new, actionable perspective in offline RL scaling (with our proposed recipe, SHARSA) aligns well with the broader goals stated in the [NeurIPS 2025 Reviewer Guidelines](https://neurips.cc/Conferences/2025/ReviewerGuidelines):\n\n> [O]riginality does not necessarily require introducing an entirely new method. Rather, a work that provides novel insights by evaluating existing methods, or demonstrates improved efficiency, fairness, etc. is also equally valuable.\n\n* **The performance gains seem to mainly come from n-step targets**\n\nWe would like to note that this is true only on one task, `humanoidmaze-giant`. In other tasks, for example, in `cube-octuple` (Figure 9), we were not able to achieve non-trivial performance only with value horizon reduction (i.e., n-step targets), whereas policy horizon reduction (i.e., hierarchical policies) led to substantial performance improvement. In `puzzle-4x5` and `puzzle-4x6` (Figure 9), we show that policy horizon reduction further improves the performance of n-step targets, as evidenced by the gap between n-step SAC+BC (which only reduces the value horizon) and SHARSA (which reduces *both* the value and policy horizons). We will further emphasize this point in the final draft.\n\n* **\"The term horizon reduction is potentially misleading\"**\n\nThank you for the feedback! We used \"horizon reduction\" as a general umbrella term to encompass *both* n-step TD targets (i.e., value horizon reduction) and hierarchical policies (i.e., policy horizon reduction). While n-step bootstrapping does not reduce the true horizon, it reduces the effective number of Bellman recursion steps within an episode, which we believe can be broadly interpreted as (effective) horizon reduction.\n\nTo prevent potential confusion, we will further clarify the term \"(effective) horizon reduction\" in the final draft. We hope this further clarification addresses the reviewer's concern, but please do let us know if you have better alternatives/suggestions for terms that can capture both n-step bootstrapping and hierarchical policies.\n\n---\n\nWe would like to thank you again for raising important questions about this work, and we believe the additional clarifications have strengthened the paper. Please let us know if you have any additional concerns or questions.\n\n[1] Park et al., Is Value Learning Really the Main Bottleneck in Offline RL?, NeurIPS 2024."}, {"review_id": "4xMTopk96R", "reviewer": "Reviewer_dG9E", "summary": "This manuscript investigates the scalability of offline RL algorithms and demonstrates, through experiments on datasets with a scale of up to one billion samples, that many standard offline RL methods struggle to perform well on complex problems. The authors identify the \"horizon\" as a key limiting factor for scaling offline RL and provide empirical evidence to support this claim. To address this issue, they propose a simple yet effective method—SHARSA—that shortens the horizon and achieves both strong asymptotic performance and favorable scalability characteristics.", "questions": "My main concern lies in the apparent disconnect between the choice of goal-conditioned RL tasks and the analysis of the horizon curse in this manuscript. Specifically:\n\n-\tThe manuscript emphasizes that error accumulation in value learning is a key cause of the horizon curse. However, if I understand correctly, this phenomenon is not necessarily unique to goal-conditioned RL. Offline RL more generally suffers from extrapolation errors due to distributional shift (OOD issues), and these errors also accumulate during the Bellman backup process — a point that has been discussed in prior work such as [1].\n\n-\tIn Lines 269–270, the authors state that in goal-conditioned settings, the mapping between optimal actions and distant goals can be highly complex, due to its dependence on the topology of the entire state space. I wonder whether this issue also applies to non-goal-conditioned settings, and if so, what the underlying reasons are.\n\nIn summary, I am unclear whether the horizon curse discussed in this manuscript is a general challenge in offline RL, or one that is specific to goal-conditioned RL tasks. If it is the latter, I suggest the authors revise the framing of the work accordingly, including the title. If it is the former, then I would strongly encourage the authors to provide additional discussion and experiments on standard offline RL benchmarks such as Mujoco, beyond goal-conditioned tasks alone.\n\n[1] Reining generalization in offline reinforcement learning via representation distinction. NeurIPS, 2023.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n\n-\tThe manuscript is well-motivated and addresses a novel and important topic.\n\n-\tThe writing is clear, well-structured, and easy to follow.\n\n-\tThe experimental validation is thorough and detailed, adding to the credibility of the findings.\n\n# Weaknesses\n\n-\tThe experimental setup is motivated by the statement: \"In particular, we are interested in the capabilities of offline RL algorithms to solve challenging tasks that require complex, long-horizon sequential decision-making given enough data.\" However, it remains unclear whether the curse of horizon is a universally limiting factor in offline RL. See Question 1 for more details.", "was_revised": true, "final_justification": "Most of my concerns have been well addressed, and I have decided to raise my score to a 5.", "rebuttal": "Thank you for the highly detailed review and constructive feedback about this work. We especially appreciate your question about non-goal-conditioned settings. We have conducted additional experiments on $4$ standard (reward-based) offline RL tasks to show that our claim applies to general offline RL as well. Please find our detailed response below.\n\n* **\"I am unclear whether the horizon curse discussed in this manuscript is a general challenge in offline RL, or one that is specific to goal-conditioned RL tasks\"**\n\nThanks for raising this point. In the paper, we mainly focus on offline goal-conditioned RL, as this problem setting naturally provides diverse challenges in offline RL (multi-task, long-horizon reasoning, stitching, etc.). However, we believe our claim (\"horizon reduction makes offline RL scalable\") is not limited to offline goal-conditioned RL, and generally applies to (reward-based) offline RL settings too. For example, our analysis in Section 5.1 is done in standard reward-based offline RL.\n\nTo further support this point, we additionally conducted experiments with $4$ standard **reward-based** offline RL tasks with `singletask` environments (`task1`, to be specific) provided by OGBench. These tasks are reward-based as in D4RL, but are more challenging than standard D4RL tasks (we would like to note that D4RL Gym MuJoCo tasks are generally already saturated [1, 2]). On these tasks, we evaluated SARSA (i.e., IQL with expectile $\\kappa = 0.5$), IQL (with $\\kappa = 0.7$), SAC+BC, n-step SAC+BC (value-only horizon reduction), and SHARSA (value and policy horizon reduction). Additionally, we consider an IQL variant of SHARSA (SHARSA with $\\kappa = 0.7$, a variant introduced in Appendix C), which can be particularly helpful as these `singletask` tasks have (much) higher suboptimality due to the absence of hindsight relabeling. \n\n|                               | $\\texttt{SARSA}$   | $\\texttt{IQL}$     | $\\texttt{SAC+BC}$ | $\\texttt{n-SAC+BC}$  | $\\texttt{SHARSA}$ $(\\kappa=0.5)$   | $\\texttt{SHARSA}$ $(\\kappa=0.7)$   |\n|:--------------------------------------------------|:-------------------|:-------------------|:--------------------|:----------------------------|:-----------------------------------|:-----------------------------------|\n| $\\texttt{Horizon Reduction Type}$ | - | - | - | $\\texttt{Value}$ | $\\texttt{Value \\\\& Policy}$ | $\\texttt{Value \\\\& Policy}$ |\n| | | | | | |\n| $\\texttt{cube-quadruple-play-singletask}$         | $0 {\\tiny \\pm 0}$  | $7 {\\tiny \\pm 6}$  | $0 {\\tiny \\pm 0}$   | $0 {\\tiny \\pm 0}$           | $50 {\\tiny \\pm 9}$                 | $\\mathbf{60} {\\tiny \\pm 8}$        |\n| $\\texttt{puzzle-4x5-play-singletask}$             | $4 {\\tiny \\pm 3}$  | $24 {\\tiny \\pm 7}$ | $0 {\\tiny \\pm 0}$   | $0 {\\tiny \\pm 0}$           | $\\mathbf{94} {\\tiny \\pm 3}$        | $\\mathbf{94} {\\tiny \\pm 2}$        |\n| $\\texttt{puzzle-4x6-play-singletask}$             | $2 {\\tiny \\pm 2}$  | $5 {\\tiny \\pm 2}$  | $0 {\\tiny \\pm 0}$   | $0 {\\tiny \\pm 0}$           | $\\mathbf{14} {\\tiny \\pm 6}$        | $\\mathbf{14} {\\tiny \\pm 7}$        |\n| $\\texttt{humanoidmaze-giant-navigate-singletask}$ | $0 {\\tiny \\pm 0}$  | $2 {\\tiny \\pm 2}$  | $44 {\\tiny \\pm 35}$ | $\\mathbf{88} {\\tiny \\pm 3}$ | $26 {\\tiny \\pm 4}$                 | $\\mathbf{87} {\\tiny \\pm 3}$        |\n\nThe table above shows the results on four tasks (at 1M steps, with 4 seeds and standard deviations), with 100M- (`cube`) or 1B- (others) sized datasets. The results suggest that both value and policy horizon reduction techniques can significantly improve the performance of reward-based standard offline RL as well. Specifically, they show that value horizon reduction is particularly helpful (and is enough) in `humanoidmaze`, but both value and policy horizon reductions are important in the other three tasks (`cube` and `puzzle`).\n\nFollowing the reviewer's suggestion, we will revise the manuscript and discuss relevant works to show that our claim applies to general long-horizon offline RL tasks (not limited to goal-conditioned settings) as well.\n\n* **Would policy horizon reduction help in non-goal-conditioned settings as well?**\n\nThe results above (especially `cube` and `puzzle`) show that having hierarchical policies often substantially helps improve performance in non-goal-conditioned tasks as well. The reason is similar: the mapping between states and optimal actions can be complex in long-horizon tasks, and hierarchical policies can help reduce this complexity by decomposing this potentially complex mapping into smaller pieces.\n\n* **\"The manuscript emphasizes that error accumulation in value learning is a key cause of the horizon curse. However, if I understand correctly, this phenomenon is not necessarily unique to goal-conditioned RL. Offline RL more generally suffers from extrapolation errors due to distributional shift (OOD issues), and these errors also accumulate during the Bellman backup process — a point that has been discussed in prior work such as [1].\"**\n\nIndeed, and as we discussed above, error accumulation in TD learning is not unique to goal-conditioned RL (and we didn't intend to imply this!). We will further clarify this point in the final version of the paper to prevent potential misunderstanding.\n\n* **Formatting concerns**\n\nWe will fix/revise them. Thanks!\n\n---\n\nWe would like to thank you again for raising important questions about the applicability of our claim to non-goal-conditioned settings. We believe the additional results and clarifications have strengthened the paper. Please let us know if you have any additional concerns or questions.\n\n[1] Tarasov et al., Revisiting the Minimalist Approach to Offline Reinforcement Learning, NeurIPS 2023. \\\n[2] Rafailov et al., D5RL: Diverse Datasets for Data-Driven Deep Reinforcement Learning, RLC 2024."}]

Author rebuttal:

Meta-review: This is a well-prepared paper that studies why offline RL fails to scale on long-horizon tasks, identifies the horizon as the primary bottleneck, and shows that reducing the effective value and policy horizons improves scaling. The proposed method is minimalistic and practical, and the empirical evidence is very thorough, using datasets up to one billion transitions on challenging OGBench tasks; the paper is clearly written and easy to follow. 

Reviews are overall positive. During discussion, reviewers raised questions about scope beyond goal-conditioned settings and about the novelty of the techniques. The authors ran additional experiments on standard, reward-based offline RL and clarified terminology and design choices, after which all reviewers converged to very positive recommendations. The AC weighs the careful analysis, scale of the evidence, and the clean takeaways for practice more heavily than concerns about incremental technical novelty. 

This is a nice paper that presents interesting and solid results. The paper offers clear insight about scaling offline RL and backs it with convincing experiments and a simple method likely to influence future work.
Decision: Accept (spotlight)