Title: Horizon Reduction Makes RL Scalable
Venue: NeurIPS 2025
Accepted: yes
Keywords: reinforcement learning

offline reinforcement learning
Abstract: In this work, we study the scalability of offline reinforcement learning (RL) algorithms. In principle, a truly scalable offline RL algorithm should be able to solve any given problem, regardless of its complexity, given sufficient data, compute, and model capacity. We investigate if and how current offline RL algorithms match up to this promise on diverse, challenging, previously unsolved tasks, using datasets up to 1000× larger than typical offline RL datasets. We observe that despite scaling up data, many existing offline RL algorithms exhibit poor scaling behavior, saturating well below the maximum performance. We hypothesize that the horizon is the main cause behind the poor scaling of offline RL. We empirically verify this hypothesis through several analysis experiments, showing that long horizons indeed present a fundamental barrier to scaling up offline RL. We then show that various horizon reduction techniques substantially enhance scalability on challenging tasks. Based on our insights, we also introduce a minimal yet scalable method named SHARSA that effectively reduces the horizon. SHARSA achieves the best asymptotic performance and scaling behavior among our evaluation methods, showing that explicitly reducing the horizon unlocks the scalability of offline RL.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript investigates the scalability of offline RL algorithms and demonstrates, through experiments on datasets with a scale of up to one billion samples, that many standard offline RL methods struggle to perform well on complex problems. The authors identify the \"horizon\" as a key limiting factor for scaling offline RL and provide empirical evidence to support this claim. To address this issue, they propose a simple yet effective method—SHARSA—that shortens the horizon and achieves both strong asymptotic performance and favorable scalability characteristics.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\n-\tThe manuscript is well-motivated and addresses a novel and important topic.\n\n-\tThe writing is clear, well-structured, and easy to follow.\n\n-\tThe experimental validation is thorough and detailed, adding to the credibility of the findings.\n\n# Weaknesses\n\n-\tThe experimental setup is motivated by the statement: \"In particular, we are interested in the capabilities of offline RL algorithms to solve challenging tasks that require complex, long-horizon sequential decision-making given enough data.\" However, it remains unclear whether the curse of horizon is a universally limiting factor in offline RL. See Question 1 for more details.\n\nQUESTIONS:\nMy main concern lies in the apparent disconnect between the choice of goal-conditioned RL tasks and the analysis of the horizon curse in this manuscript. Specifically:\n\n-\tThe manuscript emphasizes that error accumulation in value learning is a key cause of the horizon curse. However, if I understand correctly, this phenomenon is not necessarily unique to goal-conditioned RL. Offline RL more generally suffers from extrapolation errors due to distributional shift (OOD issues), and these errors also accumulate during the Bellman backup process — a point that has been discussed in prior work such as [1].\n\n-\tIn Lines 269–270, the authors state that in goal-conditioned settings, the mapping between optimal actions and distant goals can be highly complex, due to its dependence on the topology of the entire state space. I wonder whether this issue also applies to non-goal-conditioned settings, and if so, what the underlying reasons are.\n\nIn summary, I am unclear whether the horizon curse discussed in this manuscript is a general challenge in offline RL, or one that is specific to goal-conditioned RL tasks. If it is the latter, I suggest the authors revise the framing of the work accordingly, including the title. If it is the former, then I would strongly encourage the authors to provide additional discussion and experiments on standard offline RL benchmarks such as Mujoco, beyond goal-conditioned tasks alone.\n\n[1] Reining generalization in offline reinforcement learning via representation distinction. NeurIPS, 2023.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates an important issue in offline reinforcement learning (RL): the failure of current algorithms to scale effectively to long-horizon, complex tasks, even when provided with massive datasets (up to 1 billion transitions). The authors' central contribution is to identify and empirically validate the \"curse of horizon\" as a primary problem. They decompose this curse into two key issues: (1) the accumulation of value-estimation errors in temporal difference (TD) learning, and (2) the explosion in policy complexity for tasks with distant goals. As a solution, the paper systematically demonstrates that methods incorporating \"horizon reduction\" can effectively mitigate these issues and unlock scalability.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\nThe empirical work in this paper is a major strength. The use of datasets up to 1 billion samples is rare in previous research and provides evidence for their claims about performance saturation. The design of the \"combination-lock\" toy problem is an excellent example of a controlled experiment that provides clear evidence for the \"bias accumulation\" hypothesis in value learning. The choice of difficult, long-horizon tasks from OGBench demonstrates the practical relevance of the problem.\n\n### Weaknesses\n\nThe primary weakness of this paper lies in its limited technical novelty. The proposed \"Horizon Reduction\" technique for value learning is essentially the well-established n-step target method. Similarly, the approach for policy learning introduces a standard hierarchical framework. Both are familiar techniques that have been widely used in the field.\n\nThe term \"Horizon Reduction\" is potentially misleading. In the context of Markov Decision Processes (MDPs), the \"horizon\" typically refers to the length of a trajectory (H). The use of n-step returns for the value function target does not actually shorten this trajectory horizon but rather adjusts the Bellman backup mechanism. A more precise term would be \"n-step value bootstrapping\" or similar.\n\nMoreover, the empirical results suggest that the performance gains stem predominantly from the application of n-step targets. As seen in Figure 9, a simpler SAC+BC baseline already performs quite well, which questions the severity of the \"curse of horizon\" in the policy learning. This raises doubts about the necessity of introducing a hierarchical policy, a modification that adds significant complexity and may hinder the method's scalability.\n\n### Overall Assessment\nIn summary, while the technical contributions are incremental, the paper's large-scale experiments are valuable and offer insights for future research in offline reinforcement learning. For this reason, my overall assessment is positive.\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nYes. They are upfront about the dependence on data quality (Section 7), the idealized nature of the state-based environments, and the fact that even their proposed method does not achieve perfect performance.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents experiments on how value horizon and policy horizon affect the ability of offline RL algorithms to meaningfully improve from additional data. They show that, depending on the task, large value horizons and/or large policy horizons essentially prevent learning, even when given access to very large data sets. Finally, a simple method that combines existing techniques for reducing these horizons is shown to dramatically improve the ability of offline RL to take advantage of additional data. The paper also promises to release a large dataset for further investigating such large-horizon effects.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The paper is well-written and easy to read.\n- Claims are justified and/or explained and the logical argument from hypothesis to experiment is clear.\n- The paper presents generalizable knowledge and is thus likely to have significant impact.\n- The experiments are simple, convincing, and explained and justified clearly.\n- The paper offers a potentially very useful dataset.\n\nWeaknesses:\n\n- There are no major weaknesses in my opinion, other than a few additional questions I have about task choice, which I understand may be hard to address given the space.\n\nQUESTIONS:\n- How/why were the tasks chosen? Is it because they are easy to access, easy to generate data for, have an appropriate “challenge”? Some other reason?\n- How sensitive do you expect your results to be w.r.t. task choice, assuming tasks are still complex enough to have large horizons?\n- n-step SAC+BC seems best on maze. Do you suspect this is due to some quirk of how the hierarchies are implemented?\n- SHARSA seems to beat double SHARSA under some specific conditions, and not all methods increase performance with more data monotonically. Is this within expectations given the challenges of implementing SHARSA?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the scalability of standard MLP-based offline RL algorithms. Ideally, provided with a huge expert dataset that widely covers the state and action spaces, a good offline RL algorithm should be able to achieve perfect performance regardless of task difficulty. However, by increasing the size of the dataset from 1M to 1B, the author shows that current offline RL algorithms fail to achieve non-trivial performance on four challenging long-horizon OGBench tasks. Then, by demonstrating with a toy environment, Combination-Lock, the author hypothesizes that (1) the bias accumulation in TD value learning over lone horizon, and (2) the inability of direct mapping between optimal actions and distant goals in policy learning, orthogonally hinders the scalability of offline RL algorithms.  Therefore, the author discusses several horizon reduction methods that can mitigate the bias accumulation in value learning, e.g., n-step returns, or decompose the long-horizon policy extraction into a hierarchy. Moreover, the author proposes (Double) SHARSA that combines horizon reduction in both value and policy learning in a minimalist way. Experiments show that horizon reduction can effectively improve the offline RL performance as the data size scales up.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths:\n- The paper is coherent, well-organized, and easy to follow. The figures are clean, informative, and contribute effectively to the overall presentation.\n- The research question is clearly defined and appropriately scoped, with no overclaims. The hypotheses are supported by enough empirical results. A thorough ablation studies are provided in the main body and in the appendix, which addresses many potential concerns regarding the paper’s arguments. And the experimental details are well-documented in the appendix for reproduction.\n- The research question should be of interest to the RL community, as RL scalability remains underexplored compared to fields like NLP and CV. The paper touches on several types of offline RL algorithms and evaluates them across four meaningful control tasks. The insight on horizon reduction is actionable, and the proposed methods and experimental protocol have the potential to inspire future work in improving offline RL scalability.\n---\n### Weaknesses:\nI do not identify major weaknesses, and a few comments are listed below:\n- Regarding the novelty, while the curse of horizon has been discussed in the RL literature, and the proposed method (SHARSA) builds upon existing techniques, the paper's central contribution—the connection between the curse of horizon and offline RL scalability—is both insightful and actionable. The paper has the potential to serve as a benchmark and reference point for future research in this area.\n- In Line 276-277, the author borrows an analogy to Chain-of-Thought reasoning. Although I understand what it conveys, it appears a bit irrelevant and implies little to the hierarchical policy learning discussed in this section.\n- In Line 321-331, the author describes the SHARSA value learning process but does not mention how horizon reduction is done here (although details are included in the appendix). This part would be more self-contained if the author briefly mentions that n-step return is used for high-level SHARSA value learning, while 1-step return is used for low-level (double) SHARSA learning.\n\nQUESTIONS:\n1. In Line 233-235, the author describes two types of datasets used for the Combination-Lock environment, evaluates 1-step DQN and 64-step DQN on both datasets, and reports the best performance for each method. How do you perform the 64-step DQN on the dataset of uniform coverage of length-1 trajectory segments?\n2. This work mainly focuses on the *MLP-based* offline RL methods. Could the author comment on the scalability of *Transformer-based* offline RL or imitation learning methods, e.g., Behavior Transformer (BeT), Vector-Quantized Behavior Transformer (VQ-BeT), or Q-Transformer? Does transformer architecture bring any advantage of scalability to those methods naturally? Will the curse of horizon hamper their scalability to the same extent as it hinders the MLP-based methods?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}