Title: Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reinforcement learning

Offline RL

Goal-conditioned RL

Hierarchical RL

Temporal Abstraction
Abstract: Offline goal-conditioned reinforcement learning (GCRL) offers a practical learning paradigm in which goal-reaching policies are trained from abundant state–action trajectory datasets without additional environment interaction. However, offline GCRL still struggles with long-horizon tasks, even with recent advances that employ hierarchical policy structures, such as HIQL. Identifying the root cause of this challenge, we observe the following insight. Firstly, performance bottlenecks mainly stem from the high-level policy’s inability to generate appropriate subgoals. Secondly, when learning the high-level policy in the long-horizon regime, the sign of the advantage estimate frequently becomes incorrect. Thus, we argue that improving the value function to produce a clear advantage estimate for learning the high-level policy is essential. In this paper, we propose a simple yet effective solution: _**Option-aware Temporally Abstracted**_ value learning, dubbed **OTA**, which incorporates temporal abstraction into the temporal-difference learning process. By modifying the value update to be _option-aware_, our approach contracts the effective horizon length, enabling better advantage estimates even in long-horizon regimes. We experimentally show that the high-level policy learned using the OTA value function achieves strong performance on complex tasks from OGBench, a recently proposed offline GCRL benchmark, including maze navigation and visual robotic manipulation environments. Our code is available at https://github.com/ota-v/ota-v


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the performance bottleneck of the prior method HIQL and concludes that inaccurate value estimation causes the high-level policy to suggest suboptimal subgoals. Further analysis reveals that this inaccuracy arises from the large temporal distance between the current state and the goal. To address this, the authors propose a novel, simple, and effective method called OTA, which estimates the value function over a sequence of actions, effectively transforming distant goals into nearer ones. Experiments show that this technique improves order consistency, meaning that states can be ranked by the value function in the same order as their temporal sequence, and outperforms the existing methods.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n\n1. Section 4 is compelling and well-argued, offering a strong and intuitive motivation for the proposed method.\n\n2. The authors introduce a new concept, \"order consistency\", which is both interesting and potentially impactful for future research in goal-conditioned reinforcement learning.\n\n3. The experimental results are thorough and convincingly demonstrate the effectiveness of the proposed approach.\n\n**Weakness**\n\nThe paper does not provide sufficient theoretical justification for the OTA value function $V^h_{\\text{OTA}}$ as optimised in Eq. (4). Please try to address the following questions:\n\n(1) Can the value function be interpreted as the expected sum of rewards? If not, how to define this value function?\n\n(2) Does this formulation satisfy the Bellman recursion? It is the ground for optimisation following Eq. (4). \n\nI would like to increase my rating to Accept once the authors address the two questions.\n\nQUESTIONS:\n1. In the abstract, the authors state that offline GCRL trains goal-reaching policies from unlabeled (reward-free) datasets. However, the datasets used in offline GCRL include binary rewards / reward function, that indicates whether the goal has been reached. Therefore, describing them as \"reward-free\" may be inaccurate and should be clarified.\n\n2. The term \"policy extraction\" is used multiple times (e.g., line 29 and line 149) without a prior explanation. It would improve clarity if the authors could briefly define what they mean by policy extraction in this context.\n\n3. The reward function defined in lines 121 and 207 as $r(s, g) = -1(s \\neq g)$ assumes that exact equality $s = g$ is achievable. This assumption is problematic in continuous state spaces, where reaching the exact goal is often impossible. A more realistic formulation would be $r(s, g) = -1(\\|s - g\\|^2 > \\epsilon)$, for some threshold $\\epsilon > 0$.\n\n4. In Figure 2, it appears that when the state $s$ is far from the goal $g$, the high-level value function $V^h$ tends to yield over-optimistic estimates. This behavior is noteworthy, and it would strengthen the paper if the authors could provide an explanation or hypothesis for why this phenomenon occurs.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new offline goal-conditioned RL (GCRL) algorithm called OTA. They begin by noting that the main bottleneck of HIQL (one of the state-of-the-art offline GCRL methods) is in the high-level policy. To address this, they propose training a separate high-level value function with temporally extended options (which are defined by subgoals), and extracting a high-level policy based on this high-level value function. They show that the resulting algorithm (OTA) substantially improves performance on many long-horizon navigation tasks in OGBench.\n\nSTRENGTHS AND WEAKNESSES:\n[Strengths]\n* The experiments in the motivation section (Section 4) are well designed. The authors convincingly argue that high-level policy learning is the bottleneck of HIQL, and OTA can resolve this issue effectively (Figure 5).\n* The proposed method is simple and effective.\n* The authors provide several analysis experiments (Sections 6.3-6.6), which are informative and helpful to understand the effect of the proposed method.\n* OTA achieves substantially better performance than the previous methods on several challenging tasks (e.g., `humanoidmaze-giant` and `antmaze-*-explore`).\n* The paper reads well in general.\n\n[Weaknesses]\n\nI don't see significant weaknesses of this work. Though I do have some concerns, mostly about empirical evaluation.\n* From Table 4, OTA seems to require careful tuning of four hyperparameters ($\\beta^h$, $\\beta^\\ell$, $k$, and $n$) for each setting. This potentially makes the empirical comparison unfair, given that the baselines use the same policy extraction hyperparameters for each dataset type (see Table 11 of the OGBench paper).\n* The authors only selectively report performance on manipulation tasks. For example, how does OTA perform on the standard `play` datasets?\n* OTA implicitly assumes that length-n trajectory chunks in the dataset are optimal (which is implicitly used in Eq. (4)). This is not always the case, especially when the dataset is highly suboptimal. The paper lacks a discussion about this point.\n\n[Minor issues]\n* L54: \"by updating the value over a sequence of primitive actions\" -- This sounds like the method does action chunking, which (as far as I understand) seems not to be the case.\n\nQUESTIONS:\nI don't have additional questions other than the ones I asked in the weaknesses section.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates failure cases of HIQL (hierarchical implicit Q-learning) and proposes an improvement. The paper first provides a preliminary experiment that analyses failure cases of HIQL and concludes that these are probably caused by the high-level policy setting 'bad' goals, which in turn seems to come from imprecise V-value estimation. This problem is analyzed as being a problem mainly in long horizons. The proposed solution is to take a target value that instead of being based on 'regular' one-step bootstrapping, is based on effectively a n-step bootstrapping target. Experiments show that on various environments from OGBench and robot manipulation tasks, the proposed method outperforms non-hierarchical baselines as well as the original HIQL.\n\nSTRENGTHS AND WEAKNESSES:\nQuality - Strenghts\n- The paper provides a clear and well-reasoned analysis of failure cases of HIQL\n- The empirical methodology over multiple environments is thorough and convincing. Results include error bounds over 8 random seeds. \n- Beyond raw performance, a visualization is provided to confirm the intuition of why this method might work better, and an ablation is performed over different values of the 'n' parameter. \n\nQuality - Weaknesses\n- Conceptually and theoretically, the new method could have been analyzed in more detail. In more detail: \n-- Limitations of the method are not mentioned. Some of these are inherited from IQL (sensitive to stochastic dynamics, which bias the expectile approximation of the 'max' operator over actions; recursive optimality), while some of them are increased by the method (the new target depends on (more) future states and actions, increasing the mentioned bias; the definition and analysis of \"order inconsistency\" seems incompatible with stochastic dynamics). To slightly expand on this point: the notion of \"optimal trajectory' is not well defined, as in general an optimal policy in a stochastic environment can induce a distribution of trajectories. Due to non-anticipated ('noisy') transitions, \"order consistency\" should not be expected. \n-- The paper strongly links the proposed approach to options, but the link to the original options in [46] is rather weak: in the original options paper, termination conditions are a Markov function (here: non-Markov), various options provide a high-level choice (here: only a single option), updates consider the aggretate discount gamma^n (here: n). In fact, I think the more useful analogy is to that of n-step returns.\n\nClarity - strengths\nThe paper is very clearly written in understandable language. Figures provide a clear illustration of the discussed ideas and contexts. \n\nClarity - weaknesses\nAs a rather minor weakness, I think the structure could be improved, as the introduction spends a lot of time detailing the proposed approach (better saved for the technical section) and is rather concise on the problem context, problem motivation, and solution need.\n \nSignificance\nThe empirical results are quite convincing, especially on the 'giant' versions of the mazes. On robot manipulation there is only a small (statistically likely non-significant) difference to HIQL, making it unclear how significant the result is beyond maps. \n\nOriginality\nThere is originality in the analysis of the problem, but the proposed solution has very limited originality, as this is essentially doing something very close to the well-known n-step returns. \n\nMinor comments: \nline 186: I'd rephrase this sentence. \"noisy\" is a bit ambiguous. I would typically interpret this as having a high variance, but the variance of the target doesn't depend on the horizon. Do you mean a bigger discrepancy between the target and the true value ('value error', if you will?).\nLine 194: Options are clasically not considered the same as macro-actions, as macro-actions are open-loop and options can be closed loop. Macro actions would also typically be fixed-lenths while options classically have a Markovian termination condition.\n\nQUESTIONS:\nWhy are sign mismatches so crucial? To me it seems that going from 'just positive' to 'just negative' will change importance weight from slightly above 1 to slightly below one - might not be a big deal. Large errors that don't change the sign can make a bigger difference and more aversely affect the learned value function.\n\nLIMITATIONS:\nI think several limitations need to be more explicit: \n- bias caused by going from 1-step to multi-step returns where the expectile function will now encourage a optimistic estimate of the whole n-steps\n- assumption of deterministic dynamics.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 1, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the challenges of offline goal-conditional reinforcement learning (GCRL), pointing out that the failure of the hierarchical policy of HIQL is mainly due to the difficulty of generating suitable subgoals in the high-level policy, and the sign of the advantage signal frequently becomes incorrect. Thus, the paper proposes the Option aware Temporarily Abstracted (OTA) value learning method, which integrates temporal abstraction into temporal differential learning and improves the accuracy of advantage estimation. Experiments have shown that OTA performs well in complex tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\\\nThis article first analyzes whether it is the bottleneck of the high-level policy or the low-level policy, then analyzes the reasons for the bottleneck of the high-level policy, and finally proposes methods. The whole process is gradual and easy to understand. The proposed method maintains simplicity. The experimental effect has been greatly improved.\\\nWeaknesses:\\\nIntroduced a new temporal abstraction step hyperparameter n, increasing the complexity of parameter tuning. The method improves the consistency of the value function, but it cannot guarantee complete orderliness and has certain limitations.\n\nQUESTIONS:\n1. Adjusting the temporal abstraction step of the value function in the high-level policy has improved the value estimation of long-horizon states, but does this have an impact on the value estimation of short-horizon states?\n2. Would it be better to use separate value functions with different step sizes for the high and low level policy?\n3. The temporal abstract step n and subgoal step k need to be adjusted in coordination. Is there any method that can more automatically determine the optimal combination of n and k instead of parameter tuning?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}