Title: DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: reinforcement learning

test-time training

test-time reinforcement learning

sparse-reward reinforcement learning

goal selection

goal-conditioned reinforcement learning

exploration

exploration-exploitation

upper confidence bound
Abstract: Sparse-reward reinforcement learning (RL) can model a wide range of highly complex tasks.
Solving sparse-reward tasks is RL's core premise — requiring efficient exploration coupled with long-horizon credit assignment — and overcoming these challenges is key for building self-improving agents with superhuman ability.
We argue that solving complex and high-dimensional tasks requires solving simpler tasks that are *relevant* to the target task.
In contrast, most prior work designs strategies for selecting exploratory tasks with the objective of solving *any* task, making exploration of challenging high-dimensional, long-horizon tasks intractable.
We find that the sense of direction, necessary for effective exploration, can be extracted from existing reinforcement learning algorithms, without needing any prior information.
Based on this finding, we propose a method for _**di**rected **s**parse-reward goal-**co**nditioned **ve**ry long-horizon **R**L_ (DISCOVER), which selects exploratory goals in the direction of the target task.
We connect DISCOVER to principled exploration in bandits, formally bounding the time until the target task becomes achievable in terms of the agent's initial distance to the target, but independent of the volume of the space of all tasks.
Empirically, we perform a thorough evaluation in high-dimensional simulated environments. We find that the directed goal selection of DISCOVER solves exploration problems that are beyond the reach of prior state-of-the-art exploration methods in RL.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces DISCOVER, a modification of goal-conditioned reinforcement learning for sparse goal-directed tasks that utilizes a set of intermediate goals to accelerate exploration. It combines estimates of achievability and novelty with estimates of the distance between the current and target goal to the goal selection process. The goal selection is then biased to select intermediate goals that are reliably achievable, novel, and are estimated to be close to the target goal. The paper offers a theoretical bound of the number of episodes needed to achieve the target task and shows the advantage of the proposed method in multiple experiments over three different environments.\n\nSTRENGTHS AND WEAKNESSES:\n### Quality\nThe method is well-motivated and validated in the experiments by comparing to three other baselines for goal selection, as well as to three baselines without goal-conditioning. The method is not compared to other methods that could use the goal information from hierarchical RL or reward shaping, but this comparison might be out-of-scope for this work. \n\nA theoretical upper bound on the number of episodes learn to to reach the goal is given (informal intuition in the main text, formalized proofs in the appendix).\n\n### Clarity\nOverall, the paper is well-written. The motivation and description of the method are well-structured and comprehensible. The experiment setup lacks some details on the environments. Most notably the exact construction of the *pointmazes* environment (used dimensionalities and number of paths to goal) as well as the details on the *simple* and *hard* configurations for all environments including the used goals are missing. The result section is well structured and easy to follow.\n\n### Significance\nThe method transfers general goal-conditioned RL to the sparse goal-directed setting to use intermediate goals to accelerate learning of a target task. The method is shown to improve over the state-of-the-art for goal selection strategies.\n\n### Originality\nTo the best of my knowledge, the proposed method is novel and relevant citations are given where needed.\n\nQUESTIONS:\n- What are the goals used in the experiments?\n\n- How is the relevance term used before reaching the target goal for the first time?\n\n- Why is the weighting term $\\beta_t$ of the variance the same for both novelty and relevance? Also, is it set to $\\beta_t=1$ in the experiments (as described in line 171)?\n\n- If the theoretical bound is independent of the set of goals, how is it advantageous over standard RL?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents DISCOVER, a goal-directed exploration method for RL, which performs exploration using a combined metric that defines a set of goals to attempt so as to focus exploration towards the true target goal state.\n\nSTRENGTHS AND WEAKNESSES:\nOverall, I think this paper is excellent. It is very well written (easily the best read in my batch of papers to review for NeurIPS this cycle), strongly motivated by an important topic in RL, and shows strong performance from an intuitive and theoretically grounded method for high impact. While similar in some ways to prior goal directed exploration methods, the key \"relevance\" term proposed is to my knowledge novel, and the paper does a good job of grounding the other elements of the algorithm in prior work.\n\nThe main weakness I can see is that I'm still a bit confused as to why this method works so well- Given that the true goal is never sampled during early training for the evaluated tasks, I'm a little confused as to how the relevance term is providing useful information for goal selection. Some of this might be down to imperfect explanation, but I can imagine experiments that would shine light on when and for what states or goals this term provides a meaningful signal and when/where it is basically noise (perhaps comparing the relevance term computed with the current learned value function to a known oracle/pretrained value function for different states/goals?).\n\nIn aggregate, though, I think this paper is a quality submission and more than sufficient for publication as is. If my concern about the basis for outperformance in the experimental results can be addressed and added to the paper, I would raise my score even further as well.\n\nQUESTIONS:\n1. The discussion of the value of \"sense of direction\" in the abstract and figure 1 is confusing, since this isn't standard exploration terminology and thus without a definition on hand leaves the reader confused as to what is meant. Consider writing those passages.\n\n2. Does the use of the value estimate for the achievability component of the DISCOVER objective result in favoring goals reachable in fewer timesteps? If not, what prevents this bias?\n\n3, In section 4.2, how do these value estimates affect exploration? Maximizing value is simple exploitation.\n\n4. In theorem 4.2 and associated sections, I'm confused as to why the bound is linear with the distance to the target. The value function has no knowledge of the target, so why would it bias selected goals towards the target? The target has never been observed during training, by construction, no? I must be missing something about how there is useful information about an unknown target in the value function. Is this relying on generalization in function approximation? I'm unsure why that would appear in the theoretical bound, though.\n\n5. In figure 7, it seems odd to compare primarily to TD3 and SAC, since these are not really exploration methods. Ideally I'd suggest comparing to random network distillation here, as a commonly used baseline for a state novelty based exploration bonus method.\n\nLIMITATIONS:\nLimitations are suitably discussed. I don't see any direct potential negative societal impacts from this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper introduces DISCOVER, a novel goal-selection strategy for sparse-reward reinforcement learning that creates an automated curriculum of exploratory tasks. The method balances the achievability, novelty, and relevance of potential goals, using bootstrapped value function estimates to direct exploration towards a final target task without requiring prior information. The authors demonstrate that this approach significantly outperforms state-of-the-art exploration methods on 3 complex, long-horizon control tasks.

STRENGTHS AND WEAKNESSES:
Strengths

- The problem of generating a curriculum of simpler, relevant tasks is crucial for solving complex, high-dimensional problems, and this paper makes a direct and effective contribution to this area.
- Bootstrapping a "sense of direction" entirely from the agent's own value estimates is a clever and practical approach. This avoids the need for strong prior knowledge or costly auxiliary components like generative models, which are often required by other directed exploration strategies.

Weakness

- Although the method is presented as general, the empirical evaluation is confined to navigation-centric environments (Point Maze, Ant Maze, Arm)  where a geometric "sense of direction" is intuitive. This casts some doubt on the method's generality, as the vital "relevance" component may not function as effectively in more abstract or combinatorial tasks that lack a simple spatial correlation between intermediate and final goals.

QUESTIONS:
If the bootstrapped estimate of relevance is not optimal, could it trap the agent into refining its policy within a suboptimal point, thereby preventing the discovery of a better path to the goal?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a method called \"DISCOVER\" that selects exploratory goals in the direction of task. Using the critic networks in deep learning, the method can propose achievable goals that can help an agent with a better goal selection to complete the task. By learning an ensemble of critics, the method can also estimate uncertainty regarding a task and leverages value estimates to find useful intermediate goals. The method shows considers three sparse-reward long-horizon settings (Point Maze, Ant Maze and Arm) and shows improvements over the baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The method addresses an important problem of exploration over long horizons in RL and considers goal based RL settings.\n- Using a combination of metrics (achievability, novelty and relevance) sounds a reasonable and interesting metric to consider goal selection.\n\nWeaknesses:\n- The method relies on an ensemble of critics which can be computationally expensive and slow to learn in challenging environments.\n- The tasks used for evaluation are simpler tasks, as compared to more difficult to explore settings in which goal-based methods can actually be helpful in achieving good exploration.\n- The baselines considered do not consider the usual exploration based methods, especially those that are based on intrinsic rewards or RND rewards in sparse-reward settings. A comparison with those can help one understand if the goal-based methodology is even required given when these exploration based methods fail.\n- The success rate of some of the tasks is still low, eg in Easy setting of Arm, and Hard setting of PointMaze and Arm. It would be interesting to see if the curves changes as more number of timesteps are added.\n\nQUESTIONS:\n- Can you please explain why the success rate is still low for almost half of the environment settings? Also, how exactly is the success rate calculated here?\n- Have you considered comparing your method with the intrinsic reward based exploration methods that can do well over long-horizon settings?\n- How sensitive are the uncertainty estimates based on the size of the ensembles used?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-07"}