Title: Robust and Scalable Autonomous Reinforcement Learning in Irreversible Environments
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reinforcement Learning

Autonomous Reinforcement Learning
Abstract: Reinforcement learning (RL) typically assumes repetitive resets to provide an agent with diverse and unbiased experiences. These resets require significant human intervention and result in poor training efficiency in real-world settings. Autonomous RL (ARL) addresses this challenge by jointly training forward and reset policies. While recent ARL algorithms have shown promise in reducing human intervention, they assume narrow support over the distributions of initial or goal states and rely on task-specific knowledge to identify irreversible states. In this paper, we propose a robust and scalable ARL algorithm, called RSA, that enables an agent to handle diverse initial and goal states and to avoid irreversible states without task-specific knowledge. RSA generates a curriculum by identifying informative states based on the learning progress of an agent. We hypothesize that informative states are neither overly difficult nor trivially easy for the agent being trained. To detect and avoid irreversible states without task-specific knowledge, RSA encodes the behaviors exhibited in those states rather than the states themselves. Experimental results demonstrate that RSA outperforms existing ARL algorithms with fewer manual resets in both reversible and irreversible environments.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses a key challenge in real-world reinforcement learning: the dependence on frequent manual resets of the environment after episode termination. This reset requirement is both labor-intensive and impractical in physical systems such as robotics.\nThe authors propose RSA (Robust and Scalable Autonomous RL), a method that trains both forward policies (to solve the task) and reset policies (to bring the environment back to favorable conditions) without task-specific knowledge and under diverse initial/goal states.\nThe paper is evaluated on continuous-control benchmarks including robotic manipulation and locomotion tasks, comparing RSA to state-of-the-art ARL methods such as SPOT and SkewFit.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\nI really like the idea of encoding how a state is reached instead of just looking at the state itself to judge irreversibility. This is quite clever—it acknowledges that the same final state can be reversible or not depending on the trajectory taken to get there, which is often true in manipulation tasks.\n\n**Weaknesses**\n\n**1. I’d like to see more theoretical insight**\nWhile the empirical work is strong, I feel the paper would benefit from some **formal analysis**. For example:\n\n* Is the learning progress metric provably helpful under certain assumptions?\n* How does the behavior encoding affect reset success in theory?\n  I’m not asking for complete proofs, but even a toy example or intuitive analysis would help deepen the reader’s understanding of *why* RSA works.\n\n**2. I’m unsure how well this generalizes under noise**\nOne concern I have is how RSA performs in **stochastic or noisy environments**. If behavior encodings vary a lot between rollouts due to randomness, would the system still detect irreversible states reliably? I didn’t see this discussed in the paper. Including an experiment or a brief analysis on this point would strengthen the case for generality.\n\n**3. Training complexity feels high but isn’t discussed**\nRSA has multiple components (two policies, behavior encoders, progress estimators), and I wonder how **computationally heavy** the training is. This is especially relevant if someone wants to apply RSA to a real robot system. I would have appreciated a brief discussion of training time, or resource requirements.\n\nQUESTIONS:\nI think this is a **very thoughtful and practically valuable paper**. The motivation is crystal clear, and the solution is elegant and well-grounded. There’s a nice balance of algorithmic design and empirical demonstration.\n\nThat said, I would feel more confident in recommending a higher score if:\n\n* There was a discussion of robustness to noise, and\n* Some insight into the computational cost or training complexity was provided.\n\nIf these points are addressed, I would strongly lean toward an **accept** recommendation.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe manuscript proposes a robust and scalable autonomous reinforcement learning (ARL) algorithm, RSA, which addresses the limitations of existing ARL algorithms in handling diverse initial and goal states and avoiding irreversible states without relying on task-specific knowledge. RSA generates a curriculum by identifying informative states based on the agent’s learning progress and encodes behaviors to detect and avoid irreversible states. Experimental results show that RSA outperforms existing ARL algorithms in both reversible and irreversible environments with fewer manual resets.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n1. The manuscript addresses a significant and practical problem in reinforcement learning, focusing on reducing manual resets and improving applicability in real-world settings.\n    \n2. RSA relaxes assumptions on initial and goal state distributions and does not require task-specific knowledge to avoid irreversible states, enhancing its generalizability and practical value.\n    \n3. The related work section provides a comprehensive comparison of ARL algorithms, and Table 1 clearly illustrates the advantages and coverage of the proposed approach.\n    \n4. The experimental results demonstrate RSA’s effectiveness and robustness in both reversible and irreversible environments, indicating its potential for real-world deployment.\n    \n5. The authors have provided the code, and the code is well-documented, facilitating reproducibility and independent verification of the experimental results.\n    \n\n### Weaknesses\n\n1. The experimental comparison is limited, as it only includes two ARL baselines (and their multi-goal variants), omitting other relevant algorithms such as MEDAL and PAINT listed in Table 1. This limits the strength of the empirical validation.\n    \n2. The technical novelty of RSA appears incremental, and the method depends on three manually set threshold hyperparameters. The manuscript does not provide an analysis of the sensitivity or robustness of these parameters.\n    \n3. The experimental environments are relatively simple and mainly focus on navigation tasks, which may not sufficiently demonstrate the scalability and versatility of RSA in more complex or diverse domains.\n\nQUESTIONS:\n1. Why were other methods listed in Table 1, such as MEDAL and PAINT, not included in the experimental comparison? Are there technical or practical reasons for their exclusion?\n    \n2. In Section 5.2, why are experimental results on handReach and HandManipulate environments not reported for the comparsion?\n    \n3. In line 184, the authors expect that the successor feature values of irreversible state-action pairs to be much lower than those for reversible pairs. Is there theoretical justification or empirical evidence supporting this claim?\n    \n4. The explanations for experimental results are sometimes insufficient. For example, in line 227, why do narrow initial or goal state distributions lead to low success rates for R3L and LNT? In line 230, how is it observed that agents collect more biased experiences in AntOpen and AntOpen-Trap?\n    \n5. In line 243, it is mentioned that RSA and LNT require a comparable number of manual resets in irreversible environments. How does this demonstrate the advantage of RSA? Is LNT’s performance due to its use of task-specific knowledge, and how does this affect the fairness of the comparison?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes RSA, an ARL framework designed to reduce manual resets in both reversible and irreversible environments without relying on task-specific knowledge. RSA jointly trains a forward policy to solve tasks,  and a reset policy (to recover from failures. To avoid irreversible states, RSA encodes agent behaviors using successor features rather than relying on state-specific labels, identifying states where agents lose control over goal-relevant features. Through experiments on diverse navigation and manipulation tasks, RSA demonstrates better performance and reset efficiency compared to prior ARL methods, showing its robustness and scalability for real-world applications.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n\n1. RSA avoids reliance on handcrafted rewards, reversibility labels, or demonstrations, making it more generalizable to real-world settings.\n\n2. Using successor features to detect irreversible states via behavioral patterns is a novel approach.\n\n3. RSA demonstrated strong performance and fewer manual resets across both reversible and irreversible tasks in a variety of domains.\n\nWeakness \n\n1. The reset policy uses RND, which may be inefficient in complex or high-dimensional environments.\n\n2.  RSA can only identify irreversible states after sufficient behavioral data is collected for identification, potentially causing early-stage failures and costs.\n\nQUESTIONS:\n1. How sensitive is RSA to the choice of feature representation $\\phi$ used in successor features? Could poor feature choices lead to inaccurate identification of irreversible behaviors?\n\n2. In large or visually rich environments, does using Random Network Distillation (RND) provide sufficient coverage for the reset policy? Would structured exploration (e.g., skill discovery) help?\n\n3. Since RSA relies on accumulating behavioral evidence before identifying irreversible states, how can it prevent the agent from entering catastrophic states in the early stage of training?\n\n4. Can RSA handle high-dimensional observations like images or point clouds? How would its components (e.g., state estimator, SFs) scale to such settings?\n\n5. How accurate is the self-supervised estimator $I(s,g)$ during early training? Does misestimation bias curriculum generation?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an autonomous RL algorithm called RSA based on two components: 1) a curriculum created by setting goal states for forward and backward policies based on which states would be informative goal/initial states. This is done with a combination of heuristics and by training a classifier that estimates the ability of the current agent to reach goals. 2) A mechanism to detect and avoid irreversible states using successor features. They show that this algorithm needs fewer resets and learns faster on various navigation and manipulation environments compared to previous methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The algorithm presented is effective at learning in an autonomous fashion in both reversible and irreversible environments.\n- The mechanism to find irreversible states seems like it works well.\n- The idea of switching from the forward to the reset policy once the initial state is difficult enough is interesting. \n\nWeaknesses:\n- The method does introduce 3 new hyperparameters that need to be searched over. It’s also unclear how the search was done, and if the baselines were searched over in a commensurate fashion.\n- The goal setting heuristic (when the policy fails at achieving the forward goal) seems not as well motivated. It’s unclear why using the last state of the failed rollout is a good forward goal. A useful ablation to have would be if you simply chose a goal from the rest of your goal set.\n- Some parts of the detecting irreversible states method are unclear (see below).\n\nQUESTIONS:\n- What does RSA stand for? Robust and Scalable Autonomous RL? If so, can you further explain the scalable part of the name?\n- Why is using the final state of failed iterations as the goal state for the next iteration a good heuristic? In HER it makes sense since the agent is able to learn without having to rollout more trajectories, but in this case, it’s not clear why those failed intermediate states would be good future goal states. Can you run an ablation to see whether this is necessary? And if so provide a clearer understanding of why?\n- Is there any weighting of more recent trajectories for training the state information estimator given that sampling old trajectories could lead to out of date labels?\n- I am not exactly clear on how the mapping function $m$ used in computing the successor features comes from. Section 3.1 implies that it comes from the environment and is not learned? \n- Could you create a visualization of which states the agent believes are irreversible and how that evolves over time across a couple environments? It would be interesting to see how well that aligns with our notion of irreversible.\n- Is get_informative_goal in general just sampling from the goal states provided by the environment (except in the cases of failed forward trajectories)?\n- Can you provide the ablation results for the rest of the environments as well?\n- For Figure 7, right, it would be helpful to also see the success rate plotted against the timesteps in the environment as well.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-07"}