Robust and Scalable Autonomous Reinforcement Learning in Irreversible Environments
Authors: Sang-Hyun Lee
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement Learning
Autonomous Reinforcement Learning
Primary area: reinforcement_learning

Abstract: Reinforcement learning (RL) typically assumes repetitive resets to provide an agent with diverse and unbiased experiences. These resets require significant human intervention and result in poor training efficiency in real-world settings. Autonomous RL (ARL) addresses this challenge by jointly training forward and reset policies. While recent ARL algorithms have shown promise in reducing human intervention, they assume narrow support over the distributions of initial or goal states and rely on task-specific knowledge to identify irreversible states. In this paper, we propose a robust and scalable ARL algorithm, called RSA, that enables an agent to handle diverse initial and goal states and to avoid irreversible states without task-specific knowledge. RSA generates a curriculum by identifying informative states based on the learning progress of an agent. We hypothesize that informative states are neither overly difficult nor trivially easy for the agent being trained. To detect and avoid irreversible states without task-specific knowledge, RSA encodes the behaviors exhibited in those states rather than the states themselves. Experimental results demonstrate that RSA outperforms existing ARL algorithms with fewer manual resets in both reversible and irreversible environments.

Robust and Scalable Autonomous Reinforcement Learning in Irreversible Environments
Sang-Hyun Lee
Department of Automotive Engineering Ajou University Gyeonggi-do, South Korea sanghyunlee@ajon. ac. kr
Abstract
Reinforcement learning (RL) typically assumes repetitive resets to provide an agent with diverse and unbiased experiences. These resets require significant human in- tervention and result in poor training efficiency in worldworld settings. Autonomous RL (ARL) addresses this challenge by jointly training forward and reset policies. While recent ARL algorithms have shown promise in reducing human intervention, they assume narrow support the distributions of initial or goal states and rely on task-specific knowledge to identify irreversible states. In this paper, we propose robust and scalable ARL algorithm, called RSA, that enables an agent to handle diverse initial and goal states and to avoid irreversible states without task-specific knowledge. RSA generates curriculum by identifying informative states based on the learning progress of an agent. We hypothesize that informative states are neither overly difficult nor trivially easy for the agent being trained. To detect and avoid irreversible states without task-specific knowledge, RSA encodes the behaviors exhibited in those states rather than the states themselves. Experimental results demonstrate that RSA outperforms existing ARL algorithms with fewer manual resets in both reversible and irreversible environments.
Introduction
Reinforcement learning (RL) has demonstrated remarkable achievements in the field of roboties [6 14 8 18]. However, most of these achievements rely on repeated resets between episodes to provide an agent with multiple attempts and unbiased experiences. While such resets are easily performed in simulated settings, they require substantial human intervention and lead to poor training efficiency in the real world 228 12 Autonomous RL (ARL), which simultaneously learns how to solve task and how to reset an environment, has gained significant attention as promising alternative to reducing human intervention. The key idea behind ARL to generate a curriculum that determines when abort episodes and where to return agent. For example, the curriculum encourages an agent to explore space near initial or goal states in the early stages of training and gradually move to more distant space training progresses.
Recent ARL works have shown that their curricula enable an agent to learn diverse tasks with fewer manual resets 99, 31 24 26 20). However, these works require narrow support over the initial or goal state distributions to generate their curricula. These restricted distributions can lead to less robust performance. Furthermore, their curricula either do not consider irreversible states or rely on task-specific knowledge to identify them. Irreversible states are those from which an agent cannot recover without external intervention, such as when vehicle is damaged in accident manipulator pushes an object outside its workspace. Most real-world tasks involve such irreversible states, and the task-specific knowledge required to identify them is rarely available in practice.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

In this paper, we introduce RSA, robust and scalable ARL algorithm that enables an agent to handle diverse initial and goal states and detect and avoid irreversible states without task- sisk-specifii knowledge. RSA generates curriculum by identifying informative initial and goal states based on the agent's 's learning progress. We hypothesize that informative states are those that are neither overly difficult nor trivially easy for the agent under training. To detect and avoid irreversible states, RSA encodes the behaviors exhibited in such states rather than the states themselves. This is based on our observation that, while irreversible states may vary across tasks, the behaviors tend to follow common patterns.
The main contribution of this work is robust and scalable ARL algorithm called RSA. We evaluate RSA against baselines on diverse navigation and manipulation tasks. Experimental results demon- strate that RSA generates a curriculum by identifying informative states based on the agent's 's learning progress, achieving better performance with fewer manual resets than the baselines in both reversible and irreversible environments.
2 Related Work
RL agents trained in simulation often struggle to perform well in the real world due to the fidelity gap between simulated and real-world environments. A straightforward approach to avoid this gap is to train agents directly in the real world. Chebotar et al. [6] combine model-based and model-free updates to learn manipulation tasks on a PR2 robot. Kendall et al. 12] demonstrate the first RI agent capable of driving real-world vehicle along country roads. However, these works rely on manual resets after every episode, which require substantial human intervention. While several previous works employ additional instrumentation or scripted reset behaviors automate environment resets,
their reset strategies are task-specific and limited to particular scenarios [1] 30 18
29 22].
ARL, which aims to learn without manual resets, has been actively studied in recent years 9 27 31 23 25
24
26, 13 20]. Eysenbach et al. [9] train forward and reset policies simultancously
and generate curriculum by early aborting forward episodes based on the reset value function. While their curriculum encourages agents to explore space they have already learned, Patil et al. [20] generate curriculum that guides an agent explore space they have not yet sufficiently learned. Both works assume an unimodal initial state distribution with narrow support, which can cause sub-optimal and non-robust performance. To address this limitation, Zhu et al. [1 introduce random perturbation controller that discovers novel initial states, gradually broadening the support of the training distribution. Sharma al. [24] encourage the agent return states from expert demonstrations, based on the hypothesis that such demonstrations provide a desired distribution over initial states. Although these works focus on ensuring diverse initial states, their curricula do not account for multiple goal settings, which makes their algorithms less scalable. To address this limitation, our work generates a curriculum that provides an agent with diverse initial and goal states by identifying informative states based on the agent's 's learning progress.
While most previous works assume that environments are reversible, our work aims to reduce manual resets in both reversible and irreversible environments. Eysenbach et al. [9] and Xie et al. (26] are most closely related to ours, they focus reducing manual resets in irreversible environments. Eysenbach et al. [9] leverage the state-action value function, which is trained with reset reward functions, metric to determine whether an agent is in irreversible states. Xie et al. (26) propose a label-efficient binary search algorithm that detects irreversible states using a limited number of reversibility labels. Experimental results from both works demonstrate that their algorithms can identify irreversible states and prevent agents from entering them. However, in real-world scenarios, access to task-specific knowledge, such as reset reward functions or reversibility labels, often unavailable. Our work enables an agent to identify and avoid irreversible states without task-specific knowledge. Table summariizs the key features that distinguish our work from previous works.
3 Preliminaries
3.1 Goal-Conditioned Reinforcement I Learning
We consider a goal-conditioned reinforcement learning (GCRL) task represented by the tuple (S, A, T,, gy Po; 7, P. m). is the set of states 8, A is the set of actions a, G is the set of goal states 9, and T S x A x S
+ [o, 1] is the state transition model. rg : S x A x S + R is the
2

---

Table I: Comparison of Autonomous Reinforcement Learning (ARL) algorithms.
Property
LNT 9
R33 31
MEDAL 24
PAINT 26
RSA (Ours)
Diverse Initial States
Diverse Goal States
X X
No Extrinsic Reset Rewards
No. Demonstrations
X
No Reversability Labcls
/
Ireversible Environments
X
X
goal-conditioned reward function, Po is the initial state distribution, and y is the discount factor. Pg the goal state distribution, and m : -> G is a mapping function from states to corresponding goal states. The goal space may either be subset of the state space or the state space itself. In the latter case, the mapping function the identity function. An agent's s behaviors are determined by a goal-conditioned policy and value function. The coal-conditioned policy Tia 8. s.g) maps the current state and the current goal to probability distribution over actions and the goal-conditioned value function Q* (s, a. g) represents the expected return when agent takes action in state and goal and follows the policy и. The goal of GCRL is to find the optimal policy that maximizes the expected return when the state transition model is unknown. Please refer to [1 21 for further details.
3.2 Successor Features
Assume the reward function is linear combination of a feature vector Φ(s, a, s') IR and a weight
vector w E Rd such that the reward is given by r(s, a, s') = Φ(s, a, s') w. Following 12 17 [0], goal-conditioned reward function can be similarly decomposed as rry (8, a, 8') (88 a, 8') wg. as the weight vector, also referred to task vector, encodes preferences over individual feature components. With this formulation, the goal-conditioned value function can then be written as follows:
(s,a,,,)) E
W |S: =,
E 0i41 S: = s, = a]Twg (s, wg..
1=
Barreto et al. 22 call (s, a) the Successor Features (SFs), which describe the expected discounted sum of features o(s, a, under a policy æ. The SFs e)" (s, a) represent the expected discounted sum
of features encountered when following policy and can be regarded multi-dimensional value function, where a feature vector acts as a reward function. This implies that SFs can be trained with standard reinforcement learning algorithms.
Robust and Scalable Autonomous Reinforcement Learning
RSA is designed to enable agent to achieve robust performance with fewer manual resets in both reversible and irreversible environments. Similar to previous ARL algorithms ( B1 24 26), RSA defines forward policy (a s, and reset policy (a s, g), and alternates between them. The forward policy is trained to solve tasks and the reset policy is trained to reset the environment. Note that each policy takes the current goal state as additional input.
RSA considers two distinctive and challenging settings. First, RSA allows initial and goal states to be located anywhere in the state space S. This contrasts with previous works that assume unimodal and narrow supports for the initial or goal state distributions to generate their curricula. Second, RSA assumes neither reversible environments nor access to task-specific knowledge for identifying irreversible states. This presents difficulties for previous works that rely on such task-specific knowledge, including reversibility labels or reset reward functions, identify irreversible states.
To handle these settings, RSA 1) generates curriculum by identifying informative initial and goal states based on the learning progress of an agent, and 2) identifies irreversible states by encoding the behaviors exhibited those states without task-specific knowledge. the remainder this section, describe how RSA generates the curriculum and identifies irreversible states in detail.

---

Informativv
Informative
Informative
Inibal Etate Sut
Sk+1
Sk
4. Inkaal Stato
Sk+1
Sk
4 Initial Blato
fa Inital
6113 intaal State
*
13
dx Stste
5k
Dx.
Goal Stata
Informative Intiaal
Informative Goal Statos
Achivved Goal States
Informative Intial States
Non-informativve Goal States
Forward and Reset Rollouts
Figure 1: Curriculum generated by RSA. Given the current goal state Jke the reset policy is activated until the agent reaches the informative initial state set St corresponding to that goal. Left: At the end of each iteration, RSA samples the next goal state from the informative goal state set. Right: If the agent fails to reach the current goal state Ik, RSA uses the final state $1 of the current iteration as the next goal state 9k+1-
4.1 Identifying Informative Initial and Goal States
Figure [ illustrates how RSA generates a curriculum by identifying informative initial and goal states. The curriculum built our hypothesis that informative states neither overly difficult nor trivially easy for the agent being trained. If the initial or goal states are too difficult, the agent may take unsafe actions and fail reach the goal. On the other hand, the agent may struggle to obtain useful information even with sufficient time if the initial or goal states are too easy. Both cases can lead to suboptimal performance and poor sample efficiency. To identify informative states based on the agent's learning progress, RSA introduces the state information estimator, [ss, ().)). which takes pair of initial and goal states as input and estimates the probability reaching the goal state from the initial state when the agent follows the forward policy being trained.
Here describe how RSA utilizes the state information estimator identify informative goal and initial states. RSA determines the informative goal state for the k th iteration, Ik. as follows:
Dhe UniS(G)), where 9 A {g Gi.k-1 A S E.-B, (= g)] S A2).
(I)
S. is the set of informative goal states for the kth iteration, Gisk- is the set of goal states obtained until the previous iterations, B, the replay buffer for the reset policy, and Å and X2 are the lower and upper thresholds of the reachability probability over the pairs of initial and goal states, respectively. This prevents an agent from being assigned goal states that are either too difficult or too easy, which allows it to obtain informative transitions. Specifically, a goal state is uniformly sampled from the set of informative goal states the beginning of each iteration. If the agent reaches goal state and it is still identified part of the informative goal state set, RSA reuses it as the goal state for the next iteration rather than sampling new one.
After the informative goal state Ou is determined, RSA attempts to discover the informative initial state ig by activating the reset policy until the agent reaches the of informative initial states
St C S as follows:
( & T(s gk) d2 for gh Uni
This prevents the agent from resetting with too challenging too easy initial states. To continuously discover novel initial states, RSA trains the reset policy with an off-the-shelf exploration algorithm that guides an agent to under-explored states. While we implement the exploration algorithm using random network distillation (RND) [5] in our experiments due to its scalability and case of implementation, RSA is compatible with any exploration algorithm 19
Even when our curriculum provides an agent with informative initial and goal states, the agent being trained may still fail to reach the goal states. Repeated failures can cause the agent to be stuck in uninformative states. To address this challenge, when the agent fails to reach the goal state, RSA assigns the final state of the current iteration as the goal state for the next iteration. This idea is inspired by hindsight experience replay (HER) [1, which enables an agent to learn from undesired outcomes. The key distinction between RSA and HER is that, whereas HER treats the final state as the goal state within the same iteration, RSA uses the final state of the current iteration as the goal state for the next iteration.

---

Negative Examples
Intial State
Intial State
Intial Starte
Iniial Stare
Initial Stake
Iriiial
Positive Examples
B.
Goal
Figure 2: Overview of the training procedure for the state information estimator. State-goal pairs from successful forward rollouts are treated positive examples, while those from failed forward rollouts are treated as negative examples.
The state information estimator, which estimates the probability of reaching goal states from initial states, is a key component that enables RSA to provide an agent with diverse initial and goal states. The main challenge in training the state information estimator is the absence of explicit supervisory signals to estimate reachability based on the agent's learning progress. To address this, RSA extracts reachability labels e, from forward rollouts and uses them to train the state information estimator in a self-supervised manner. Specifically, state-goal pairs from the rollouts in which the agent successfully reaches the goal state are treated as positive examples, while those from failed rollouts are treated as negative examples. The objective of the state information estimator can be written as follows:
min - -eer......) fee (st, 9x)) + (1 - et) log(1 I(st, 9:))],
where e, is 1 for positive examples and otherwise, and By is the replay buffer for the forward policy. TrigureBilluster how RSA extracts supervisory signals from forward rollouts to train the state information estimator.
4.2 Detecting and Avoiding Irreversible States
RSA enables agent identify irreveriible states without task-specific knowledge by encoding the behaviors exhibited in those states, rather than the states themselves. This is based on our observation that, while inrevesiblee states have task-specific features, the behaviors tend to follow shared pattern: an agent in irreversible states loses control over goal-relevant features, regardless of the actions takes. To identify this shared pattern, RSA leverages SFs to encode behaviors induced by the reset policy. The SFs are denoted by via (s, a) = ET, 1>>=, A > -
8, A, a). Note that the of
reset policy continues generate diverse behaviors and the feature vector defined as the difference in goal-relevant features, Φ4 [| 1) ma m((s))] where m is the mapping function discussed in Section 3 RSA is agnostic to the form of the mapping function, and learning it with state-of of- the-art representation learning is complementary to our work. However, we leave this for future work,
is orthogonal to our main contribution.
We expect the SFs for irreversible state-action pairs to be much lower than those for reversible pairs. The set of irreversible states, Sur S, can then be identified as follows:
S E S 1f (8, S X3, for a~ n, (als,g))
where A3 is the reversibility threshold. RSA leverages two techniques to enable an agent to detect and avoid the set of irreversible states S First, RSA employs a surrogate reward function that penalizes the agent when it enters the identified irreversible states. This is similar to Eysenbach et al. [9] and Xie et al. (26), in which the agent also receives penalties for visiting irreversible states. However, unlike both previous works, RSA does not depend on task-specific knowledge to identify such states. Second, RSA conservatively identifies the irreversible states, by setting the threshold parameter Ag higher than that for the actual set, and aborts forward episodes when the agent enters any state in the identified set. We empirically found that this conservative identification helps prevent the agent from entering irreversible states in experiments.
5

---

Algorithm Robust and Scalable Autonomous Reinforcement Learning
1: Initialize reset and forward policies π, (a(s. f
2: Initialize reset and forward replay buffers By: B3
3: Initialize state information estimator [s,, g) and successor features & (s,a)
4: for k < 1... do
5: Sample goal state St with Bec_intommatveegallBB.ii.i.e.ee
6: for t + Treset do
7: Select reset action 0₂ (a,(aa3s
8:
if A S (St, (t.) S X2 then
9:
Set 8, as initial state and switch to forward policy
10:
end if
11:
Obtain reset transition (st; dz, rt, 8t+1, 9k)
12:
Compute surrogate reset reward ft
13: Add transition to reset buffer B, f- B, U {(t,, Oz, Prr 11+1, 9k))
Update reset policy n., (als,, g) and successor features v)" (s, (s.0) end for
15:
for 1... do Sclect forward action a₂ (T
17:
18:
(st, a²) I A then
19:
Abort and switch to reset policy
20:
end if
21:
Obtain forward transition (8t, dt, rts 1.
22:
Compute surrogate forward reward ft
23:
Add transition to forward buffer B5 +- By U [ss, at, . .9:)]
24:
Update forward policy Ty (als) and state information estimator ((8
25:
end for
26:
end for
4.3 Algorithm Summary
Algorithm|] [ outlines the overall training procedure of RSA. At the beginning of each iteration, RSA utilizes the state information estimator identify a set of informative goal states from all goal states encountered in previous forward rollouts. It then samples one informative goal state from this set for the current episode. Once the goal state is determined, RSA activates the reset policy until the agent reaches an informative initial state, such that the predicted probability of reaching the goal state lies between Xr and λ2. Upon reaching such state, RSA designates the current state the initial state and switches to the forward policy. When the forward policy is activated, RSA aborts the episode early if the SFs for the reset policy are less than equal the reachability threshold Ag. Please refer to Appendix A for additional details.
Experiments
We design our experiments to investigate the following questions: (1) Can RSA achieve more robust performance with fewer manual resets than previous algorithms in both reversible and irreversible environments? (2) Can RSA generate a curriculum by identifying informative initial and goal states based on the learning progress of an agent? (3) How does each main component of RSA contribute to its performance improvement?
To answer these questions, we evaluate RSA against the following baselines: (1) LNT, (2) LNT-MG, (3) R3L, and (4) R3L-MG. LNT y
uses reset reward functions identify irreversible states and assumes narrow initial state distributions to generate its curriculum. R3L [31] generates a curriculum with diverse initial states but cannot handle irreversible states. Since neither algorithm considers multi-goal settings, we implement LNT-MG and R3L-MG as variants of LNT and R3L that sample goals randomly and periodically from their buffers. Note that RSA generates a curriculum that provides agent with diverse initial and goal states and does not use reset reward functions to detect irreversible states. While all baselines perform manual resets either when the agent fails to return to starting states within a fixed number of episodes enters irreversible states, RSA triggers manual

---

E
PointiMaee
PointMaaz-Trap
AntOpen
AntOper-TTap
HandReach
HandMannppllte
Figure 3: Navigation and manipulation tasks used in our experiments. PointIMaze and AntOpen reversible, while PointIMaze-Trra and AntOpen- include irreversible states, indicated by purple boxes. HandReach is reversible, whereas HandManipulate involves irreversible states, such as dropping the object from the hand.
AntOpen
AmDDee-Trap
Su0000 Rdb
Timoetepp
#f Tiresttpp
Wot Timesrppe
Timeedopp
of Roceb
1550
1000
200
530%
1.00
of Timosteps
of Tirosppp
of Timootoss
LNT
UNTAIG
ROL
R3:-MG
RSA(Dun))
Figure 4: Learning curves for navigation tasks. The x-axis indicates the number of training steps, while the y-axis shows the success rate (top row) and the number of manual resets (bottom row). Darker-colored lines represent the means, and shaded regions denote the standard deviations across random seeds.
resets only when the agent enters irreversible states. Appendix B provides additional details our experimental setup and implementation.
5.1 Environments
Figure shows the navigation and manipulation tasks used our experiments. All tasks are provided by Hommaaiuu-RRboteee [7]. The common objective across these tasks is to move an agent, such as mobile robot, the fingertips of the hand, or an object, to target locations as quickly as possible. PointIMaze, AntOpen, and HandReach do not involve irreversible states, whereas PointI Trap, AntOpen-Trap. and MandMannplllte contain such states. example, when an agent in navigation task enters a trap shown as purple box, it struggles to move and cannot escape without external intervention. This is analogous In the manipulation tasks, once object is dropped, an agent For a fair comparison, we evaluate both the baselines and our algorithm using the same set episodes with diverse initial and goal states. Appendix C provides additional details on the environments.
vehicle damaged in an accident that is unable to move on its own.
no longer pick it up control it.
5.2 Experimental Results and Analysis
Figure 4 illustrates the learning curves computed over 5 random seeds for both reversible and ir- reversible navigation tasks. R3L and LNT achieve low success rates to their narrow initial or goal state distributions. While R3L-MG and LNT-MG obtain robust performance in PointIMaze and PointlMaze-Trap both suffer performance drops in AntOpen and AntOpen- Trap. We empirically ob- served that in AntOpen and AntOpen-Trae Trap, agents tend to collect more biased experiences compared

---

I
T
T
Figure Changes in informative initial state sets over training. Each column shows how the informative initial states change training progresses from left right. The color of each initial state indicates its predicted reachability probability for a fixed goal state. Top: The goal state is located at (0.8, 0.8). Bottom: The goal state is located at (2.8, 1.8).
T
Figure Changes in informative goal state sets over training. Each column shows how the informative goal states for a fixed initial state change as training progresses from left right. The initial state is fixed at (4.8, 2.8), and the color of each goal state indicates its predicted reachability probability from that initial state.
to PointIMaze and PointIMaze- Trap. This suggests that randomly sampling initial and goal states from the buffer fails to provide an agent with informative transitions when struggles to explore diverse states. RSA achieves the best performance across all tasks and converges faster and more stably than the baselines.
LNT and LNT-MG require significantly fewer manual resets than R3L and R3L-MG. This result expected, as LNT and LNT-MG assume narrow initial state distributions and leverage privileged information about irreversible states in the form of reset reward functions. However, constrained initial states make it difficult for an agent to collect diverse transitions, leading to suboptimal performance. Moreover, privileged information about irreversible states is rarely available in practice. RSA does trigger manual resets in reversible navigation tasks, PointIMaze and AntOpen. This suggests that RSA can generate curriculum that enables an agent to collect informative transitions even when it fails to return to initial states or reach to goal states. In the irreversible tasks, PointIMaze- Trap and AntOpen-Tree Trap, RSA requires fewer manual resets than RSL. and RSL-MG, and triggers comparable number resets to LNT and LNT-MG. These results indicate that, unlike LNT and LNT-MG, RSA can identify inreversible states without task-specific knowledge.
To investigate whether RSA identifies informative state sets based on the agent's learning progress, we visualize how the informative initial and goal state sets evolve during training on PointIMaze. Note that, in our work, informative initial and goal states are those whose estimated reachability probabilities fall between the hyperparameters X1 and Xz. Figure illustrates the changes in the informative initial state set for two different goal states: (0.8, 0.8) and (2.8, 1.8). the early stages of training, the identified initial state sets are close to the respective goal states and gradually move farther away as training progresses. F shows the changes in the informative goal state set for an initial state (4.8, 2.8). We observed that similar to the informative initial state sets shown in Figure

---

5 the informative goal states are identified near the given initial state at the beginning of training and gradually spread out in diverse directions over time. These results demonstrate that RSA identifies informative initial and goal states based on the agent's 's learning progress. They also suggest that the curriculum generated by RSA enables an agent to efficiently bootstrap from success on easier initial and goal states to tackle more challenging ones. Appendix C provides additional results on the identification of irreversible states.
Finally, we conducted an ablation study to examine the benefits of two key com- ponents of RSA: (1) the state information estimator for identifying informative states, and (2) the successor features (SFs) for detecting and avoiding irreversible states. The left plot of Figure the perfor- mance of RSA and its variant, RSA w/o SIE, which does not identify informative states, on HandReach. Note that in this reversible task, neither RSA nor RSA w/o SIE triggers manual resets, as both algo- rithms reuse the final state of the current iteration as the next goal state instead of re- setting the environment. The performance gap between RSA and RSA w/o SIE in- dicates that identifying informative initial and goal states contributes to better asymp- totic performance and sample efficiency. The right plot of Figure Sesureeillume iillstratts the performance of RSA and another variant, RSA w/o SFs, which does not identify irreversible states, on HandManipulatee. Although both RSA and RSA w/o SFs can identify informative states using the state information estimator, we observed that RSA achieves better performance with the same number of manual resets. This result confirms that SFs of RSA play an important role in reducing manual resets.
HandReach
HandManipulete
Sucoeee
Aow MaddWol
0.
02
Figure 7: Ablation results of RSA components. The darker lines and shaded areas represent the mean and standard deviation over 5 random seeds. Left: Effect of identifying informative states on performance in Han- dReach. Right: Effect of identifying irreversible states on performance in HandManipulate.
6 Conclusion
We introduce a robust and scalable autonomous reinforcement learning (ARL) algorithm, referred to as RSA, that reduces manual resets in both reversible and irreversible environments. RSA identifies informative states to generate curriculum that provides an agent with diverse initial and goal states. Furthermore, RSA enables an agent to detect and avoid irreversible states without task-specific knowledge. Experimental results demonstrate that RSA allows an agent to identify informative states based on the agent's ' learning progress and to avoid irreversible states, achieving better performance with fewer manual resets than existing ARL algorithms. RSA has several limitations that should be addressed in future work. First, RSA uses randomized reset behaviors, which are incfficient for discovering initial states in complex environments. We plan to investigate whether leveraging structured and consistent exploration behaviors can mitigate this limitation. Second, RSA uses predefined features to encode behaviors via successor features. We expect that combining RSA with state-of of-the--rrt representation learning algorithms will be a promising approach to alleviating this limitation. Finally, RSA cannot prevent an agent from entering irreversible states before the associated behaviors have been encoded. We believe that the common-sense reasoning abilities of foundation models will make them promising for identifying irreversible states across diverse tasks, even in the early stages of training.
Acknowledgements
This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) under the Artificial Intelligence Convergence Innovation Human Resources De- velopment 20--2202--S-22---222 Ministry of Trade Industry and Energy. Korea Planning & Evaluation Institute of Industrial Technol- (gE(KEIT) under Grant RSS-2024-00448797 and Convergence and Open Sharing System Project granted by the Ministry of Education and National Research Foundation of Korea.
grant funded by the Korea goVer)menn(MSIT), Korea

---

References
[1] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
[2] André Barreto, Will Dabney, Rémi Munos, Jonathan Hunt, Tom Schaul, Hado van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. Advances in neural information processing systems, 30, 2017.
[3] Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
[4] Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A. Efros. Large-scale study of suriosity-drive learning. In ICLR, 2019.
[5] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. International Conference on Learning Representations, pages 4003- 4019, 2019.
[6] Yevgen Chebotar, Karol Hausman, Marvin Zhang, Gaurav Sukhatme, Stefan Schaal, and Sergey Levine. Combining model-based and model-free updates for trajectory-centric reinforcement learning. In International conference on machine learning, pages 703-711. PMLR. 2017.
[7] Rodrigo de Lazcano, Kallinteris Andreas, Jun Jet Tai, Seungjae Ryan Lee, and Jordan Terry. Gymnasium robotics, 2024. URL http:: htce:p//gttauo.ccoaaaaaaaa-rooop Foundatiio Gymnasium
Himmasiu--Roooiccs Robotics
[8] Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International conference on machine learning,
pages 1329-1338. PMLR, 2016.
[9] Benjamin Eysenbach, Shixiang Gu, Julian Ibarz, and Sergey Levine. Leave no trace: Learning to reset for safe and autonomous reinforcement learning. In International Conference on Learning Representations, pages 1799-1816, 2018.
[10] Jiaocheng Hu, Yuexin Ma, Haiyun Jiang, Shaofeng He, Gelu Liu, Qizhen Weng, and Xiangwei Zhu. A new representation of universal successor features for enhancing the generalization of target-driven visual navigation. IEEE Robotics and Automation Letters, 2024.
[11] Leslie Pack Kaelbling. Learning to achieve goals. In IJCAI, volume 2, pages 1094-1098.
Citeseer, 1993.
[12] Alex Kendall, Jeffrey Hawke, David Janz, Przemyslaw Mazur, Daniele Reda, John-Mark Allen, Vinh-Dieu Lam, Alex Bewley, and Amar Shah. Learning to drive in a day. In 2019 international conference robotics and automation (ICRA), pages 8248 8254. IEEE, 2019.
[13] Sang-Hyun Lee and Seung- Woo Sco. Self-supervised curriculum generation for autonomous reinforcement learning without task-specific knowledge. IEEE Robotics and Automation Letters, 2024.
[14] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17(39):1-40) 2016.
[15] Sergey Levine, Peter Pastor, Alex Krizhevsky, Julian Ibarz, and Deirdre Quillen. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. The International journal of robotics research, 37(4-5):421 -3,,, 2018
[16] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver. and Daan Wierstra. Continuous control with deep reinforcement learning, arXiv preprint arx: 10XY:1550... 1509.099711 2015.
[17] Chen Ma, Dylan Ashley, Junfeng Wen, and Yoshua Bengio. Universal successor features for transfer reinforcement learning. arXiv preprint arXiv:2001.04025,, 2020.
10

---

[18] Anusha Nagabandi, Kurt Konolige, Sergey Levine, and Vikash Kumar. Deep dynamics models for learning dexterous manipulation. In Conference on robot learning, pages 1101-1112. PMLR, 2020.
[19] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity exploration by self-supe prediction. In International conference on machine learning, pages 2778-
2787. PMLR, 2017.
[20] Darshan Patil, Janarthanan Rajendran, Glen Berseth, and Sarath Chandar. Intelligent switching for reset-free rl. arXiv preprint M.Xiv:2405.01660 2024.
[21] Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function ap- proximators. In International conference on machine learning, pages 1312-1320. PMLR, 2015.
[22] Archit Sharma, Michael Ahn, Sergey Levine, Vikash Kumar, Karol Hausman, and Shixiang Gu. Emergent real-world robotic skills via unsupervised off off-policy reinforeement learning. arXiv preprint 20X.:::2004.127744 2020.
[23] Archit Sharma. Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Au- tonomous reinforcement learning via subgoal curricula. Advances in Neural Information Processing Systems, 1111184441888 2021.
[24] Archit Sharma, Rehaan Ahmad, and Chelsea Finn. state-distribution matching approach to non-episodic reinforcement learning. In International Conference on Machine Learning, pages 19645-19657. PMLR, 2022.
[25] Archit Sharma, Kelvin Xu, Nikhil Sardana, Abhishek Gupta, Karol Hausman, Sergey Levine, and Chelsea Finn. Autonomous reinforcement learning: Formalism and benchmarking. In International Conference on Learning Representations, 2022.
[26] Annie Xie, Fahim Tajwar, Archit Sharma, and Chelsea Finn. When to ask for help: Proactive interventions in autonomous reinforcement learning, Advances in Neural Information Processing Systems, 35:16918- 1118-66,,,0-- 2022.
[27] Kelvin Xu, Siddharth Verma, Chelsea Finn, and Sergey Levine, Continual learning of control primitives: Skill discovery via reset-games, Advances Neural Information Processing Systems, 33:4999-5010, 2020.
[28] Ali Yahya, Adrian Li, Mrinal Kalakrishnan, Yevgen Chebotar, and Sergey Levine. Collective robot reinforcement learning with distributed asynchronous guided policy search. In 2017 IEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pages 79-86. IEEE, 2017.
[29] Andy Zeng, Shuran Song. Johnny Lee, Alberto Rodriguez, and Thomas Funkhouser. Tossingbot: Learning to throw arbitrary objects with residual physics. IEEE Transactions Robotics, 36 (4):1307-131, 2020.
[30] Henry Zhu, Abhishek Gupta, Aravind Rajeswaran, Sergey Levine, and Vikash Kumar. Dexterous manipulation with deep reinforcement learning: Efficient, general, and low-cost. In 2019 International Conference on Robotics and Automation (ICRA), pages 3651-3657. IEEE, 2019.
[31] Henry Zhu, Justin Yu, Abhishek Gupta, Dhruy Shah, Kristian Hartikainen, Avi Singh, Vikash Kumar, and Sergey Levine. The ingredients of real-world robotic reinforcement learning. In International Conference on Learning Representations, pages 10896-109 10916, 2020.
11

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main contribution of this paper is robust and scalable autonomous reinforcement learning (ARL) algorithm. We believe that this contribution and its scope are clearly reflected in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of our work are discussed in the conclusion.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
12

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
The paper provides sufficient information to reproduce our experimental results, as detailed in the experiments and Appendices and C.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
13

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
The supplementary materials include all of our code well as README.nd file with instructions for reproducing the main experimental results.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.cc.
aublic/guiddes/Codeubbbmsss www.eeeeuuuuisssiooooooooooo for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wwtp:/cccppppgguuuiiaeeeae
Stteee///ooSSSSeissSioica www.ssssiicgcyo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper provides all training and evaluation details, including hyperparame- ters and the type of optimizer, in Appendices B and C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The quantitative results in the experiments report the means and standard deviations computed over five random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
14

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper provides sufficient information the computer resources to repro- duce our experimental results in Appendix C.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't itdn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We have read the NeurIPS Code of Ethics and confirm that work fully complies with it.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses both the potential positive and negative societal impacts
in Appendix D.
Guidelines:
15

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We used Gymnasium, set of benchmark tasks, evaluate our work and properly cited it the experiments and Appendix C.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
16

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, papprsaipphaeeoooccaaappp thcode. com/ com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: While no new assets are released with this submission, the code and its instructions in the supplementary materials will be made publicly available if the paper is accepted.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
17

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMS as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy hatt:p//eearrpppcc///ooeeeeeeeee /ct/////oeeeeeccc/////////. for what should or should not be described.
18


Reviews:
[{"review_id": "JvKRg889tD", "reviewer": "Reviewer_wpan", "summary": "This paper proposes an autonomous RL algorithm called RSA based on two components: 1) a curriculum created by setting goal states for forward and backward policies based on which states would be informative goal/initial states. This is done with a combination of heuristics and by training a classifier that estimates the ability of the current agent to reach goals. 2) A mechanism to detect and avoid irreversible states using successor features. They show that this algorithm needs fewer resets and learns faster on various navigation and manipulation environments compared to previous methods.", "questions": "- What does RSA stand for? Robust and Scalable Autonomous RL? If so, can you further explain the scalable part of the name?\n- Why is using the final state of failed iterations as the goal state for the next iteration a good heuristic? In HER it makes sense since the agent is able to learn without having to rollout more trajectories, but in this case, it’s not clear why those failed intermediate states would be good future goal states. Can you run an ablation to see whether this is necessary? And if so provide a clearer understanding of why?\n- Is there any weighting of more recent trajectories for training the state information estimator given that sampling old trajectories could lead to out of date labels?\n- I am not exactly clear on how the mapping function $m$ used in computing the successor features comes from. Section 3.1 implies that it comes from the environment and is not learned? \n- Could you create a visualization of which states the agent believes are irreversible and how that evolves over time across a couple environments? It would be interesting to see how well that aligns with our notion of irreversible.\n- Is get_informative_goal in general just sampling from the goal states provided by the environment (except in the cases of failed forward trajectories)?\n- Can you provide the ablation results for the rest of the environments as well?\n- For Figure 7, right, it would be helpful to also see the success rate plotted against the timesteps in the environment as well.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The algorithm presented is effective at learning in an autonomous fashion in both reversible and irreversible environments.\n- The mechanism to find irreversible states seems like it works well.\n- The idea of switching from the forward to the reset policy once the initial state is difficult enough is interesting. \n\nWeaknesses:\n- The method does introduce 3 new hyperparameters that need to be searched over. It’s also unclear how the search was done, and if the baselines were searched over in a commensurate fashion.\n- The goal setting heuristic (when the policy fails at achieving the forward goal) seems not as well motivated. It’s unclear why using the last state of the failed rollout is a good forward goal. A useful ablation to have would be if you simply chose a goal from the rest of your goal set.\n- Some parts of the detecting irreversible states method are unclear (see below).", "was_revised": true, "final_justification": "The authors have answered my most pressing questions. I will be maintaining my score.", "rebuttal": "We would like to thank you for taking the time to review our work and for providing thoughtful suggestions. We are pleased that you found the algorithm to be effective in both reversible and irreversible environments, the mechanism for finding irreversible states to work well, and the idea of switching between the forward and reset policies to be interesting. Below are the responses regarding the raised concerns and suggestions. We believe that our responses address all issues in the review.\n\n---\n\n**[W1] How the hyperparameters are searched:** Thank you for your constructive feedback. To provide a fair comparison, **we performed grid search for the hyperparameters in both RSA and the baselines, and tried our best to find the optimal values.** Table 2 in Appendix B summarizes the key hyperparameters used in our experiments.\n\nTo address your concern, **we also conducted additional experiments on PointIMaze-Trap to examine the effects of our key hyperparameters.** First, we analyzed the performance according to the hyperparameters, $\\lambda_1$ and $\\lambda_2$, which are the lower and upper thresholds of the reachability range of informative initial or goal states. We observed that too difficult initial and goal states ($\\lambda_1=0.0$ and $\\lambda_2=0.1$) cause even more manual resets than random initial states. Conversely, too easy initial and goal states ($\\lambda_1=0.9$ and $\\lambda_2=1.0$) result in lower sample efficiency than the informative initial and goal states ($\\lambda_1=0.1$ and $\\lambda_2=0.6$). These results confirm the benefits of identifying informative initial and goal states. \n\nNext, we investigated the performance according to the reversibility threshold, $\\lambda_3$. We observed that RSA maintains good performance and properly identifies irreversible states, except when the reversibility threshold is much higher than the output of the initialized successor features, $\\psi^{\\pi_r}(s,a)$, which causes the agent to classify all states as irreversible at the beginning of training. This discussion and additional experiments will be added to Appendix C in the revised paper.\n\n---\n\n**[W2 & Q2] Using the last state of the failed rollout as the goal state for the next iteration:** The last state of the failed rollout and its neighboring states are clearly not trained enough for at least one goal state, including the current goal state that the agent failed to reach. Placing a goal state near these states can provide meaningful learning signals to them. We empirically found that using the last state of the failed rollout as the goal state for the next episode is a simple and efficient way to provide such signals without any additional computation.\n\nFollowing your constructive suggestion, **we conducted additional experiments to examine the benefits of using the last state of the failed rollout as the next goal state.** We compared RSA to the following two variants: 1) randomly sampling the next goal state from all previous goal states, and 2) randomly sampling the next goal state from the identified set of informative goal states. As expected, RSA outperforms the first variant and achieves comparable performance with the second variant while requiring less computational cost. We will update the fourth paragraph of Section 4.1 to provide a clearer understanding of using the last state of the failed rollout as the next goal state, and add additional experimental results to Appendix C in the revised paper. \n\n---\n\n**[W3 & Q4] The mapping function $m$ used in computing the successor features:** Following previous goal-conditioned RL algorithms, we use mapping functions provided by the tasks in our experiments. While learning the mapping functions is orthogonal and complementary to the main contribution of our work, we believe that combining RSA with state-of-the-art representation learning algorithms to learn these functions is an interesting direction for future work. Note that RSA is agnostic to the form of the mapping function. We will add this discussion to Section 6 in the revised paper.\n\n---\n\n**[Q1] What RSA stands for:** RSA refers to the Robust and Scalable Autonomous (RSA) RL algorithm introduced in this paper. We originally denoted the term as RSA-RL but decided to use the more concise form, RSA, for brevity and readability.\n\nRSA is more scalable than previous ARL algorithms, which typically assume either reversible environments or access to task-specific knowledge, such as demonstrations or reversibility labels, for identifying irreversible states. These assumptions are impractical and difficult to scale to real-world tasks, where each task may have its own unique set of irreversible states. **To the best of our knowledge, RSA is the first autonomous RL algorithm that can reduce manual resets in both reversible and irreversible environments without task-specific knowledge.** Table 1 in the main paper summarizes the key features that distinguish RSA from previous ARL algorithms.\n\n---\n\n**[Q3] Weighting of more recent trajectories for training the state information estimator:** We did not use the idea of weighting more recent trajectories in our experiments. Specifically, Figures 5 and 6 in the main paper, which demonstrate that the state information estimator can identify informative initial and goal states based on the agent’s learning progress, were obtained by sampling trajectories randomly.\n\nAs a side note, we find the idea of weighting more recent trajectories to be quite interesting. We believe this idea could make the training of the state information estimator more efficient and stable. Therefore, we would be glad to further explore this idea in our future work.\n\n---\n\n**[Q5] Visualizing how irreversible states evolve over training time:** Thank you for your constructive and insightful feedback. We definitely agree with your comment that visualizing how irreversible states evolve over training time would be interesting. Following your suggestion, **we implemented additional evaluation code to visualize irreversible states in a manner similar to Figures 5 and 6.** The visualization results on PointIMaze-Trap and AntOpen-Trap show that at the beginning of training, all states are identified as reversible, as the state information estimator is initialized to output values higher than the reversibility threshold. As training progresses, irreversible states are gradually detected around the edge of the purple boxes, which represent the boundaries between reversible and irreversible regions of the state space. These results indicate that irreversible states identified by RSA align well with our intuitive notion of irreversibility. We will add the visualization results to Appendix C in the revised paper.\n\n---\n\n**[Q6] get_informative_goal function:** We are pleased to discuss the get_informative_goal$(B_r, G_{1:k-1}, I(s,g))$ function, presented in Algorithm 1 of the main paper. This function outlines how RSA takes into account the agent’s learning progress to select an informative goal state $g_k$ at iteration $k$. **We would like to emphasize that selecting informative goal states is fundamentally distinct from randomly sampling ordinary goal states from the predefined set provided by the environment.** RSA utilizes the state information estimator to identify a subset of informative goal states from all goal states encountered in previous forward rollouts, including both successful and failed rollouts. It then samples one informative goal state for the current episode from this identified subset. This process prevents the agent from being assigned goal states that are either too difficult or too easy. Note that, unlike previous ARL algorithms, RSA does not assume that specific candidate goal states are predefined by the environment but rather allows goal states to be located anywhere within the entire state space. **Algorithm 2 in Appendix A describes the procedure of get_informative_goal$(B_r, G_{1:k-1}, I(s,g))$ in further detail.**\n\n---\n\n**[Q7] Ablation results for the rest of the environments:** To address your concern, **we conducted additional ablation studies to examine the benefits of the key components of RSA in maze navigation tasks.** First, we compared RSA with its variant, RSA w/o SIE (which does not identify informative states), in the reversible environments PointIMaze and AntOpen. Similar to the left plot of Figure 7, RSA achieves better asymptotic performance and sample efficiency than RSA w/o SIE. This result confirms the benefits of identifying informative initial and goal states with SIE. \n\nNext, we compared RSA with its variant, RSA w/o SFs (which does not identify irreversible states), in the irreversible environments PointIMaze-Trap and AntOpen-Trap. Unsurprisingly, RSA w/o SFs failed to learn both navigation tasks as it is unable to detect and avoid irreversible states. This outcome is consistent with the results shown in the right plot of Figure 7. We will update Figure 7 to include these additional ablation results and add the corresponding discussion to the fourth paragraph of Section 5.2 in the revised paper.\n\n---\n\n**[Q8] Plotting the success rate against timesteps (Figure 7):** We definitely agree with your comment that including a plot of the success rate against timesteps, even in the irreversible environments, would be helpful in improving our work. Changing the x-axis of the plot poses no difficulty and does not require additional effort. We will add the plot against timesteps to Appendix C in the revised paper.\n\n---\n\nThank you again for your time and effort in reviewing our work. We have carefully considered your valuable comments and tried our best to address your concerns with the clarifications and the additional new experiments. If our response has addressed your primary concerns, we would greatly appreciate it if you could update your score to reflect that. We are also willing to discuss with you if you have any further concerns."}, {"review_id": "RJY9YCIoIL", "reviewer": "Reviewer_RMHP", "summary": "The paper proposes RSA, an ARL framework designed to reduce manual resets in both reversible and irreversible environments without relying on task-specific knowledge. RSA jointly trains a forward policy to solve tasks,  and a reset policy (to recover from failures. To avoid irreversible states, RSA encodes agent behaviors using successor features rather than relying on state-specific labels, identifying states where agents lose control over goal-relevant features. Through experiments on diverse navigation and manipulation tasks, RSA demonstrates better performance and reset efficiency compared to prior ARL methods, showing its robustness and scalability for real-world applications.", "questions": "1. How sensitive is RSA to the choice of feature representation $\\phi$ used in successor features? Could poor feature choices lead to inaccurate identification of irreversible behaviors?\n\n2. In large or visually rich environments, does using Random Network Distillation (RND) provide sufficient coverage for the reset policy? Would structured exploration (e.g., skill discovery) help?\n\n3. Since RSA relies on accumulating behavioral evidence before identifying irreversible states, how can it prevent the agent from entering catastrophic states in the early stage of training?\n\n4. Can RSA handle high-dimensional observations like images or point clouds? How would its components (e.g., state estimator, SFs) scale to such settings?\n\n5. How accurate is the self-supervised estimator $I(s,g)$ during early training? Does misestimation bias curriculum generation?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength\n\n1. RSA avoids reliance on handcrafted rewards, reversibility labels, or demonstrations, making it more generalizable to real-world settings.\n\n2. Using successor features to detect irreversible states via behavioral patterns is a novel approach.\n\n3. RSA demonstrated strong performance and fewer manual resets across both reversible and irreversible tasks in a variety of domains.\n\nWeakness \n\n1. The reset policy uses RND, which may be inefficient in complex or high-dimensional environments.\n\n2.  RSA can only identify irreversible states after sufficient behavioral data is collected for identification, potentially causing early-stage failures and costs.", "was_revised": true, "final_justification": "The paper proposes RSA, an ARL framework designed to reduce manual resets in both reversible and irreversible environments without relying on task-specific knowledge. After the detailed rebuttal from the authors, the reviewer is convinced by the supporting answers to the proposed questions. Hence, the reviewer would maintain the preliminary rating.", "rebuttal": "Thank you for your insightful and constructive feedback. We are pleased that you found the algorithm to be more generalizable to real-world settings, using of successor features to detect irreversible states to be novel, and the performance to be strong with fewer manual resets across both reversible and irreversible tasks in a variety of domains. We would like to address your comments and questions below:\n\n---\n\n**[W1 & Q2] Using RND as the reset policy:** We definitely agree with your comment that using RND as the reset policy may be inefficient in complex environments. Your idea to leverage structured exploration with skill discovery algorithms for sufficient coverage is quite insightful.\n\n**Inspired by your idea, we found an interesting recent work called LSR [1] that utilizes skills to discover diverse initial states.** The key insight behind LSR is that the need to reset the agent with diverse initial states provides a natural setting for discovering distinct skills. We believe LSR can be a strong starting point for integrating your idea into our work. We would be glad to further explore this research direction in future work.\n\n---\n\n**[W2 & Q3] Preventing the agent from entering irreversible states in the early stage of training:** We are glad to discuss the issue of entering irreversible states in the early stage of training. While this issue is not considered in this paper, we are developing a key idea to address it as follows: unlike informative states that should be identified based on the agent’s learning progress, irreversible states can be identified regardless of the agent’s learning progress. For example, once an agent in HandManipulate drops an object, it can no longer pick up or manipulate the object, regardless of how much it has learned.\n\nBased on our key idea, we are exploring the use of foundation models, such as large language models (LLMs) or vision-language models (VLMs), to identify irreversible states. These models are typically pre-trained on massive offline datasets and then frozen during downstream use. Therefore, they are not well-suited to estimating an agent’s learning progress to identify informative states every iteration. However, their common-sense reasoning abilities make them promising for identifying irreversible states across diverse tasks even in the early stage of training. Suppose we have access to GPT-4o, a multimodal foundation model that can handle both text and images. If we provide GPT-4o with the current image observation and a text query such as “Does this image represent a reversible state?”, it could respond with a binary answer like “yes” or “no” without requiring further training. This binary signal can then be used to identify irreversible states even in the early stage of training. We will add this discussion to Section 6 in the revised paper. We hope this discussion addresses your concern about early-stage failures.\n\n---\n\n**[Q1] Choice of feature representation used in successor features:** Designing mapping functions that extract relevant feature representations from states or observations is critical for identifying irreversible behaviors. **We did not consider this issue as it is orthogonal to our main contribution of reducing manual resets in both reversible and irreversible environments without task-specific knowledge.** To be specific, following previous goal-conditioned RL algorithms, RSA extracts relevant feature representations by using the mapping functions provided by the tasks in our experiments.\n\nBefore concluding this discussion, we would like to emphasize that combining RSA with state-of-the-art representation learning algorithms to learn the mapping functions is an interesting direction for future work. Note that RSA is agnostic to the form of the mapping function. We will modify the first paragraph of Section 4.2 to make the importance of feature representation clearer in the revised paper.\n\n---\n\n**[Q4] Handling high-dimensional observations: RSA is fully compatible with all types of observations, including high-dimensional observations such as images and point clouds.** To handle such observations, the models within RSA should be constructed using more sophisticated networks, such as Transformers or MLP-Mixers. While training these advanced networks may require additional techniques, we expect that such techniques can be easily integrated into our work. We will add this discussion to Section 6 in the revised paper.\n\n---\n\n**[Q5] How accurate the state information estimator is during early training:** The initialized state information estimator cannot accurately estimate how informative initial and goal states are in the early stage of training. Note that RSA does not pretrain the state information estimator, as doing so would require task-specific knowledge, such as expert demonstrations. Instead, RSA initializes the state information estimator to identify all pairs of initial and goal states as informative. We empirically observed that this simple and practical initialization trick enables the agent to collect diverse pairs of initial and goal states during early training, which alleviates biased curriculum generation. The implementation of this initialization trick corresponds to the weight_init function in utils.py of our code. We hope this discussion addresses your concern about the state information estimator.\n\n---\n\nThank you again for your time and effort in reviewing our work. We have carefully considered your valuable comments and tried our best to address your concerns with the clarifications and the additional new experiments. If our response has addressed your primary concerns, we would greatly appreciate it if you could update your score to reflect that. We are also willing to discuss with you if you have any further concerns.\n\n[1] Kelvin Xu, Siddharth Verma, Chelsea Finn, Sergey Levine. “Continual Learning of Control Primitives: Skill Discovery via Reset-Games.” Advances in Neural Information Processing Systems. 2020."}, {"review_id": "V4AP60toRG", "reviewer": "Reviewer_wsCV", "summary": "The manuscript proposes a robust and scalable autonomous reinforcement learning (ARL) algorithm, RSA, which addresses the limitations of existing ARL algorithms in handling diverse initial and goal states and avoiding irreversible states without relying on task-specific knowledge. RSA generates a curriculum by identifying informative states based on the agent’s learning progress and encodes behaviors to detect and avoid irreversible states. Experimental results show that RSA outperforms existing ARL algorithms in both reversible and irreversible environments with fewer manual resets.", "questions": "1. Why were other methods listed in Table 1, such as MEDAL and PAINT, not included in the experimental comparison? Are there technical or practical reasons for their exclusion?\n    \n2. In Section 5.2, why are experimental results on handReach and HandManipulate environments not reported for the comparsion?\n    \n3. In line 184, the authors expect that the successor feature values of irreversible state-action pairs to be much lower than those for reversible pairs. Is there theoretical justification or empirical evidence supporting this claim?\n    \n4. The explanations for experimental results are sometimes insufficient. For example, in line 227, why do narrow initial or goal state distributions lead to low success rates for R3L and LNT? In line 230, how is it observed that agents collect more biased experiences in AntOpen and AntOpen-Trap?\n    \n5. In line 243, it is mentioned that RSA and LNT require a comparable number of manual resets in irreversible environments. How does this demonstrate the advantage of RSA? Is LNT’s performance due to its use of task-specific knowledge, and how does this affect the fairness of the comparison?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n1. The manuscript addresses a significant and practical problem in reinforcement learning, focusing on reducing manual resets and improving applicability in real-world settings.\n    \n2. RSA relaxes assumptions on initial and goal state distributions and does not require task-specific knowledge to avoid irreversible states, enhancing its generalizability and practical value.\n    \n3. The related work section provides a comprehensive comparison of ARL algorithms, and Table 1 clearly illustrates the advantages and coverage of the proposed approach.\n    \n4. The experimental results demonstrate RSA’s effectiveness and robustness in both reversible and irreversible environments, indicating its potential for real-world deployment.\n    \n5. The authors have provided the code, and the code is well-documented, facilitating reproducibility and independent verification of the experimental results.\n    \n\n### Weaknesses\n\n1. The experimental comparison is limited, as it only includes two ARL baselines (and their multi-goal variants), omitting other relevant algorithms such as MEDAL and PAINT listed in Table 1. This limits the strength of the empirical validation.\n    \n2. The technical novelty of RSA appears incremental, and the method depends on three manually set threshold hyperparameters. The manuscript does not provide an analysis of the sensitivity or robustness of these parameters.\n    \n3. The experimental environments are relatively simple and mainly focus on navigation tasks, which may not sufficiently demonstrate the scalability and versatility of RSA in more complex or diverse domains.", "was_revised": true, "final_justification": "Throughout the review process, my initial concerns centered on the lack of comprehensive empirical comparisons with all relevant baselines and the absence of quantitative analysis supporting a key design choice. The authors have responded to each point, conducting new experiments that directly address these issues. The expanded empirical evaluation now situates RSA more clearly within the broader ARL literature, and the quantitative analysis of successor features substantiates the method’s rationale. These improvements have shifted my opinion from a position of skepticism to one of support.", "rebuttal": "Thank you for your insightful and constructive feedback. We are pleased that you found the solved problem to be significant and practical, the algorithm to improve applicability and generalizability in real-world settings, the related work section to provide a comprehensive comparison of ARL algorithms, Table 1 to clearly illustrates the advantage and coverage of RSA, the experimental results to demonstrate the effectiveness and robustness of RSA in both reversible and irreversible environments, and the submitted code to be well-documented. We would like to address your comments and questions below:\n\n---\n\n**[W1 & Q1] Comparison with other recent ARL algorithms (MEDAL and PAINT):** One of the most challenging aspects of designing our experiments was selecting baselines that can be compared fairly. In particular, unlike RSA, previous ARL algorithms require task-specific knowledge to reduce manual resets in reversible or irreversible environments. For example, MEDAL relies on expert demonstrations to generate its curriculum, and PAINT needs reversibility labels for all states to identify irreversible states. The performance of these algorithms can vary significantly depending on how such task-specific knowledge is constructed. For this reason, we selected ARL algorithms such as LNT and R3L, which leverage task-specific knowledge inherently available from the evaluation tasks, as baselines, while excluding algorithms like MEDAL and PAINT that require us to manually construct the task-specific knowledge they depend on.\n\n---\n\n**[W2-1] Technical novelty of RSA:** We are glad to discuss the technical novelty of RSA. **To the best of our knowledge, RSA is the first autonomous RL algorithm that can reduce manual resets in both reversible and irreversible environments without task-specific knowledge.** The key ideas behind RSA are 1) generating a curriculum by identifying informative initial and goal states based on the agent’s learning progress, and 2) detecting and avoiding irreversible states by encoding the behaviors exhibited in those states rather than the states themselves. **These key ideas are fundamentally different from those of previous algorithms and are not merely incremental extensions.** \n\n---\n\n**[W2-2] Analysis of the effects of hyperparameters introduced in RSA:** Thank you for your constructive feedback. To address your concern, **we conducted additional experiments on PointIMaze-Trap to examine the effects of our key hyperparameters.** First, we analyzed the performance according to the hyperparameters, $\\lambda_1$ and $\\lambda_2$, which are the lower and upper thresholds of the reachability range of informative initial or goal states. We observed that too difficult initial and goal states ($\\lambda_1=0.0$ and $\\lambda_2=0.1$) cause even more manual resets than random initial states. Conversely, too easy initial and goal states ($\\lambda_1=0.9$ and $\\lambda_2=1.0$) result in lower sample efficiency than the informative initial and goal states ($\\lambda_1=0.1$ and $\\lambda_2=0.6$). These results confirm the benefits of identifying informative initial and goal states. \n\nNext, we investigated the performance according to the reversibility threshold, $\\lambda_3$. We observed that RSA maintains good performance and properly identifies irreversible states, except when the reversibility threshold is much higher than the output of the initialized successor features, $\\psi^{\\pi_r}(s,a)$, which causes the agent to classify all states as irreversible at the beginning of training. This discussion and additional experiments will be added to Appendix C in the revised paper.\n\n---\n\n**[W3] Relatively simple experimental tasks:** Our experiments involve both reversible and irreversible tasks, consisting of four navigation tasks and two manipulation tasks. We would like to emphasize that, unlike standard reinforcement learning benchmarks, these tasks do not rely on periodic resets or provide access to any task-specific knowledge, such as reversibility labels or demonstration data. **These constraints make our tasks significantly more challenging than typical benchmarks, which may appear complex but are less demanding in practice.** In particular, one of our irreversible tasks, HandManipulate, is a well-known challenging task where standard RL algorithms struggle to learn even with periodic resets. We therefore believe that our experimental tasks are sufficient to demonstrate the scalability and versatility of RSA.\n\n---\n\n**[Q2] Comparison results on HandReach and HandManipulate:** The reason we reported ablation studies on the key components of RSA instead of comparison results on HandReach and HandManipulate was to minimize redundancy and present as many diverse findings as possible within the page limit of the main paper. To be specific, the comparison results for these two manipulation tasks were largely consistent with those from the manipulation tasks, with the exception that all the baselines obtained extremely poor performance on HandManipulate. We will add the comparison results on both manipulation tasks to Appendix C in the revised paper.\n\n---\n\n**[Q3] Empirical evidence supporting the claim that the SFs of irreversible pairs are much lower than those for reversible pairs:** Irreversible states are those from which an agent cannot recover without manual resets. As you mentioned, we claim that the successor features (SFs) for irreversible state-action pairs are much lower than those for reversible pairs. **This claim is based on our empirical observation that, in irreversible states, an agent loses control over goal-relevant features, regardless of the actions it takes.** For example, a vehicle damaged in an accident is unable to move toward its goal state by itself, and a manipulator cannot control an object once it leaves its workspace. Our experimental results on diverse irreversible tasks provide empirical evidence supporting this claim.\n\nInspired by your constructive question, **we implemented additional evaluation code to visualize irreversible states in a manner similar to Figures 5 and 6.** The visualization results on PointIMaze-Trap and AntOpen-Trap show that at the beginning of training, all states are identified as reversible, as the state information estimator is initialized to output values higher than the reversibility threshold. As training progresses, irreversible states are gradually detected around the edge of the purple boxes, which represent the boundaries between reversible and irreversible regions of the state space. These results indicate that irreversible states identified by RSA align well with our intuitive notion of irreversibility. We will add the visualization results and the corresponding discussion to Appendix C in the revised paper.\n\n---\n\n**[Q4-1] Low success rates due to narrow initial or goal state distributions:** The experimental results shown in Figure 4 describe that LNT and R3L achieve low success rates on reversible and irreversible maze navigation tasks. Several previous works support the interpretation that these poor performance is due to their narrow initial or goal state distributions. Sharma et al. [1, 2] empirically demonstrate that narrow initial state distributions lead to poor asymptotic performance and sample efficiency, as they constrain the state distributions visited by the agent. Kakade and Langford [3] provide a theoretical result showing that a more uniform initial state distribution can accelerate policy improvement by encouraging policy improvement at states that are unlikely to be visited. We will add this discussion to the first paragraph of Section 5.2 in the revised paper. \n\n---\n\n**[Q4-2] More biased experiences in AntOpen and AntOpen-Trap:** LNT-MG and R3L-MG randomly sample initial and goal states, as these baselines cannot identify which states are informative. We empirically observed that in AntOpen and AntOpen-Trap, such random sampling leads to more biased experiences compared to PointIMaze and PointIMaze-Trap. Specifically, in PointIMaze and PointIMaze-Trap, the agent can collect diverse experiences even with randomly sampled initial and goal states due to its simple dynamics. In contrast, the agent in AntOpen and AntOpen-Trap struggles to collect diverse experiences unless informative initial and goal states are continously provided, as it has more complex dynamics. We will add this discussion to the first paragraph of Section 5.2 in the revised paper. \n\n---\n\n**[Q5] Advantage of RSA over LNT:** Figure 4 shows that RSA and LNT require a comparable number of manual resets in irreversible tasks. However, we would like to note that RSA achieves better asymptotic success rates and sample efficiency than LNT. Furthermore, while LNT relies on task-specific knowledge to identify irreversible states, RSA identifies them without any task-specific knowledge. **These results suggest that RSA outperforms LNT under more challenging and general conditions.**\n\n---\n\nThank you again for your time and effort in reviewing our work. We have carefully considered your valuable comments and tried our best to address your concerns with the clarifications and the additional new experiments. If our response has addressed your primary concerns, we would greatly appreciate it if you could update your score to reflect that. We are also willing to discuss with you if you have any further concerns.\n\n[1] Archit Sharma, Rehaan Ahmad, Chelsea Finn. “A State-Distribution Matching Approach to Non-Episodic Reinforcement Learning.” International Conference on Machine Learning. 2012.\n\n[2] Archit Sharma, Kelvin Xu, Nikhil Sardana, Abhishek Gupta, Karol Hausman, Sergey Levine, Chelsea Finn. “Autonomous Reinforcement Learning: Formalism and Benchmarking.” International Conference on Learning Representations. 2022.\n\n[3] Sham Kakade, John Langford. “Approximately Optimal Approximate Reinforcement Learning.” International Conference on Machine Learning. 2002."}, {"review_id": "KAunCWZNhn", "reviewer": "Reviewer_tTbb", "summary": "This paper addresses a key challenge in real-world reinforcement learning: the dependence on frequent manual resets of the environment after episode termination. This reset requirement is both labor-intensive and impractical in physical systems such as robotics.\nThe authors propose RSA (Robust and Scalable Autonomous RL), a method that trains both forward policies (to solve the task) and reset policies (to bring the environment back to favorable conditions) without task-specific knowledge and under diverse initial/goal states.\nThe paper is evaluated on continuous-control benchmarks including robotic manipulation and locomotion tasks, comparing RSA to state-of-the-art ARL methods such as SPOT and SkewFit.", "questions": "I think this is a **very thoughtful and practically valuable paper**. The motivation is crystal clear, and the solution is elegant and well-grounded. There’s a nice balance of algorithmic design and empirical demonstration.\n\nThat said, I would feel more confident in recommending a higher score if:\n\n* There was a discussion of robustness to noise, and\n* Some insight into the computational cost or training complexity was provided.\n\nIf these points are addressed, I would strongly lean toward an **accept** recommendation.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\nI really like the idea of encoding how a state is reached instead of just looking at the state itself to judge irreversibility. This is quite clever—it acknowledges that the same final state can be reversible or not depending on the trajectory taken to get there, which is often true in manipulation tasks.\n\n**Weaknesses**\n\n**1. I’d like to see more theoretical insight**\nWhile the empirical work is strong, I feel the paper would benefit from some **formal analysis**. For example:\n\n* Is the learning progress metric provably helpful under certain assumptions?\n* How does the behavior encoding affect reset success in theory?\n  I’m not asking for complete proofs, but even a toy example or intuitive analysis would help deepen the reader’s understanding of *why* RSA works.\n\n**2. I’m unsure how well this generalizes under noise**\nOne concern I have is how RSA performs in **stochastic or noisy environments**. If behavior encodings vary a lot between rollouts due to randomness, would the system still detect irreversible states reliably? I didn’t see this discussed in the paper. Including an experiment or a brief analysis on this point would strengthen the case for generality.\n\n**3. Training complexity feels high but isn’t discussed**\nRSA has multiple components (two policies, behavior encoders, progress estimators), and I wonder how **computationally heavy** the training is. This is especially relevant if someone wants to apply RSA to a real robot system. I would have appreciated a brief discussion of training time, or resource requirements.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful and constructive feedback. We are pleased that you found the paper to be very thoughtful and practically valuable, the motivation to be crystal clear, the balance of algorithmic design and empirical demonstration to be nice, and the solution to be elegant and well-grounded. We would like to address your comments and questions below:\n\n---\n\n**[W1] Intuitive analysis of why RSA works:** We definitely agree with your comment that an intuitive analysis would help deepen the reader’s understanding of why RSA works. RSA generates a curriculum by identifying informative initial and goal states based on the agent’s learning progress. We hypothesize that informative states are neither overly difficult nor trivially easy for the agent, which is motivated by the fact that **humans learn more effectively when tasks are provided in a meaningful order rather than at random, particularly when those tasks are around the edge of the current knowledge.** \n\nTo detect and avoid irreversible states without task-specific knowledge, RSA encodes the behaviors exhibited in irreversible states rather than the states themselves. **In Appendix A, we provide a theoretical analysis demonstrating that the goal-conditioned value function for reversible states is greater than that for irreversible states when RSA penalizes visiting irreversible states.** We believe this theoretical analysis helps the readers understand how RSA encourages the agent to avoid irreversible states identified with the successor features. \n\n---\n\n**[W2 & Q1] Robustness of detecting irreversible states to noise or stochasticity:** Thank you for your constructive feedback. We believe that RSA can effectively detect behaviors in irreversible states by using successor features even in stochastic environments. This is because the successor features in our work are trained to minimize temporal difference (TD) error on subsequent states. In other words, the successor features naturally take into account stochasticity in transitions by updating with expectations over transitions rather than deterministic transitions. \n\nFollowing your suggestion, **we implemented a new stochastic task called PointIMaze-Trap-Stochastic, a variant of PointIMaze-Trap, and conducted additional experiments.** In this task, the agent executes a randomly chosen alternative action with 25% probability instead of the action it originally selects. To provide a thorough analysis, **we visualized the irreversible states identified on both PointIMaze-Trap and PointIMaze-Trap-Stochastic in a manner similar to Figures 5 and 6 of the main paper.** We observed that, in PointIMaze-Trap, only states very close to the edges of the purple boxes are identified as irreversible. In contrast, in PointIMaze-Trap-Stochastic, states farther from the edges are also classified as irreversible. Given that uncertainty in the transitions expands the region where states are more susceptible to entering irreversible states, these experimental results suggest that RSA can robustly identify irreversible states even in stochastic environments. We will add the visualization results and the corresponding discussion to Appendix C in the revised paper. We hope this discussion addresses your concern.\n\n---\n\n**[W3 & Q2] Resource requirements and computational costs:** We are glad to discuss the resource requirements and computational cost of RSA. Our experiments were conducted on two machines: a PC equipped with an RTX 5080 GPU and a workstation with four RTX 4090 GPUs. Each RSA run uses a fraction of a single GPU. \n\nWe have to admit that RSA may require more computational cost than standard RL algorithms. This is because RSA trains two policies and two value functions to learn both how to solve tasks and how to reset environments simultaneously. In addition, RSA utilizes the state information estimator (SIE) to identify informative initial and goal states and the successor features (SFs) to detect irreversible states. Since the SIE and SFs have network structures similar to the policy and value function, respectively, RSA can be roughly considered as consisting of three policies and three value functions.\n\n**However, we would like to emphasize that the main bottleneck for training real-world robots is not that the computational cost is too high, but rather that manual resets entirely stop collecting data.** Our experimental results demonstrate that RSA requires significantly fewer manual resets even without task-specific knowledge. This indicates that RSA achieves better sample efficiency than previous state-of-the-art ARL algorithms in the real world. We will add this discussion to Appendix B in the revised paper. We hope this discussion addresses your concerns about resource requirements and computational costs.\n\n---\n\nThank you again for your time and effort in reviewing our work. We have carefully considered your valuable comments and tried our best to address your concerns with the clarifications and the additional new experiments. If our response has addressed your primary concerns, we would greatly appreciate it if you could update your score to reflect that. We are also willing to discuss with you if you have any further concerns."}]

Author rebuttal:

Meta-review: A. This paper proposes an RL method to minimize manual resets by learning a forward and backward policy. The method uses successor features to identify states where agents lose control over goal-relevant features. During training, the method uses a curriculum for learning. Experiments show that the proposed method requires fewer resets and learns faster on various navigation and manipulation environments compared to previous methods.

B. Reviewers appreciated that the method lifts assumptions of prior work (e.g., no hand-crafted rewards) while achieving strong empirical performance. They also appreciated that the paper included code.

C. Reviewers had questions about a few aspects of the paper (e.g., computational costs, performance in stochastic environments) and suggested some new experiments and baselines (e.g., MEDAL, PAINT); these concerns were addressed through the discussion and additional experiments. 

D. Reviewers unanimously voted to accept the paper.

E. Reviewers conducted new experiments (e.g., stochastic settings, MEDAL and PAINT baselines). Reviewers confirmed that authors answered questions.
Decision: Accept (poster)