When to Ask for Help: Proactive Interventions in Autonomous Reinforcement Learning
Authors: Annie Xie
Fahim Tajwar
Archit Sharma
Chelsea Finn
Venue: NeurIPS 2022 Accept (2022)

Abstract: A long-term goal of reinforcement learning is to design agents that can autonomously interact and learn in the world. A critical challenge to such autonomy is the presence of irreversible states which require external assistance to recover from, such as when a robot arm has pushed an object off of a table. While standard agents require constant monitoring to decide when to intervene, we aim to design proactive agents that can request human intervention only when needed. To this end, we propose an algorithm that efficiently learns to detect and avoid states that are irreversible, and proactively asks for help in case the agent does enter them. On a suite of continuous control environments with unknown irreversible states, we find that our algorithm exhibits better sample- and intervention-efficiency compared to existing methods.

When to Ask for Help: Proactive Interventions in Autonomous Reinforcement Learning
Annie Xie, Fahim Tajwar; Archit Sharma, Chelsea Finn
Department of Computer Science Stanford University
fanniexie, tajwar93, architsh, cbf cbfinn))sttaffrdd edu
Abstract
A long-term goal of reinforcement learning to design agents that can au- tonomously interact and learn in the world. A critical challenge such au- tonomy is the presence of irreversible states which require external assistance recover from, such as when robot has pushed an object off of table. While standard agents require constant monitoring to decide when to intervene, we aim design proactive agents that can request human intervention only when needed. To this end, we propose an algorithm that efficiently learns to detect and avoid states that are irreversible. and proactively asks for help in case the agent does enter them. On suite of continuous control environments with un- known irreversible states, we find that our algorithm exhibits better sample- and intervention-effiiiency compared to existing methods. Our code is publicly avail- able at www.itess.oogge.. com/
help please
autonomous Learning
external
intervention
Figure 1: Autonomous agents struggle make progress without external interventions when they stuck an irreversible state. Reinforcement learning agents therefore need active monitoring throughout training to detect and intervene when the agent reaches an irreversible state. Enabling the agents to detect inreversible states and proactively request for help can substantially reduce the human monitoring required for training agents.
Introduction
reinforcement learning (RL) agent should able to autonomously learn behavior by exploring in and interacting with its environment. However, in most realistic learning environments, there irreversible states from which the agent cannot recover on its own. For example, a robot arm can inadvertently push object off the table, such that external supervisor must return it back to the robot's workspace to continue the learning process. Current agents demand constant monitoring to decide when the agent enters irreversible state and therefore when intervene. In this work, aim to build greater autonomy into RI. agents by addressing this problem. In particular, we envision
equal contribution
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

proactive agents that can instead detect irreversible states, proactively request interventions when needed, and otherwise learn autonomously.
Prior works have studied autonomy in RI., aiming to minimize the number of human-provided resets the end of each episode, but generally assume the environment is fully reversible 19
[X]
K55
177. K4
Our work focuses on settings with potential irreversible states and algorithms to avoid such states, A related desiderata, however, arises in the safe RL setting; safe RL methods aim learn policies that minimize visits to unsafe states, and the developed approaches are designed to avoid those particular parts of the state space IC. 39 E 4T Prior safe RL algorithms assume that the agent is given a safety label on demand for every state it visits. In contrast, an autonomous agent may not know when it has reached an irreversible state (such as knocking an important object off a table), and an algorithm in this setting should instead learn to both detect and avoid such states, while minimizing queries about whether state is reversible.
With this in mind, we design our setup to provide help to the agent in two ways: through an environment reset or through the reversibility label of state. However, unlike in safe RL, we can reduce the labeling requirement with a simple observation: all states proceeding an irreversible state are irreversible, and all states preceding reversible state will be reversible. Based on this observation, we design scheme based on binary search to generate reversibility labels for a trajectory of length T using at most (log T) label queries, compared to the (T)) queries made by safe RL methods. We further reduce labeling burden by only querying labels in a large batch at the end of each extended episode, i.e. typically only after tens of thousands of steps. By combining this label efficient scheme with proactive requests for an intervention and batch of labels, we can enable agents to learn amidst irreversible states with a high degree of autonomy.
Concretely, we propose a framework for reversibility-waat autonomous RL, which we call proactive agent interventions (PAINT), that aims to minimize human monitoring and supervision required throughout training. First, we train a reversibility-awan Q-value function that penalizes visits to irreversible states. Second, the reversibility labels are generated by a label-efficient binary search routine, which makes at most a logarithmic number of queries in the length of the interaction with the environment. Finally, the labeled states can be used learn a classifier for predicting irreversible states, which can then be leveraged to proactively call for interventions. Our proposed framework PAINT can be used to adapt any value-based RI. algorithm, in both episodic and non-episodic settings, to learn with minimal and proactive interventions. We compare PAINT to prior methods for autonomous RL and safe RL on a suite of continuous control tasks, and find that PAINT exhibits both better sample- and intervention-efficiee object manipulation tasks, PAINT only requires around 100 interventions while training for 3 million steps, which is up to 15x fewer than those required by prior algorithms.
compared to existing methods. On challenging autonomous
2 Related Work
Deployment of many RL algorithms in physical contexts is challenging, because they fail to avoid undesirable states in the environment and require human providd resets between trials. Safe RL, reversibility- RL, and autonomous RL, which we review next, address parts of these problems.
Safe RL. The goal of our work is to learn to avoid irreversible states. Algorithms for safe RL also need to avoid regions of the state space, and achieve this by formulating a constrained optimization
Another class of algorithms construct saclding-basce policies that yield control to backup policy if following the Critically, however, all of these approaches assume that safety labels for each state can be queried freely at every time-step of training, whereas our objective is to minimize labeling requirements over training.
problem IC 39
37
50 or by assigning low rewards to unsafe states A3 41
learning policy leads to unsafe state [1 3
40
Reversibility-wwar RL. Reversibility and reachability have been studied in the context of RL to avoid actions that lead to irreversible states [24 25,, B1l [5] or, conversely, to guide exploration towards difficult-to-reach states 3K 5 Unlike prior work, our study of reversibility primarily focuses on the non-episodic setting to minimize the number of human interventions during learning. While prior methods are self-supervised, our experiments also find that our algorithm learns with significantly fewer interventions than prior methods by leveraging some binary reversibility labels.

---

Autonomous RL. Multiple prior works have also studied autonomy in RL, motivated by the fact that deployments of RL algorithmss on real robots often require numan-providee resets between To avoid the supervision needed for episodic resets, prior work has proposed to learn controllers to return to specific state distributions, such as the initial state [99 11] the uniform distribution over states [33 or demonstration states [50 adversarially learned distributions [47], or perriculum-based distributions
episodes 13 16 14
However, most work in the reset-free setting assumes the agent's M4
whereas we specifically tackle the setting where this is not the case. One notable exception is the Leave No Trace algorithm [[]], which checks whether the agent has successfully returned back to the initial state distribution and requests intervention otherwise. Our approach differs from Leave No Trace by requesting a reset based on the estimated reversibility of the state, which we find requires significantly fewer interventions in evaluation.
environment is reversible 300
35
Human-in-the-long learning. Learning from human feedback has enabled RI. agents to acquire complex skills that are difficult to encode in reward function 27 45 12 4. However, interactive
RL algorithms are often to difficult to scale as they rely on feedback at every time-step of training. More feedback-efficient algorithms have learned reward models from human-provided preferences [2] 26 which removes the need for constant feedback. Similarly, the interactive imitation learning learning has seen more query-efficient algorithms, which query expert actions based on the estimated risk or novelty of a visited state [51 28 21 20). While these algorithms augment the agent with human- human-provided preferences or expert actions, our approach leverages different mode of feedback, that is, reversibility labels for visited states.
38
3 Reinforcement Learning in Irreversible Environments
Consider a Markov decision process M = (S, Po: with state space S, action space A, transition dynamics P : S x A x S + [0, 1]. bounded reward function r : S A + [Rmin Rmax] initial state distribution Po: S + [0, 1] and discount factor 7 E [o, 1). In this work, we build on the 35 but we remove the assumption that the environment is reversible,
formalism of autonomous RL ie., the MDP is no longer strongly connected (see example in 2 Chapter 38] and description below). The environment is initialized at Su p and algorithm continually interacts with the environment till it requests the environment to be reset via an external intervention to state s6 ~ p. Specifically, an algorithm A {&s, aj 81+1; I' } M H (at, me) generates a sequence (so, ao $1,...) in M, mapping the states, actions, and rewards seen till time t- 1 to action at E AA {Areset) ). and the current guess at the optimal policy N. : S x A + [0, oo). Here, Greset is a special action the agent can execute reset the environment through extrinsic interventions, i.e. P (- 8, Gzeset) = po(.).
MDP is strongly connected if for all pairs of states 81,8, E S, there exists policy π such that 8j has a non-zero probability of being visited when executing the policy n from state 8;. This assumption can easily be violated in practice, for example, when a robot arm pushes object out of its reach. At an abstract level, the agent has transitioned into a component of MDP that is not connected with the high reward states, and thus cannot continue making progress, as visualized in Figure 2 The agent can invoke an extrinsic agent (such as a human supervisor) through Grezet; and the extrinsic agent can reset the environment to a state from the initial state distribution. For example, the human supervisor can reset the object to the initial state, which is within the reach of the robot arm. For every state E S, define R, S - {0, 1} the indicator whether the state is in the same component as the initial state distribution. State s is defined to be reversible if (s) 1, and irreversible if R, = We assume that the R2 is unknown, but can be queried for a state s.
S
Example of an MDP with irreversible states (in red). The agent starts in the state 'S' and its goal is to reach the state 'G', which are connected.
Figure 2:
While we do not assume that the MDP is strongly connected, we assume that the states visited by the optimal policy are in the same connected component as the initial state distribution. Under this assumption, we can design agents that can autonomously practice the task many times. Otherwise, the environment would need to be reset after every successful trial of the task.
The objective is to learn optimal policy x' E arg max, J(r) arg max, E E 2*r(st, 0t)] Note, J(m) is approximated by computing the return when the policy is rolled out from so Po- Algorithms are typically evaluated on the sample efficiency, that is minimizing
3

---

However, since we care about minimizing the human supervision required and resetting the environment can entail expensive human supervision, we will primarily evaluate algorithms on intervention-elliienncyy defined as I((A) Ex= [(x)) (TT). where me is the policy learned after l interventions.
Preliminaries
Episodic settings reset the environment to state from the initial state distribution after every trial, typically after every few hundred steps of interaction with the environment. Such frequent resetting of the environment entails an extensive amount of external interventions, typically from human. Prior works on autonomous RL have sought to reduce the supervision required for resetting the environments by learning a backward policy that resets the environment [1]
Meaningfully improving the autonomy of RL in irreversible environments requires us to curb the requirement of episodic resets first. While our proposed framework is compatible with any autonomous RL algorithm, we describe MEDAL 36 which will be used in our experiments.
52 34
MEDAL learns a forward policy T and backward policy alternately executed for a fixed number of steps in the environment. The forward policy maximizes the conventional cumulative task reward, that is E IEt= 7'r(st, de)], and the backward policy minimizes the Jensen-Shannow divergence Das (p)(s) ll (s)) between the marginal state distribution of the backward policy p and the state distribution of the optimal forward policy ~", approximated by a small number of expert demonstrations. Thus, the backward policy keeps the agent close to the demonstration states, allowing the forward agent to try the task from a mix of easy and hard initial states. The proposed divergence can be minimized via the objective mill,, maxe Es-p- log C(s)] + Ex-pp [oog(1 - C(s))] where C' : S + [0, [] is a classifier that maximizes the log-probability of states visited in the forward demonstrations, and minimizes the probability of the states visited by the backward policy. The optimization problem for the backward policy can be written as RL problem:
min Bh [log (1 C(s)) = max E 1
Σ 7 log 1 C((s))
where Ty maximizes the reward function r(s, a) log (1 C(s)). Correspondingly, C(s) is trained to discriminate between states visited by the backward policy and the demonstrations.
5 Proactive Agent Interventions for Autonomous Reinforcement Learning
To minimize human monitoring and supervision when an agent is learning in an environment with irreversible states, the agent needs to (a) learn to avoid irreversible states over the course of training and (b) learn to detect and request an intervention whenever the agent is stuck. For the former, we first describe a simple modification to the reward function to explicitly penalize visitation of reversibility irreversible states in Section 5.T However, such a modification requires the knowledge of the visited states, which is not known apriori. We learn a classifier to estimate reversibility, proposing label-efficient algorithm to query reversibility labels of visited states Section 5.2 Since both the dynamics and the set of irreversible states are unknown apriori, the agent will inevitably still visit irreversible states as a part of the exploration. To ensure that a human does not have to monitor the agent throughout training, the agent should have mechanism to decide and request for an intervention. We discuss such a mechanism Section 5.3 Finally, we put together all these components in Section 5.4 for our proposed framework Proactive Agent INTerventions (PAINT), an overview which is given in Figure B
5.1 Penalizing Visitation of Trreversible States
Our goal is to penalize visitation of irreversible states by ensuring all actions leading to irreversible states are 'worse" than those leading to reversible states. To this end, we adapt the reward-penalty framework from safe RL An for learning in the presence of irreversible states. For transition (s, a, s'), consider a surrogate reward function F:
(r(s,a), Rp((") 1
F(8, a)
Rmin -,, E, Rp's')

---

request intervention
autonomous
Algorithm 1: Reversibility Labeling via Binary Search
Agent
input: {s; [ unlabeled trajectory
while len(7) do
train reversibility-
m 6- [1)2]] get midpoint
reset
samples
aware Q-function
query for midpoint
if R,( = 1 then
Human
label first half reversible and query
Buffer
reversibility labels
for second half
label {s: Jē=o
Figure 3: Overview our framework PAINT for minimiz-
4 } h=m+11
ing human monitoring and supervision when learning in the presence of irrevesiibee states. The agent proactively requests interventions, freeing the human from active monitoring of training. When intervention is requested. the human sets the environment and provides reversibility labels the latest experience since the previous intervention.
else
// label second half inreversible and
query for first half
label {s; fcmmm O:
< {a)
Whenever the state s' is reversible state, the agent gets the environment reward. Otherwise if it has entered an irreversible, it gets constant reward Rmin that is worse than any reward given out by the environment. Whenever an agent enters an irreversible state, will continue to remain in an irreversible state and get constant reward of Rmin e. Therefore, the Q-value whenever Rp(s') = 0 given by:
Q (s
Σ y /i(ss at) so s, au
(Runin c)
ΣΣ
00
2 1 1-7
Rmin
This observation allows bypass the need perform Bellman backups irreversible states, and instead directly regress to the Q-value. More specifically, we can rewrite the loss function for the Q-value function as, {(Q) ([ss,,x".))> (Q(s,a) B (=Q(s,a)) a)], where the application of Bellman backup operator 8 (s,, a) can be expanded as:
r(s,, + 7B (11))) Q(s, Rp(s')= = Rp(s')= =
()) = 1 (Rainn <) (1 7),
(3)
= RR'(s') (r(s, a) + - ( Qw) + (1 (1- (Rp(sss)) Rmin
(4)
Here, denotes the replay buffer, Q denotes the use target networks commonly used Q-learning algorithms to stabilize training when using neural networks as function approximators [29]. This surrogate reward function and the modified Bellman operator can be used for any vasue-based RL algorithm in both episodic and autonomous RL settings. The hyperparameter e controls how aggressively the agent is penalized for visiting irreversible states. In general, Q-values for actions leading to irreversible states will be lower than those keeping the agent amongst reversible states, encouraging the policies to visit irreversible states fewer times over the course of training. More details and proofs can be found in Appendix A.I.I
5.2 Estimating Reversibility
In general, R2 is not known apriori and will have to be estimated. We define R, S [0, as an estimator of the reversibility of state E S. We can then define an empirical Bellman backup operator 8" from equation by replacing R, with the estimator R.p:
R,(s') (r(s,a) + " + (1- (1-Rp(8')) Rmin
T Q(s a) = -Es=P()).))
Analogous to 8= we can define the empirical Bellman optimality operator 8 (Eq 12 for value iteration when using R,, The following theorem bounds the suboptimality of the policy learned by value iteration under 8":
5

---

Theorem 5.1. Let x² denote the optimal policy and Q" denote the corresponding optimal Q-value function. Let He denote the optimal policy returned by empirical Bellman optimality operator B
Assuming R Rp S
26 (8max Rminn + +c) Q*1(s,0 (s, Q*())))
(1 (1-9) -)
for all (s,a) E SSx A.
The proof and related discussion can be found in Appendix A.1.2 The result guarantees that closer R.p R under the oc-norm, the closer #" is n².. To this end, we propose to learn R, by minimizing (s) log (s) + (1 - Rs())) log(1 R,(ss))
the binary cross-entropy loss ((RR,) = - SDD (R)(s)
where the states D represent the states visited by the agent.
Minimizing ((R requires the reversibility labels Rs(ss for ~ D. Since labeling requires supervi- sion, critical query R2 efficiently. Given a trajectory of states T = (su.
sr). naive approach would be query the labels (x; for all states 84, leading to O(T) queries per trajectory. However, observe that we have the following properties: (a) all states following an irreversible state will be irreversible and (b) all states preceding reversible state will be reversible. It follows from these properties that every trajectory can be split into a reversible segment T, = (so: 81.....9 88))
and irreversible segment T~r ($k+1,.... ST), where the irreversible segment T~y can be empty potentially. Identifying 5k+1, the first irreversible state, generates the labels for the entire trajectory automatically. Fortunately, we can construct scheme based on binary search to identify 8 in (log 7) queries: SK+11 occurs after the midpoint of the trajectory if the midpoint is reversible, otherwise it occurs before it. The pseudocode for this routine is given in Alg
The total number of labels required would be O (N log |mlma)), where N is the number of trajectories in the replay buffer D and the |T] denotes the maximum length of the trajectory. This represents a reduction in label requirement of O (N |mlma) by prior safe RL methods. Furthermore, agent trains to avoid irreversible states, resulting in fewer and longer trajectories over the course of training. Thus, labeling reduces over time because the labels required is linear in N and logarithmic in [T] |mxx
5.3 Proactive Interventions
Despite trying to avoid irreversible states via reward penalties, an agent will inevitably encounter some irreversible states due exploratory behaviors. It is critical that the agent proactively asks for help in such situations, so that a human does not need to constantly monitor the training process. More specifically, an agent should request intervention when is an irreversible state. Since (s) is not available, the agent again needs to estimate the reversibility of the state. It is natural to reuse the learned reversibility estimator R, for this purpose. We propose the following rule: the agent executes Greset whenever the reversibility classifier's prediction falls below 0.5, i.e., Rp(s) < 0.5.
5.4 Putting it Together
Algorithm 2: PAINT
With the key components in place, we summa- rize our proposed framework. High-level pseu- docode is given in Alg. [ and more detailed A.2 A.2
input: P: // agent, params abstracted away
initialize R,p, DD; // classifier, replay buffer
while not dome do
$ Pai // reset environment
pseudocode is deferred to Appendix
I/ continue till classifier detects irreversibility
PAINT can modify any value-based RL algo- rithm, in both episodic and autonomous settings. This description and Alg. 2 focus on the latter setting, although adapting it to the episodic set- ting straightforward. The agent's interaction with the environment consists of a sequence of trials that end whenever the environment is re- set to a state ~ Pa. During each trial, the agent operates autonomously, and the Bellman update for the critic is modified according to the empirical Bellman backup 18. Whenever the
while RR((s) > 0.5 do
// step in the environment
a ~ P(s), ~ P s,
// update replay buffer and agent
update 2, P:
If optionally explore environment
for explore steps do
a ~ unif((A), s ~ P(- 8, a, update D;
// reversibilitty labels via binary search update reversibility labels in D; // train classifier all labeled data, new and old
train R,p

---

agent
Figure 4: A subset of our evaluation tasks: Tabletop Manipulation, Peg Insertion, and Half-Cheent Cheetah Velocity, Irreversible states in the first two environments are when the agent drops the object outside the red boundary (leff) and off of the table (middle)) The cheetah is in irreversible state whenever it is flipped over (right).
reversibility classifier R, 0.5, parameterized
as a neural network, the agent requests an intervention. The agent can execute fixed number of exploration steps after requesting an intervention and before the intervention is performed. Whenever the classifier predicts an irreversible state correctly, these exploration steps can help the agent gather more information about irreversible states. At the time of the intervention, all new states visited since the previous intervention are labeled for reversibility via Algorithm | Finally, the reversibility classifier is trained on all the labeled data before the environment reset state ~ po for next trial. Full implementation details can be found in Appendix A.4
The agent is provided reversibility labels only when the external reset provided. This simplifies supervision as the human can reset the environment and provide labels at the same time. This means the replay buffer D will contain states with and without reversibility labels, since states from the current trial will not yet have labels. We use Eq. g for states that have reversibility labels to avoid emors from the classifier affecting the critic update and use 8 for those that do not have labels.
Experiments
We design several experiments to study the efficiency of our algorithm in terms of the required number of reset interventions and number of queried reversibility labels. Our code and videos of our results are at: https://sitte wwwesitessgoogleeee.eeeee wtt.:/oo//iiiieee http:///....o////////aaaaeeeeete intereeeetieiinneeeettiiin...
6.1 Experimental Setup
Environments. illustrate the wide applicability of our method, we design environments that represent three distinct RI. setups: episodic, wawwardbacckward, and continuing.
Maze (episodic). A 2D continuous maze environment with trenches, which represent groups of connected irreversible states. The agent can fall into a trench, and once entered, it can roam freely within the trench but cannot leave it without an environment reset. In this task, resets are infrequently provided to the agent after 500 time-steps.
Tabletop Organization
The agent must grasp the mug and put down on one of the four goal positions. Dropping the mug outside of the red boundary is irreversible. 35 (forward-backward) The agent must insert peg into the goal but can potentially drop it off the table, which is irreversible.
Peg Insertion
Half-Cheetah Vel [9] (continuing). The agent must run at the specified target velocity, which
changes every 500 steps, and can potentially flip over onto its back, which is irreversible.
We visualize and fully describe each environment in Fig. 4 and in Appendix A.3 respectively.
Comparisons. In the episodic and continuing settings, we consider safe RL baselines that rely on
reversibility labels at every time-step of training.
Safe Model-Based Policy Optimization (SMBPO)
This comparison implements the modi- E in Section 5.1 using the true reversibility labels.
fied Bellman operator defined in Eqn.
Safety Q-functions for RL (SQRL) k77 A safe RL. method that trains safety critic, which estimates the future probability of entering irreversible state for constraedd policy.
7

---

Maze
1.0
VF
Task
Method
Labels
SMBPO/SQRL
200K
0.6
Maze
PAINT (Ours)
3260 + 12
0.4
Tabletop
PAINT (Ours)
1021 + 69
0.8
Peg Insertion
PAINT (Ours)
2083 1: 149
25 50 75 100 125 100 175 200
Number Interventions
SMBPO Term.
3M
SAC
SQIIL
DAINT (Ound)
Cheetah
SMIPO
PAINT (Ours)
8748 + 3762
Figure 5: (left) Task success versus interventions. Shaded regions denote the standard error over 5 seeds. (right) Predictions generated by our reversibility classi- fier, where the purple region is predicted to be reversible.
Figure 6: Number of queried reversibility labels. For method, average the number of labels used across 5 seeds and report the standard error.
In the forward- backdddd setting, we consider methods designed for the autonomous learning setup. These methods do not require any reversibilityy labels. Hence, our goal here is to compare the reset-efficiency of our method to prior work.
Leave No Trace (LNT) [.] An autonomous RL method that jointly trains a forward policy and reset policy. When the reset policy fails to return to the initial state, the agent requests a reset. Matching Expert Distributions for Autonomous Learning (MEDAL) [36]. This method trains reset policy that returns the distribution of demonstration states provided for the forward policy. MEDAL does not have built-in intervention rule.
In all tasks, we compare recently proposed reversibility- aware RL method, Reversihility-Awan Exploration (RAE) 1 which does not require any reversibility labels. It instead trains a self- supervised reversibility estimator to predict whether a state transition (s, is more likely than the reverse (8, s). We augment RAE with an intervention rule, similar to our method, defined in terms of predictions from its self-supervised classifier. In the borkard-backward setting, train both the forward and backward policies with RAE. Finally, we also evaluate Episodic RL, which represents the typical RL setup with frequent resets and thus provides an upper-bound on task success. In Appendix A.4. we provide full implementation details of each comparison, and in Appendix A.5 we discuss the set of assumptions made by each comparison.
6.2
Main Results
In Fig. 5 (left) and Fig. 7 we plot the task success versus the number of interventions in the 4 tasks. For methods that use reversibility labels, report the total number of labels queried in Table G
Maze. While the safe RL methods, SMBPO and SQRL, require reversibility labels at every stee-step. our approach PAINT only requires on average 3260 queries to label all 200K states visited. In Fig, (right), we visualize predictions from our reversibility classifier at the end of training, where zero predicts 'reversible' and one predicts 'irreversible.. The classifier correctly identifies the path that leads to the goal as reversible. Interestingly, classifies all other regions as stuck states, including the states that are reversible. Because these states are irrelevant to the task, however, classifying them as irreversible, and therefore to be avoided, advantageous to our policy as reduces its area of exploration.
More complex domains. In the Tabletop Organization and Peg Insertion tasks, each agent is reset every 200K and 100K time-steps, per the EARL benchmark [35]. However, we allow agents to request earlier resets, and under this setting, we compare PAINT to other methods that implement intervention rules. Compared to Leave No Trace and Reversibility-Awar requires significantly fewer resets 80 and 124 resets respectively. which corresponds to roughly one intervention for every 25K steps. Importantly, the number of interventions plateaus as training progresses, and the agent requires fewer and fewer resets over time (see Appendix A.6 for additional plots of number of interventions versus time-steps). The exception is MEDAL (green segment near the origin), which not equipped with an early termination rule and so only uses 10 interventions total. However, it also fails to make meaningful progress the task with few resets.
Exploration, PAINT
8

---

Tabletop Organization
Peg Insertion
Half-Cheetah
1.0
1000
Task Success
Success
Episode Reward
1200
0.6-
0.6
1000
0.4
0.4
800
0.0
0.0
400
250
500
750
1000
1250
1500
250
500
750
1000
1250
1500
250
500
750
1000 1250 1500
Number of Interventions
Number Interventions
Number of Interveations
LNT
PAINT
MEDAL
RAF
SMIPPO w Terr
PAINT (Ours)
Figure 7: Task success versus interventions averaged seeds. Methods with stronger assumptions, i.e. SAC resets every steps SMBPO requires labels at every time-step. are dotted. Note the short green segment near the origin representing MEDAL.
Early Termination
Demonstrations
Average Task Suuneess
0.45
0.35
0.25
0.15
0.05
25 50 75 100
125 150 175 200
5
10
50
100
Number of Interventions
Number of Demonstrations
Ours
Ours w/o Early Termination
SAC
SQRL
SMBPO
Figure 8: (leff) After removing the carly termination condition, which initiates random exploration, we find that PAINT learns less efficiently. (right) Varying the number of demonstrations suggests that PAINT and SMBPO are robust to the amount of available demonstrations.
On the continuing Half-Cheetah task, agents do not receive any resets, unless specifically requested. Here, we compare PAINT to SMPBO with early termination, an oracle version of our method, which assumes that reversibility labels available at every time-step and immediately requests an intervention if the agent is flipped over. PAINT converges to its final performance after around 750 resets, on par with the number of resets required by SMPBO with early termination. On the other hand, a standard episodic RL agent, which receives resets at every 2K steps, and RAE, which trains a self- superviedd classifier, learn significantly slower with respect to the number of interventions.
6.3 Ablations and Sensitivity Analysis
Early termination. In the episodic Maze setting, our algorithm switches to uniform-random policy for the remainder of the episode the termination condition is met. In Fig. R (left), we plot the performance without early termination, i.e., rumning the agent policy for the full episode. Taking random explorations, after the agent believes has entered an irreversible state, significantly helps method, as it increases the number and diversity of irreversible states the agent has seen.
Varying the number of demonstrations. Our method leverages demonstrations in subset of environments. While we provide these demonstrations to all comparisons well, want to study how much our method relies on them. We plot the average task success during training versus number of demonstrations in Fig. (right). While PAINT and SMBPO are robust to the amount, alternative methods tend to achieve significantly lower success when given fewer demonstrations.
7 Discussion
In this work, we sought to build greater autonomy into RL agents, particularly in irreversible environments. We proposed an algorithm, PAINT, that learns detect and avoid irreversible states, and proactively requests an intervention when in an irreversible state. PAINT leverages reversibility

---

labels to learn to identify inreversible states more quickly, and improves upon existing methods on range of learning setups terms of task success, reset-efficiency, and label-efficiency.
Despite these improvements, PAINT has multiple important limitations. In environments where invevesiible states are not encountered until further into training, the reversibility classifier may produce false positives which would significantly delay the next intervention. Further, while PAINT is far more label-efficient than prior safe RL methods, it still requires around thousands of reversibility labels. We expect that this limitation may be mitigated with more sophisticated querying strategies, e.g. that take into account the classifier's 's confidence. Finally, we hope that future work can validate the ability for reversibility aware techniques to improve the autonomy of real robotic learning systems.
Acknowledgments and Disclosure of Funding
AX was supported by an NSF Graduate Research Fellowship. The work was also supported by funding from Google. Schmidt Futures, and ONR grants 100014-21-1-2688 and N00014-20- 22-1-22775 The authors would also like to thank members of the IRIS Lab for helpful feedback on an early version of this paper.
References
[1] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning, pages 22-31. PMLR, 2017.
[2] Riad Akrour, Marc Schoenauer, and Michele Sebag. Preference-basee policy learning. In Joint European Conference on Machine Learning and Knowledge Discovery in Databuses, pages 12-27. Springer, 2011.
[3] Mohammed Alshickh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, and Ufuk Topeu. Safe reinforcement learning via shielding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[4] Christian Arzate Cruz and Takeo Igarashi. A survey on interactive reinforcement learning: design principles and open challenges. In Proceedings of the 2020 ACM designing interactive systems conference, pages 1195-1209, 2020.
[5] Adrià Puigdomênech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski. Olivier Tieleman, Martín Arjovsky, Alexander Pritzel, Andew Bolt, et al. Never give up: Learning directed exploration strategies. arXiv preprint arXiv:2002.06038. 2020.
[6] Osbert Bastani, Shuo Li, and Anton Xu. Safe reinforcement learning via statistical model predictive shielding. In Robotics: Science and Systems, 2021.
[7] Homanga Bharadhwaj. Aviral Kumar, Nicholas Rhinehart, Sergey Levine, Florian Shkurti, and Animesh Garg. Conservative safety critics for exploration. arXiv preprint arXiv:2010.14497,
2020.
[8] Erdem Biyik and Dorsa Sadigh. Batch active preference-basee based learning of reward functions. In Conference on robot learning, pages 519-528. PMLR, 2018.
[9] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv: 160660156000,,, 2016.
[10] Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk- cisktconstrained reinforcement learning with percentile risk criteria. The Journal of Machine Learning Research, 18(1 (((6))60061220 2017.
[11] Benjamin Eysenbach, Shixiang Gu. Julian Ibarz, and Sergey Levine. Leave no trace: Learning to reset for safe and autonomous reinforcement learning. arXiv preprint arXiv: / 171000782, 2017.
[12] Taylor A Kessler Faulkner, Elaine Schaertl Short, and Andrea L. Thomaz. Interactive reinforce- ment learning with inaccurate feedback. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 7498-7504. IEEF, 2020.
[13] Chelsea Finn, Xin Yu Tan, Yan Duan, Trevor Darrell, Sergey Levine, and Pieter Abbeel. Deep spatial autoencoders for visuomotor learning. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 512-519. IEEE, 2016.
10

---

[14] Ali Ghadirzadeh, Atsuto Maki, Danica Kragic, and Marten Björkman. Deep predictive policy training using reinforcement learning. In 2017 JEEE/RS/ International Conferree on Intelligent Robots and Systems (IROS), pages 2351-2355 IEEE, 2017.
[15) Nathan Grinsztajn, Johan Ferret, Olivier Pietquin, Matthieu Geist, et al. There is no turning back: self-supervised approach for reversibility-awe Neural Information Processing Systems, 34, 2021.
reinforcement learning. Advances in
[16] Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In 2017 IEEE international conference robotics and automation (ICRA), pages 3389-3396. IEEE, 2017.
[17] Abhishek Gupta, Justin Yu, Tony Zhao, Vikash Kumar, Aaron Rovinsky, Kelvin Xu, Thomas Devlin, and Sergey Levine. Reset-free reinforcement learning via multi-task learning: Learning dexterous manipulation behaviors without human intervention. ArXiv, abs/2104.1122) 12033, 2021.
[18] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference Machine Learning, pages
PMLR, 2018.
[19] Weiqiao Han, Sergey Levine, and Pieter Abbeel. Learning compound multi-step controllers under unknown dynamics. In 2015 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 6435-6442. IEEE, 2015.
[20] Ryan Hoque, Ashwin Balakrishna, Ellen Novoseller, Albert Wilcox, Daniel S Brown, and Ken Goldberg. Thriftydagger: Budget-aware novelty and risk gating for interactive imitation learning. arXiv preprint 20X.0:2 2021.
[21] Ryan Hoque, Ashwin Balakrishna, Carl Putterman, Michael Luo, Daniel S Brown, Daniel Seita, Brijen Thananjeyan, Ellen Novoseller, and Ken Goldberg. Lazydagger: Reducing context switching in interactive imitation learning. In 202/ IEEE 17th International Conference on Automation Science and Engineering (CASE), pages 502-509. IEEE, 2021.
[22] W Bradley Knox and Peter Stone. Interactively shaping agents via human reinforcement: The tamer framework. In Proceedings of the fifth international conference on Knowledge capture, pages 9-16, 2009.
[23] Victoria Krakovna, Laurent Orseau, Ramana Kumar, Miljan Martic, and Shane Legg. Penalizing side effects using stepwise relative reachability. arXiv preprint 1rXiv::806.000 186, 2018
[24] Maarja Kruusmaa, Yuri Gavshin, and Adam Eppendahl. Don't do things you can't undo: reversibility models for generating safe behaviours. In Proceedings 2007 IEEE International Conference on Robotics and Automation, pages 1134-1139, IEEE, 2007.
[25] Tor Lattimore and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020.
[26] Kimin Lee, Laura Smith, and Picter Abbeel. Pebble: Feedback-efficiien interactive rein- forcement learning via relabeling experience and unsupervised pre-training. arXiv preprint arXiv:2106.00 050911 2021.
[27] James MacGlashan, Mark K Ho. Robert Loftin, Bei Peng, Guan Wang, David Roberts, Matthew E Taylor, and Michael L. Littman. Interactive learning from policy-dependens human feedback. In International Conference on Machine Learning, pages 2285 2294. PMLR, 2017.
[28] Kunal Menda, Katherine Triggs-Campbele and Mykel J Kochenderfer. Ensembledagger. bayesian approach to safe imitation learning. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages
IEEE, 2019.
[29] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-53) 2015.
[30] Teodor Mihai Moldovan and Pieter Abbeel. Safe exploration markov decision processes.
arXiv preprini /rXiv:/205.00 18005,44..2222 2012.
[31] Nasim Rahaman, Steffen Wolf, Anirudh Goyal, Roman Remme, and Yoshua Bengio. Learning the arrow of time for problems in reinforcement learning. 2020.
[32] Dorsa Sadigh, Anca Dragan, Shankar Sastry, and Sanjit A Seshia, Active preference-based learning of reward functions. 2017.
11

---

[33] Nikolay Savinov, Anton Raichuk, Raphaël Marinier, Damien Vincent, Mare Pollefeys. Tim- othy Lillicrap, and Sylvain Gelly. Episodic curiosity through reachability. arXiv preprint 180.022/1,.02274, 2018.
[34] Archit Sharma. Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Au- tonomous reinforcement learning via subgoal curricula. Advances in Neural Information Processing Systems, 34, 2021.
[35] Archit Sharma, Kelvin Xu, Nikhil Sardana, Abhishek Gupta, Karol Hausman, Sergey Levine, and Chelsea Finn. Autonomous reinforcement learning: Formalism and benchmarking. arXiv preprint arXiv:2 /2.09605, 2021.
[36] Archit Sharma, Rehaan Ahmad, and Chelsea Finn. A state-distributiion matching approach to non-episodic reinforcement learning. arXiv preprint arXiv:2205.052212 2022.
[37] Krishnan Srinivasan, Benjamin Eysenbach, Sehoon Ha, Jie Tan, and Chelsea Finn. Learning to be safe: Deep rl with a safety critic. arXiv preprint arXiv:2010. 14603, 2020.
[38] Hiroaki Sugiyama, Toyomi Meguro, and Yasuhiro Minami. Preference-learning based inverse reinforcement learning for dialog control. In Thirteenth Annual Conference of the International Speech Communication Association, 2012.
[39] Chen Tessler, Daniel Mankowitz, and Shie Mannor. Reward constrained policy optimization.
arXiv preprini arXiv: /805 / 1074, 2018.
[40] Brijen Thananjeyan, Ashwin Balakrishna, Suraj Nair, Michael Luo, Krishnan Srinivasan, Minho Hwang, Joseph E Gonzalez, Julian Ibarz, Chelsea Finn, and Ken Goldberg. Recovery rl: Safe reinforcement learning with learned recovery zones, arXiv preprint arXiv:2010.1592 2020.
[41] Garrett Thomas, Yuping Luo, and Tengyu Ma. Safe reinforcement learning by imagining the near future. Advances in Neural Information Processing Systems, 34, 2021.
[42] Matteo Turchetta, Andrey Kolobov, Shital Shah, Andreas Krause, and Alekh Agarwal. Safe reinforcement learning via curriculum induction. Advances in Neural Information Processing 2020.
Systems,
[43] Nolan C Wagener, Byron Boots, and Ching-An Cheng. Safe reinforcement learning using advantage-base based intervention. In International Conference on Machine Learning, pages 10630-
10640. PMLR, 2021.
[44] Xiaofei Wang. Kimin Lee, Kourosh Hakhamaneshi, Pieter Abbeel, and Michael Laskin. Skill preferences: Learning to extract and execute robotic skills from human feedback. In Conference on Robot Learning, pages 1259-1268. PMLR, 2022.
[45] Zhaodong Wang and Matthew E Taylor. Interactive reinforcement learning with dynamic reuse of prior knowledge from human/agent's 's demonstration. arXiv preprint arXiv: 04400, 2018.
[46] Christian Wirth and Johannes Fürnkranz. Preference- based reinforcement learning: prelim- inary survey, In Proceedings of the ECML/PKDD-13 Workshop on Reinforcement Learning from Generalized Feedback: Beyond Numeric Rewards. Citeseer, 2013.
[47] Kelvin Xu, Siddharth Verma, Chelsea Finn, and Sergey Levine. Continual learning of control primitives: Skill discovery via reset-games. ArXiv, abs/201 1. .05286, 2020.
[48] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved fata-augmented reinforcement learning. arXiv preprint arXtv:2107.09645. 2021.
[49] Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regu- larizing deep reinforcement learning from pixels. In International Conference on Learning Representations, 2021. URL. htttttt::::://reetttte
[50] Moritz, A Zanger, Karam Daaboul, and Marius Zöllner. Safe continuous control with con- strained model-based policy optimization. 2021 JEFF/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3512-3519. IEEF, 2021.
[51] Jiakai Zhang and Kyunghyun Cho. Query-efficient imitation learning for end-to-end autonomous driving. arXiv preprint arXiv: 1605.0645, 2016.
[52] Henry Zhu, Abhishek Gupta, Aravind Rajeswaran, Sergey Levine, and Vikash Kumar. Dexterous manipulation with deep reinforcement learning: Efficient, general, and low-cost. In 2019 International Conference on Robotics and Automation (ICRA), pages 3651-3657. IEEE, 2019.
12

---

[53] Henry Zhu, Justin Yu, Abhishek Gupta, Dhruv Shah, Kristian Hartikainen, Avi Singh, Vikash Kumar, and Sergey Levine. The ingredients of real-world robotic reinforcement learning. arXiv preprint arXiv:2004. 12570, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix A.1
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "Rs40p1OvtHA", "reviewer": "Reviewer_pQyz", "summary": "The paper proposes and uses the observation that states preceding a reversible state are reversible and states succeeding an irreversible state are irreversible. Leveraging this observation, the paper proposes an approach that predicts and tries to avoid irreversible states that makes is more efficient that than traditional safe-RL methods. The proposed approach has been validated on different RL setups and shown to be better than baseline approaches.", "questions": "1. Why is the font size changed during the introduction of the PAINT acronym?\n2. Why is it sufficient to only rely on the states preceding a reversible state and states following an irreversible state?\n3. How do the safe RL approaches perform if they use the observation regarding reversible and irreversible states proposed in the present study?\n", "limitations": "Yes, the conclusion describes a couple of the limitations of the proposed method. It might be worth conjecturing how PAINT might function on shared autonomy tasks.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. Work is related to a popular subtopic in the field.\n2. Videos of experimental results are available online.\n3. Decent coverage of ablation studies and limitations of current work.\n\n\nGrowth opportunities\n1. Some thresholds, such as R_{p} appear to be arbitrary.\n2. The form of the reversibility classifier is unclear. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "O82-R6Rl24-", "reviewer": "Reviewer_EVu5", "summary": "This paper presents an approach to reduce the number of user interactions necessary to reset the underlying system in a value function based reinforcement learning task with states that the system can not reset itself. To achieve this the approach addresses 2 main issues, namely i) reducing the likelihood that the system will enter an irreversible state through a change in the reward function, ii) reducing the number of times that a user has to provide information about the state by identifying relevant candidates to find the first irreversible state.\nThe resulting approach can be used in the context of any value-function\nbased RL algorithm and the paper presents performance comparisons with other methods (although those are not exactly designed for the same problem).", "questions": "The querying method for label information used in this paper uses a binary search based criterion to elicit label information to build the classifier for irreversible states. Why does the certainty in the prediction not influence whether a label query is made ?\n\nIn the performance comparisons, the proposed algorithm seems to end up with lower overall success than some comparison methods although with convergence with lower interaction numbers. How does this trade off play into decisions ?\n", "limitations": "The paper discusses limitations of the approach in the last section and touches on the most important aspects. A little more discussions on the approach’s impact on other performance measures would have been useful.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nThe paper addresses a problem in autonomous learning and human-machine  collaboration that has potential applications in a number of domains. It is well written and clearly introduces the concepts as well as the formalism where it builds on a reward function modification and a binary search-based strategy to guide the learning and the querying process.\nThe biggest weaknesses of the paper are in the somewhat limited amount of overall impact and in the experimental evaluation where results are compared with algorithms that were not directly designed with the same objectives in mind. It would have been good to see more discussion on the different assumptions these algorithms make.\nAnother issue is in Figure 7 where for the left two graphs the legend indicates 5 curves but only 4 seem to be visible.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "DmQpAL8fqn2", "reviewer": "Reviewer_pWgT", "summary": "This paper tackles the problem of training an RL system in an environment where not all actions are reversible, and therefore the environment must be reset before further training can take place. The approach learns a classifier which labels states with a reversibility label R -> [0,1] and only requests an intervention to reset the environment when this passes a threshold. The classifier training exploits the fact that irreversibility is deterministic once an irreversible state has been reached (i.e. all future states are irreversible too). The approach is evaluated on suitably complex domains, and demonstrates that it requests fewer resets than competitor methods.", "questions": "1. How would system performance change if the threshold of R>=0.5 was varied.\n2. Why does task performance not improve to levels of the baselines. \n3. How easy or hard is it to label and the reversibility of states in control tasks like the peg in hole? Is the object observable once it falls?\n4. How does uncertainty impact the problem, e.g. if an action has only some probability of leading to an irreversible state.\n", "limitations": "Some limitations are mentioned, and there are no obvious negative social implications. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall I think the paper well written. I like that the approach is straightforward and clearly justified. It leverages existing work (MEDAL) and extends it clearly, adding a new capability. The system is evaluated on domains with well-justified irreversibility, and on these domains the results demonstrate that the proposed method reduces the number of resets requested.\n\n\nOn the negative side, I was disappointed to see that there was no exploration of the threshold used to trigger the resets. Setting this to 0.5 feels like a reasonable thing to do, but I imagine your results depend on this parameter in interesting ways. Similarly, the idea that the agent can execute a fixed number of exploration steps after a reset occurs seems to be thrown in rather late in the paper and not sufficiently justified or explored. I could imagine settings (e.g. maze with all trenches connected) where a single failure could allow all mapping of irreversible states - and this could be a bug or a feature. \n\nI also would have liked to have seen more discussion on the task metrics in the evaluation. My reading of the results is that PAINT always produces better performance with fewer resets, but then plateaus and does not ask for further resets. This seems to hinder task performance. I understand that this is part of the motivation of the method (fewer resets/reversibility labels), but being unable to obtain competitive task performance seems like a major weakness, particularly if the method is terminating early. \n\nI like the title, but I think it's misleading. In particular, there are two forms of \"help\": reversibility labelling and environment resetting. I think the title starts the paper by blurring these ideas, and it doesn't recover totally (even though both types of help are delivered at the same time). ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept