Intelligent Switching for Reset-Free RL
Authors: Darshan Patil
Janarthanan Rajendran
Glen Berseth
Sarath Chandar
Venue: ICLR 2024 poster (2024)
Keywords: Reset-Free RL
Primary area: reinforcement learning
TL;DR: Intelligently switching between controllers leads to state-of-the-art performance on Reset-Free RL.

Abstract: In the real world, the strong episode resetting mechanisms that are needed to train
agents in simulation are unavailable. The resetting assumption limits the potential
of reinforcement learning in the real world, as providing resets to an agent usually
requires the creation of additional handcrafted mechanisms or human interventions.
Recent work aims to train agents (forward) with learned resets by constructing
a second (backward) agent that returns the forward agent to the initial state. We
find that the termination and timing of the transitions between these two agents
are crucial for algorithm success. With this in mind, we create a new algorithm,
Reset Free RL with Intelligently Switching Controller (RISC) which intelligently
switches between the two agents based on the agent’s confidence in achieving its
current goal. Our new method achieves state-of-the-art performance on several
challenging environments for reset-free RL.

Published as a conference paper at ICLR 2024
INTELLIGENT SWITCHING IN RESET-FREE RL
Darshan Patil
Janarthanan Rajendran Dalhousie University
Mila, Université de Montréal
Glen Berseth Mila, Université de Montréal Canada CIFAR Al Chair
Sarath Chandar
Mila, École Polytechnique de Montréal
Canada CIFAR AT Chair
ABSTRACT
In the real world, the strong episode resetting mechanisms that are needed to train agents in simulation are unavailable. The rexetting assumption limits the potential of reinforcement learning in the real world, providing resets to agent usually requires the creation of additional handcrafted mechanisms or human interventions. Recent work aims to train agents (forward) with learned resets by constructing a second (backward) agent that returns the forward agent to the initial state. We find that the termination and timing of the transitions between these two agents are crucial for algorithm success. With this in mind, we create a new algorithm, Reset Free RL with Intelligently Switching Controller (RISC) which intelligently switches between the two agents based on the agent's s confidence in achieving its current goal. Our new method achieves state-of state-of-the-ar performance on several challenging environments for reset-free RI. |
INTRODUCTION
Despite one of reinforcement learning's original purposes being as way to emulate animal and human learning from interactions with the real world (Sutton & Barto 2018), most of its recent successes have been limited to simulation (Mnih et al., 2015| Silver et al. 2016 Team et al. (023). One of the reasons for this that most work trains agents episodic environments where the agent is frequently and automatically reset to starting state for the task. Environment resetting, while simple to do in a simulator, becomes much more expensive, time consuming, and difficult to scale for real world applications such as robotics, as it requires manual human intervention the use of highly specialized scripts.
Our current algorithms designed around the ability to reset the environment, and do not transfer to settings without resets (Co-Reyes et al. 2020). Resets allow the agent to practice a task from the same initial states, without needing to learn how get those initial states. This revisiting is critical for RL agents that learn through trial and error, trying different actions from the same states to learn which are better. Resets also allow the agent automatically exit problematic regions of the state space. It is often much easier to enter certain regions of the state space than it is to exit them (c.g.. falling down casier than getting back up).
Recent works have started to explore learning in environments where automatic resets are not available setting known as reset-free or autonomous RL. A common approach for such settings is have the agent switch between a forward controller that tries to learn the task, and reset controller that learns to reset the agent to favorable states that the forward controller can learn from (Eysenbach et al. 2017 Han et al, 2015 Zhu et al. 2020 Sharma et al. 2021a 2022). Crucially, the part of these algorithms that switches between the controllers has gone understudied.
This paper explores how we can improve performance by switching between controllers more intelligently. Prior works have not established a consistent strategy on how to bootstrap the value
Coorespondence to darshan, patil @ milaaquebec Work done during Postdoc Mila. Université de Montréal "Code available at httpps: / / thub. com whw./aaaanaaaaaaa 1ab/Ria

---

Published as a conference paper at ICLR 2024
Reverse Curriculum Reset-Free RI.
Intelligent Switching Controller
Episodic RL
I.
I.
Figure I: In RL, the agent usually starts learning about the task in areas around rewarding states, and eventually propagates the learning to other parts of the state space. (Left) In episodic learning, the agent starts its trajectories at a state the initial state distribution. Through exploration, it might find a trajectory that produces a reward, but might struggle to reach that goal state again, particularly on sparse reward long horizon tasks. (Center) A common approach Reset-Free RL build curriculum outward from the task's goal states. While this allows the agent to frequently visit rewarding states, also means the majority of the agents experience will be generated in areas has already learned. (Right) RISC switches directions when it feels confident in its ability to achieve its current goal (both the forward (task's goal states) or the backward direction (task's intial states)). This not only reduces the time spent in already explored regions of the state space, but also reduces the average distance to the goal which makes it easier for the agent to find high value states.
of last state before the agent switches controllers. Bootstrapping is when the agent updates some value estimate using an existing value estimate of a successor state (Sutton & Barto 2018). It is common idea used across many RL algorithms. Pardo et al. (2022) empirically established that different bootstrapping strategies for the last state in a trajectory can result in different policies. We extend this result by analyzing why different bootstrapping strategies policies. We show that bootstrapping the last state in the trajectory is crucial for reset-free RL agents maintain consistent learning targets, and doing so greatly improves their performance.
result in different optimal
Another underexplored aspect of the switching mechanism is learning when to switch. Because reset-free RL is unique setting in that there no episode time limit imposed by the environment while training, the duration of the agent's controllers' trajectories becomes parameter of the solution method. Prior work generally uses a fixed time limit, but can learning when switch also help the agent learn? For example, if the agent enters a part of the state space where it already knows how to accomplish its current goal, gathering more experience in that area is unlikely to help further in learning the task. Switching controllers and thus the goal of the agent can allow the agent to learn more efficiently by gathering experience for states and goals it has not mastered yet.
Based on these two ideas, we propose Reset Free RL with Intelligently Switching Controller (RISC), a reset-free RL algorithm that intelligently switches between controllers. To reduce the amount of time spent in parts of the state space that the agent has already learned well in, learn score corresponding to the agent's ability to reach its current goal (the current goal could either in the forward the backward direction). The agent then switches directions with probability proportional to that score. This allows the agent to maximize the experience generation in areas of the state space that it still has learn more from (Figure [ right). We evaluate our algorithm's 's performance on the recently proposed EARL benchmark (Sharma et al. (0211). The benchmark consists of several robot manipulation and navigation tasks that need to be learned with minimal environment resets, and we show that our algorithm achieves state-of offtthe-art art performance on several of these reset-free environments.
RELATED WORK
RL algorithms developed for episodic settings often fail on even simple tasks when trained in reset- free environments (Co-Reyes et al. 2020 Sharma al.) 202 Several settings have been proposed to address the shortcomings of episodic RL environments including continuing RL (Khetarpal et al. 2022), which tries to maximize the training reward over the lifetime of the agent, and single lifetime
2

---

Published as a conference paper at ICLR 2024
RL (Chen al. 2022) which tries to accomplish task just once in a single lifetime. Other works have explored the problem of learning to practice in separate training environments in episodic settings (Rajendran et al. 2019). Our setting is different in that the agent practices in non-episodic environment, but is eva evaluated on task in an episodic environment. Several approaches have been proposed to enable learning without resets, including unsupervised skill learning (Xu et al. 2020 Lu
ct al.
2021> and framing the problem as multi-task learning problem (Gupta et al. 202 2022)
simple approach to learn in this setting is Forward Backward RL (FBRL), which alternates between a forward controller that tries to accomplish the task and a reset controller that tries to recover the initial state distribution (Eysenbach al., 2017 Han al. 2013). Other works explore different reset strategies, such as R3L where the reset controller tries to reach novel states (Zhu et al. (2020) or MEDAL which tries recover the distribution of states in demonstration data (Sharma et al. [022
2023). VapRI. uses demonstration data to make reset controller that builds reverse curriculum for the forward controller (Sharma et al., 2021a). IBC, a concurrent work, uses optimal transport to create curriculum for both the forward and reset agents without demonstrations (Kim et al. 2023).
Early resets have been used the context safe reset-free RL where the reset policy triggered when the forward policy is about to do something unsafe (Eysenbach et al.) [017]] Kim et al. 2022). Notably, these resets happen from the opposite direction compared to our method (see Center vs Right in Figure I Instead of taking the agent back to areas it is confident in, as these methods do, our method aims to maximize the amount of experience gathered in areas of the state-goal space where the agent has not learned.
PRELIMINARIES
In this section, define the problem of reset-free RL. Consider a goal-conditioned Markov Decision Process (MDP): M = A. T, Py, ?) where S denotes the state space, .A denotes the action space, denotes the goal space, T(-la, s) transition dynamics function, r(s, a, g) is reward function based on goal
is the initial state distribution, Pg is the desired goal distribution, and 7 [0, 1] is discount factor denoting preference for long term rewards over short term rewards. In episodic RL, the objective is to find policy π that maximizes
Ext(s,, an
(TT) Exg pep.pp.rr-paaeemmmm....
((la,s))
(I)
In this formulation, sampling se P starts episode and "resets" the environment. Episodic RL assumes regular access to resets, usually happening every few hundred or thousand timesteps.
Our problem setting of reset-free RL, formalized the "deployment" setting in Sharma al. (b021b) involves maximizing the same objective as above. The main difference between the episodic setting and the reset-free setting that the reset-free setting allows for much less frequent resetting of the environment during training (anywhere from once every few hundred thousand steps to just once the beginning of training). In the episodic case, because the agent reset to the initial state distribution frequently, it can directly evaluate the objective and much more easily get signal for how is performing. In the reset-free setting, because the agent is reset to the initial state distribution infrequently, the agent has to improve its performance on the objective while getting much less direct feedback on its performance.
RESET FREE RL WITH INTELLIGENTLY SWITCHING CONTROLLER
In this section, we describe Reset Free RL with Intelligently Switching Controller (RISC) and its components. Section 4. discusses the importance of proper bootstrapping when switching controllers in Reset-Free RI, Section 4.2 describes RISC's early switching mechanism, and Section 4.3 summarizes our entire approach.
4.1 ON BOOTSTRAPPING FOR RESET-FREE RL
In this section, we highlight the importance of proper bootstrapping when switching controllers in Reset-Free RI. For any policy æ, the value function / (s) measures the expected discounted sum of
3

---

Published as a conference paper at ICLR 2024
rewards when taking actions according to from state
V (=) = Ex Σ > R(s,, ax)] 8, at [(-
For brevity and clarity of exposition, we focus our analysis on the value function, but our analysis can be extended to state-action value functions with basic modifications. The value function satisties the Bellman equation (Bellman T957):
V (s) = E, [R(s, a) + + (())),, T()ss,a)).
(3)
Most RL algorithms involve learning either the value function or the state-action value function, usually through temporal difference (TD) learning (Sutton & Barto 2018). The main mechanism of TD learning is bootstrapping, where the value of a state is updated using the value of a successor state with update mirroring the Bellman equation. When learning in an episodic environment, is common to have time limits that end the episode before the agent has reached a terminal state. These early resets useful because they can diversify the agent's s experience. The value of terminal states is generally not bootstrapped, as by definition, the value of terminal state zero and it has no successors. Pardo al. (2022) empirically showed that bootstrapping the value of the last state in a trajectory that ended because of a timeout can result in not only a different policy than not bootstrapping, but also better performance and faster learning. We investigate why this is the case and discuss the implications for reset-free RL.
Let us denote the strategy where the value function is learned by bootstrapping the last state in trajectory that ended in a timeout bootstrapped as umneout-termingg Based on equation the loss for the timeout nontermmn strategy
and the strategy where such states are not
corresponds to
(0) = Ba (-1s) [IV (s) vV
Here & corresponds to the weighting of different states according to the experience that is gathered by the agent, and P (-1s) is the transition distribution under policy æ. The loss for the timeout-terminal strategy corresponds to
(0) =
[T (r (1- d)
(5)
The loss uses the same terms as the timeout-nonterminal strategy, except for binary random variable d drawn from some process that represents whether the episode ended because of timeout or not.
In episodic RL, E is the stationary distribution over states for policy x. and depends only on n, the initial state distribution p, and the transition dynamics function T. K is the rule that returns I when the trajectory gets over certain length, and thus E[d[s] only depends on n, p. T, and k. The difference in the two loss terms results in the magnitude of the value of some states being reduced in the timeout-terminal strategy, depending on B []]]s). This explains why different policies can emerge from the two strategies.
In reset-free RL, these parameters become more complicated. We can think of there being two policies: the forward policy Ty π T-Iss, gy) and the reset policy म. = x-1ss, gr) where 9/ and gr forward goals and reset goals respectively. Examining the bootstrap loss for the forward policy, E is now no longer dependent just TT p. T, and forward timeout process K2 but also π, and reset timeout process Ky- Furthermore, Ki and K,r are no longer static rules provided by the environment, but are decision rules based on the agent. Despite this, the nimeouttnontermmiall loss in Equation 4 at least maintains the same targets for the value of each state. [[[[ss). however, is now dependent on TT Tr, p. T, Ks,, and Ky. This means that when using the timeout-terminal bootstrap loss (Equation 5). the value function for the forward policy depends on the actions of the agent's 's reset policy and vice versa. Thus, in order to maintain consistent targets while training, RISC uses the timeout- centrurmnatt loss when switching controllers.
4.2 BUILDING A CURRICULUM WITH INTELLIGENT SWITCHING
Using a curriculum of tasks through the course of training has been shown to accelerate learning in reinforcement learning (Portelas al.) 2020: Florensa al.) 2017: Sukhbaatar et al.) 2018). A common approach to building these curriuuaa is to start with tasks with initial states near the goal, and expand outwards (Florensa al. 2017). known reverse curriculum.
4

---

Published as a conference paper at ICLR 2024
Algorithm 1: Reset Free RL with Intelligently Switching Controller (RISC)
Algorithm 2: Switching function
Input:
Minimum trajectory length: m
Maximum trajectory length: M
Input:
Trajectory switching probability: < env. env.reset()
Conservative factor: /
Current state, goal: 8,g
Length of current trajectory: t
t 0
Success Critie: F
swicchhswtch-ooo random() <S
Whether to perform switching check:
while True do
boock_switch
a agent g)
// Swi if reached goal if == 9 then
s s'r = env. (t)
return True
agent.update(s. a, f =
else if M then// Truncate after M steps
return True
sshuldhawiiteeee switch(t, agent.Q,, 5', & check switch) then
else if m then
// Min. Length Check
return False
else if check_ -errckhhsshhhhhhhh then
= F(s.9):
// agent competency
g switchh_oolss(
cx (1 3)):
// (switch)
check_swii switch = random() <
return random)) <
else end
return False
end
8
end
Some Reset-Free RL methods explicitly build a reverse curriculum (Sharma al., 2021a); other methods while not building a curriculum still follow the paradigm of always returning to the goal state. While reverse curriculums are popular idea, we ask if they are the best approach for reset-free RL, they can involve the agent spending significant amount of environment interactions in parts of the state space it has already mastered (Figure []] middle). Instead, we hypothesize that reset-free RL can benefit from a new type of curriculum that works from the outside in.
Our algorithm is based on the intuition that the agent is already competent in an area the state space, then it does not need to continue generating more experience in that part of the state space. Instead, it could use those interactions to learn in parts of the state space it still needs to learn, leading more sample efficient learning. To that end, we introduce Reset Free RL with Intelligently Switching Controller (RISC). Similar to Forward-Backward RI. (Han er al. 2015 Eysenbach et al. 2017). RISC alternates between going forward towards the task goal states and resetting the task initial states. Unlike Forward-Backward RL and other reset-free methods, RISC switches not only when it hits some predefined time limit or when it reaches its current goal (in the forward or the backward direction), but also when it is confident that it can reach its current goal. This allows it to spend more time exploring regions of the state-goal space that it is unfamiliar with (Figure [ right).
4.2.1 EVALUATING THE AGENT'S ABILITY TO ACHIEVE ITS GOALS
To build our intelligent switching algorithm, we need way to estimate if agent is competent in an area of the state space. A natural way to do this is to incorporate some notion of how many timesteps it takes the agent achieve its goal from its current state. Thus, define the competency of policy
r in state & on goal as
(6) where Yec E [0, 1] is the success critic discount factor and represents the minimum number of steps it would take to get from state to goal y when following policy æ. This metric has several appealing qualities. First, it is inversely correlated with how many timesteps it takes the agent to reach its goal, incorporating the intuition that becoming more competent means achieving goals faster. It also naturally assigns lower competency to states that are further away from a goal, which incorporates the intuition that the agent should be more uncertain the further away it is from its goal. Finally, this metric can be decomposed and computed recursively, similar to value functions in RL. This allows us to use the same techniques to learn this metric as use to learn value functions.
Fx g) = 7 t= min(t' : Sp = 9. so = s, a, ~ T(s..)) $1+1 ~ T(-aa, s)},
We assume the ability to query whether given state matches a goal, but only for goals corresponding to the task goal distribution or the initial state distribution. As the goal and initial state distributions
5

---

Published as a conference paper at ICLR 2024
represent small portion of states, learning such function is not unreasonable requirement for real- worl-world upplications. We can then train a success critic (Qr) to estimate (s g). Specifically, given success function f S ( G + {0, that determines if state matches goal g. train network to predict
[ [(((+1) + ff
QF(st, = Bx
where Qr and Qr are the success critic and target success critic respectively. The structure and training of the success critic is similar to that of value critics used in Q-learning or actor critic algorithms. In fact, it is exactly Q-function trained with rewards and terminations coming from the success function. The value of this success critic should always be between O and 1. At inference time, compute the value of the success critic as (s. g) (=-=(s-a)) 1 a, g)].
Because we want the success critic's 's value to mirror the agent's 's ability to actually achieve the goal, train Qr the same batch of experience used to train the agent's 's policy и and critic Q and set Ise to the value of the agent's 's critic's discount factor.
4.2.2 MODULATING THE SWITCHING BEHAVIOR
Because F(s,9) .) [0, 1], it might be tempting to use it directly probability of switching. It's important to note that the output of the success critic does not actually correspond to any real probability; it is simply metric corresponding to the competency of the agent. Furthermore, switching with probability based solely on the values output by success critic can lead to excessive switching. If the agent is in area of the state-goal space where F(s, g) 0.1, the expected number of steps the agent will take before switching goals is only 10. If the agent is in a region with even moderately high success critic values for both forward and reset goals, could get stuck switching back and forth in that region and not explore anywhere else. The switching behavior also alters the distribution of states that the agent samples when learning, shifting it further from the state distribution of the current policy, which can lead to unstable learning.
To counteract the excessive switching behavior, we introduce three mechanisms to modulate the frequency of switching: (1) Switching on Portion of Trajectories: Inspired by how e-greedy policies only explore for certain portion of their actions, we only apply our intelligent switching any given forward or backward trajectory with probability Ç. This decision is made at the start of the trajectory. (2) Minimum Trajectory Length: For the trajectories we do early switching for, we wait until the trajectory has reached a minimum length before making switching decisions. (3) Conservative Factor: Finally, we define the probability of switching as a function of F(s.g) g) and trajectory length, that increases with trajectory length. Specifically, P( P(switch) F(s. g) x (1 (8)) where is the number timesteps in the current trajectory and [0, is parameter that decides how conservative to be with the switching. As f oo, P(switch) F(s,g).
4.3 ALGORITHM SUMMARY
The early switching part of our method is summarized in Algorithms 2 and At the start of trajectory, our method decides whether to perform early switching checks for the current trajectory. Switching checks are only performed the trajectory reaches some minimum length. Finally, the probability of switching is computed as function of the success critic that increases with trajectory length. The agent samples a bernoulli random variable with this probability to decide whether to switch goals.
We also do montrrminnnttttin bootstrapping as described Section 4.1 when training our agent's critic and success critic.
EXPERIMENTS
In this section, we empirically analyze the performance of RISC. Specifically, we: (1) Investigate whether reverse curriculums are the best approach for reset-free RL; (2) Compare the performance of RISC to other reset-free methods on the EARL benchmark: (3) Evaluate the necessity of both;
bootstrapping and early switching for RISC with ablation study.

---

Published as a conference paper at ICLR 2024
(a) Tabletop Manipu- Sawyer Door lation
Sawyer Peg
(d) Minitaur
(e) 4 Rooms
Figure 2: The environments used in Section 5
The first 4 are part of the EARL benchmark (Sharma
al. 202Tb), and the last based on Minigrid Ghhevalier-ooiveer et al.) 2018).
Environments. The experiments in Section 5.1 use 4 gridworld where the agent needs go from one corner to the opposite corner. The representation consists of 3 binary image channels, corresponding to the agent, the walls, and the goal location, while each cell corresponds
spatial
location. Sections 5.2 and 5.3 use four environments from the EARL benchmark (Sharma et al. 20216): the Tabletop Manipulation environment (Sharma al.) 2021a) involves moving mug to one of four locations with a gripper; the Sawyer Door environment Yu et al. 2019) has sawyer robot learn close door; the Sawyer Peg environment (Yu et al.) (2019) has sawyer robot learning to insert peg into goal location; the Minitaur environment (Coumans & Bail 2016) is locomotion task where a minitaur robot learns to navigate to set of goal locations. The first environments are sparse reward tasks where set of forward and resetting demonstrations are provided to the agent, while the last environment is a dense reward environment with no demonstrations provided.
These environments all provide low dimensional state representation for the agent. We follow the evaluation protocol outlined by Sharma et al. (2021b). evaluating the agent for 10 episodes every 10,000 steps. Every agent
for 5 seeds. For all environments, assume access to reward function and success function that can be queried for goals in the task goal distribution or initial state distribution.
Comparisons. We compare against the follow- ing baselines and methods: (1) Forward Back- ward RL (FBRL) (Eysenbach et al., 2017: Han al., 20155), where the agent alternates between going forward to the goal and resetting back to the initial state distribution; (2) R3L (Zhu et al.) 2020), which uses novelty based reset controller; (3) VapRL (Sharma al.) (021a), where the re- set controller builds curriculum using states in demonstration data for the forward agent based on the forward agent's performance; (4) MEDAL Sharma et al. 2022). where the reset controller states in the demonstration data; (5) Naive RL, where the agent only optimizes for the task reward throughout training i.c., an episodic agent being trained in a reset-free environ- an episodic environment with frequent automatic resets and meant as an unrealistic oracle to understand the best possible performance. The performances for these methods were either sourced from pub- licly available numbers (Sharma al.) 2b221b) recreated with public implementations.
GoiMaes RBS
RUSC
learns
Oateme Eifivencee
ment; (6) Episodic RL, agent
RISC
Revaso Cunkslum
2002
10000
24000
50000
Traniig Slaae
Figure The heatmaps show the progression of the Q-values (top) and the visitation frequencies (middle) of the agent in forward mode in the 2000 steps after the noted timestep. The reverse curriculum method tends to be biased towards states that it has already learned, while RISC is more evenly distributed, and slightly biased towards states it hasn't 't fully learned.
SAC agent (Haarnoja et al. 2018) as the base agent, with similar hyperparameters to earlier works. For the rooms experiments, all agents use a DQN (Mnih al. 2015) agent their base.
For the EARL experiments, all agents

---

Published as a conference paper at ICLR 2024
Episodic RISC (nun)
MEDAL VapRL
FBRL
Naive
1.0
IQM of Normalized Score
IQM
Mean
0.8
Episodic (ours)
0.6
MEDAL
0.4
VapRL
FBRL
R3L
Naive
0.3
0.6
0.99
0.25
0.50
0.75
1.00
0.2
0.4
1.0
Normaliize Score
Fraction of Training Run
Figure 4: 95% confidence intervals for the interquartile mean (IQM) and mean normalized per- formance of RISC and other baselines aggregated across environments the EARL benchmark (Tabletop Manipulation, Sawyer Door, Sawyer Peg, and Minitaur). Because MEDAL and VapRL require demonstrations and thus do not work Minitaur, exclude Minitaur from their calculations (left). IQM of RISC and other baselines on EARL benchmark as a function of progress through the training Shaded regions represent 95% confidence intervals (right). RISC outperforms and learns much faster than other reset-free baselines.
5.1 CURRICULA IN RESET-FREE RL
We first investigate the performance of RISC on a rooms minigrid environment, comparing it to simple Reverse Curriculum (RC) agent. The RC agent maintains success critic similar to RISC. When going backward, it switches to the forward controller if the success critic value falls below given threshold. While this is not the same as other reset-free reverse curriculum methods in the literature, simple to implement, does not require mechanisms such as demonstrations or reward functions for arbitrary states, and is useful tool to see differences in behavior of the agents.
Figure 3 tracks the behavior and internals an RISC and RC agent across time. shows, for the noted timesteps (1) the Q-values of each policy (Figure 3 top). (2) the visitation frequency of both agents for the 2000 steps following the timestep (Figure 3 middle) (3) the difference in the agent's state value estimate and the optimal state value estimate for the experience that each agent gathers (Figure 3 bottom). We call this the Optimal Value Percent Difference (OVPD), and is calculated as OV (V)) mmr, Qa(s.)) (s) We use it as proxy for how useful the state is for the agent to collect.
(*)
The figure shows that the RISC agent tends to gather experience areas that has not yet mastered (i.e., Q-values are low), while the RC agent gathers much more experience in areas that has already mastered. This is supported both by comparing the local visitation frequencies of the agents to the Q-values, and the OVPD values. The OVPD of the RC agent drops very low even before it has converged, meaning that the RC agent is not collecting transitions that it has inaccurate estimates for. Meanwhile, the RISC agent's OVPD stays relatively high throughout training, until it converges. This analysis suggests that using reverse curriculum might not be the best approach for Reset- Free RL.
5.2 EVALUATING RISC THE EARL BENCHMARK
Figures and 5 show the results of RISC on the EARL benchmark environments compared to previously published results. The aggregate performance of RISC on the EARL benchmark is better than any other reset-free method. RISC also learns the fastest both in aggregate and across multiple individual environment the benchmark. In the scope of reset-free RI, it is fairly expensive to collect data, any method that can reach the better policy faster is of great value.
is also important to note the requirements and complexity of some of these methods. Both MEDAL and VapRL require demonstrations, meaning that they cannot be run environments such as Minitaur. VapRL also requires access to a reward function that can be queried for goal states in the demonstration data. RISC uses demonstration data to seed the replay buffer, but does not require demonstrations to provided. When comparing RISC to the methods that can run on all environments, RISC clearly outperfoms them all.
8

---

Published as a conference paper at ICLR 2024
Tabletop Marisslation
Sawyer Ooor
Sunyer Pax
Meniaur
1.00
1.00
L00
0.20
200
Renm
0.30
0.50
Retum
0.30
-000
0.35
M
0.90
0.00
Triring
Traning Staps
Trining Staps
Traning Steps
RISC [eurs)
Epiodi
FBRL.
MEDAL
haiv
RSL
VapRI
Figure 5: Average test returns on EARL benchmark tasks over timestep. RISC improves upon or matches the state-of-the-ar for reset-free algorithms on the 4 environments (Tabletop Manipula- tion, Sawyer Door, Sawyer Peg), and even outperforms/matchee Door and Sawyer Peg. Learning curves and code were not available for several baselines for Minitaur, so only the final performance is plotted. Results are averaged over 5 seeds and the shaded regions represent standard error.
the Episodic baseline on Sawyer
IQM
P(RISC Y)
RISC
RISC w/o switching RISC w/o timeaut aware bootstrapping
FBRL
0.8
0.2
0.6
0.5
0.6
0.9
1.0
Normalized Score
Figure 6: 95% confidence intervals for the interquartile mean (IQM), mean, and Probability of Improvement of RISC for ablations of RISC.
5.3 ABLATING THE COMPONENTS RISC
We now ablate the components of RISC to better understand the contributions of each. We rerun RISC with early switching removed and with the timeout aware bootstrapping removed. Removing both components results in method that is equivalent to FBRL., and so we also include that in our comparison. From Figure 6 we see that removing the timeout aware bootstrapping results in significant drop in performance, and removing early switching also results in modest drop. This suggests that both components are important for the success of RISC.
These results are also consistent in the context of our analysis in Section Our arguments there imply that not doing timeout aware bootstrapping can large issue for reset-free methods, and especially methods such ours where the trajectory lengths be significantly shorter than the episode lengths for the deployment task.
CONCLUSION AND FUTURE WORK
We propose a method that learns to intelligently switch between controllers in reset-free RI. It does so by (1) careful management of bootstrapping of states when switching controllers and (2) learning when to actually switch between the controllers. Our method results in performance that sets or matches the state-of-the-art
of the 4 environments from the recently proposed EARL benchmark. We show success across both sparse reward tasks with demonstrations and dense reward tasks with no demonstrations. Finally, we note that least two of the environments, Sawyer Door and Tabletop Munipulation, seem close to being saturated in terms of performance, and future work should explore other environments or ways of increasing the difficulty of those environments.
One limitation of our method is that it may not be well suited for environments with irreversible states, where the agent could get stuck. RISC tries to take advantage of the reset-free RL setting by exploring the state space more aggressively. While this is not an issue with the environments we study in this work, as they do not contain any irreversible states, it likely could be problematic in environments where there demonstrations to guide our agent intelligently as previous works do, and learning to do so could be an interesting avenue for future work.
inreversible states and safety is concern. We also do not leverage

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
Sarath Chandar is supported by the Canada CIFAR Al Chairs program, the Canada Research Chair in Lifelong Machine Learning, and the NSERC Discovery Grant. Glen Berseth acknowledges funding support from the Canada CIFAR AI Chairs program and NSERC. Janarthanan Rajendran acknowledges the support of the IVADO postdoctoral fellowship. The authors would like to thank Nishanth Anand, Mohammad Reza Samsami, and anonymous reviewers for their helpful feedback and discussions. We would also like to acknowledge the material support of the Digital Research Alliance of Canada (alliancecan.ca) Mila IDT (mila.quehec), and NVidia in the form of computational resources.
10

---

Published as a conference paper at ICLR 2024
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Mare G Bellemare. Deep reinforcement learning the edge of the statistical precipice. Advances in Neural Information Processing Systems, 2021.
Richard Bellman. A Markovian Decision Process. Journal of Mathematics and Mechanics, 6(5): 679-684, 1957. ISSN 0095-9057.
Annie S Chen, Archit Sharma, Sergey Levine, and Chelsea Finn. Single-Life Reinforcement Learning. In Neural Information Processing Systems 2022, 2022.
Maxime Srevwllir-Boisveedd Lucas Willems, and Suman Pal. Minimalistic Gridworld Environment for Gymnasium, 2018.
John D. Co-Reyes, Suvansh Sanjeev, Glen Berseth, Abhishek Gupta, and Sergey Levine. Ecological Reinforcement Learning. arXiv:2006. 12478 [cs, stat], June 2020.
Erwin Coumans and Yunfci Bai. PyBullet, a Python module for physics simulation for games, robotics and machine learning, 2016.
Benjamin Eysenbach, Shixiang Gu, Julian Ibarz, and Sergey Levine. Leave Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning. 17X1v:1711.06782 [es), November 2017.
Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. Reverse Curriculum Generation for Reinforcement Learning. In Proceedings of the 1st Annual Conference on Robot Learning, pp. 482 PMLR, October 2017.
Abhishek Gupta, Justin Yu, Tony Z. Zhao, Vikash Kumar, Aaron Rovinsky, Kelvin Xu, Thomas Devlin, and Sergey Levine. Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human Intervention. arXiv:2 1004. 104.1/20. 2021.
1200 [cs], April
Abhishek Gupta, Corey Lynch, Brandon Kinman, Garrett Peake, Sergey Levine, and Karol Hausman. Demonstration-Boooksee Bootstrapped Autonomous Practicing via Multi Task Reinforcement Learning. arXiv:2203.15755 [cs], March 2022.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. In Proceedings of the 35th International Conference on Machine Learning, pp. 1861-1870. PMLR, July 2018
Weiqiao Han, Sergey Levine, and Pieter Abbeel. Learning compound multi-step controllers under unknown dynamics. In 2015 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 6435-6442, September 2015, doi: 10.1 1091100/ 2015..552922......29..
Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup. Towards Continual Reinforcement Learning: Review and Perspectives, November 2022.
Jigang Kim, J. hycon Park, Dacsol Cho, and H. Jin Kim. Automating Reinforcement Learning with Example-based Resets. arXiv:2204.0200 02044 [cs], April 2022.
Jigang Kim, Daesol Cho, and H. Jin Kim. Demonstration-fret free Autonomous Reinforcement Learning via Implicit and Bidirectional Curriculum. In Proceedings of the 40th International Conference on Machine Learning, pp. 16441-16457. PMLR, July 2023.
Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. Reset-Free Lifelong Learning with Skill-Space Planning, June 2021.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, loannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533 February 2015. ISSN 1476-4687. doi: 10. 1..10388/ature142236
11

---

Published as a conference paper at ICLR 2024
Fabio Pardo, Arash Tavakoli, Vitaly Levdik, and Petar Kormushev. Time Limits in Reinforcement Learning, January 2022.
Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, and Pierre-Yves Oudeyer. Automatic Curriculum Learning For Deep RL: Short Survey, May 2020.
Janarthanan Rajendran, Richard Lewis, Vivek Veeriah, Honglak Lee, and Satinder Singh. How Should an Agent Practice?, December 2019.
Archit Sharma, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Autonomous Reinforcement Learning via Subgoal Curricula. orXiv:2107.12931 [cs], October 2021a.
Archit Sharma, Kelvin Xu, Nikhil Sardana, Abhishek Gupta, Karol Hausman, Sergey Levine, and Chelsea Finn. Autonomous Reinforcement Learning: Formalism and Benchmarking.
arXiv:2112.09605 [cs], December 2021b.
Archit Sharma, Rehaan Ahmad, and Chelsea Finn. A State-Distribttion Matching Approach to Non-Epissoicc Reinforcement Learning, May 2022.
Archit Sharma, Ahmed M. Ahmed, Rehaan Ahmad, and Chelsea Finn. Self-Improving Robots: End-to-End Autonomous Visuomotor Reinforcement Learning, March 2023.
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonogiou, Veda Panncershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):484 489, January 2016. ISSN 1476-4687. doi: 10. 1038/nature 16961.
Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Rob Fergus. Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play. International Conference Learning Representations, February 2018.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, USA, October 2018. ISBN 978-0-262-0392446.
Adaptive Agent Team, Jakob Bauer, Kate Baumli, Satinder Baveja, Feryal Behbahani, Avishkar Bhoopchand, Nathalie Bradley-Schmiee Michael Chang. Natalie Clay, Adrian Collister, Vibhavari Dasagi, Lucy Gonzalez, Karol Gregor, Edward Hughes, Sheleem Kashem, Maria Loks-Thompsae Hannah Openshaw, Jack Parker-Holder, Shreya Pathak, Nicolas Perez-Nieves, Nemanja Rakicevic, Tim Rocktäschel, Yannick Schroecker, Jakub Sygnowski, Karl Tuyls, Sarah York, Alexander Zacherl, and Lei Zhang. Human-Timescall Timescale Adaptation in an Open-Ended Ended Task Space, January 2023.
Kelvin Xu, Siddharth Verma, Chelsea Finn, and Sergey Levine. Continual Learning of Control Primitives: Skill Discovery via Reset-Games, November 2020.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta- Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning. In Conference on Robot Learning (CoRL), 2019.
Henry Zhu, Justin Yu, Abhishek Gupta, Dhruv Shah, Kristian Hartikainen, Avi Singh, Vikash Kumar, and Sergey Levine. The Ingredients of Real- World Robotic Reinforcement Learning. arXiv:2004. 12570 (cs, stat), April 2020.
12

---

Published as a conference paper at ICLR 2024
A
AGENT DETAILS
All of our agents for the experiments on the EARL benchmark (Sharma et al. (Haarnoja et al. 2018) the base agent. The neural network architecture used for the SAC agent was laken from the publicly available implementation of the MEDAL algorithm (Sharma et al. 2022). A Linear Layer (50 units), LayerNorm, and Tanh activation are applied to the observation before being fed to the actor or critic networks. The other hyperparameters used for the base agent are described in Table [ The experiments on the 4-rooms gridworld (Chevalier-ooiseert Boisvert al.] 2018) use DQN (Mnih et al. 2015) as the base agent. The corresponding hyperparameters for those experiments are shown Tables
2021b) use SAC
The additional hyperparameters for RISC are shown in Table 3
For the environments that provide demonstrations (listed in B we simply insert those demonstrations into the replay buffer of the agents at the start of training.
Hyperparameter
Value
Actor Network
MLP(256, 256)
Critic Network
MLP(256, 256)
Optimizer
Adam
Hyperparameter
Value
Actor LR
3e-4
Conv([16,16,16), 3), FC(64)
Critic LR
3e-4
Q-Network
L.R
3e-4
Optimizer
Adam 1e-3
Weight Initialization
Xavier Uniform Fill(O)
Q-Network LR
Bias Initialization
Target hard
Target entropy scale
0.5
500
update frequency
Reward Scale Factor
10
Critic Loss Weight
0.5
Discount Factor
0.95
Batch Size
128
Target update
0.005
Initial Collect Steps
512
Target update frequency
Replay Capacity
50,000
Discount Factor
0.99
Collect Steps
Batch Size
256
Update
Initial Collect Steps
10,000
e-greedy init value
1.0
Replay Capacity
10,000,000
<-gredy end value
0.1
Collect Steps
c-greddy decay steps
10,000
Update
Min Log Std
-20
Table 2: Hyperparameters for base DQN Agent
Max Log Std
10
Log Std Clamp function
Clip
Table 1: Hyperparameters for base SAC Agent
B EXPERIMENTAL SETUP
All experiments were run as CPU jobs. Table 4 gives further details about the experimental setup for each environment. Each configuration is run for seeds. Hyperparameter search was done over the following hyperparameters: conservative factor /3 {0.0, 0.9, 0.95]. minimum trajectory length as a fraction of the maximum trajectory length E {0.0, 0.25, 0.5, 0.75]. and switching trajectories proportion Ç (0.25, 0.5, 0.75,
C METRICS COMPUTATION
The metrics in Figures 4 and 6 were all calculated using the rliable library (Agarwal et al. 2021). Each metric was calculated using 2000 bootstrap replications over normalized and aggregated results
As fraction of the maximum trajectory length
13

---

Published as a conference paper at ICLR 2024
Hyperparameter
Value
Conservative Factor
0.9
Minimum Trajectory Lengt[
Trajectories Proportion {
1.0
Tabletop Manipulation
Success Critic Network
MLP(256, 256)
Success Critic Optimizer
Adam
Success Critic LR
3e-4
Success Critic Output Activation
-0 0.5cos(x))
Actions Sampled
Conservative Factor
0.9
Minimum Trajectory Length
0.75
Trajectories Proportion <
1.0
Sawyer Door
Success Critic Network
MLP(256, 256)
Success Critic Optimizer
Adam
Success Critic LR
3e-4
Success Critic Output Activation
0.5cos(x)
Actions Sampled
Conservative Factor
0.9
Minimum Trajectory Length
0.5
Trajectories Proportion Ç
0.5
Sawyer Peg
Success Critic Network
MLP(256, 256)
Success Critic Optimizer
Adam
Success Critic LR
3e4
Success Critic Output Activation
-..cco((z)
Actions Sampled
5
Conservative Factor
0.95
Minimum Trajectory Length
0.25
Trajectories Proportion Ç
0.25
Minitaur
Success Critic Network
MLP(256, 256)
Success Critic Optimizer
Adam
Success Critic LR
3e4
Success Critic Output Activation
-0.5cos(x) -
# Actions Sampled
Conservative Factor 3
0.95
Minimum Trajectory Length
Trajectories Proportion Ç 0.5
4 Rooms
Conv(16,16.166). 3),
Success Critic Network
FC(64)
Success Critic Optimizer
Adam
Success Critic Output Activation
sigmoid(z)
Success Critic LR
le-3
Table 3: Hyperparameters for RISC
# Train Steps
# Train Hours
Hard Reset Frequency
Epsisode length limit
# of Demo Transitions
Reward Type
Tabletop
3,000,000
24
200,000
200
2,534
Sparse
Manipulation
Sawyer Door
5,000,000
42
200,000
300
1,095
Sparse
Sawyer Peg
7,000,000
68
100.000
200
1.815
Sparse
Minitaur
5,000,000
50
100,000
1,000
0
Dense
4-Rooms 50,000
0.5
50,000
100
Sparse
Table 4: Experimental setup details for each environment.
from all tasks. For Minitaur, did not have the data for all of the baselines, only the final mean and standard error. To overcome this and integrate Minitaur results in our comparisons, we generated 1000 sets of samples from the distribution represented by the mean and standard error of
14

---

Published as a conference paper at ICLR 2024
Tabletop Organization
Sawyer Door
Sawyer Peg
Minitaur
Naive RL
0.32 0.17
0.00
0.00
1
1041.10 44.58
R3L
0.96 1 0.04
0.54 + 0.18
0.00 1 0.00
-18630 1 34.79
VaPRL
0.98 1 0.02
0.94 0.05
0.02 0.02
MEDAL
0.98 + 0.02
1.00 0.00
0
FBRL
0.94 0.04
1.00
-986.34 67.95
RISC (owrs)
1.0 10 0.0
0.0
0.42 = 0.07
-321.74 53.73
Oracle RI. 0.80 0.11
1.00
0.00
+ 0.00 -41.50 3.40
Table 5: Average return of the best policy over random seeds, reported with standard error. Higher returns are better, and the best method for each environment (excluding oracle) in bold. For Tabletop Manipulation, Sawyer Door, and Sawyer Peg, minimum performance is 0.0 and maximum performance 1.0.
IQM
Mean
Episodic
RISC (ours)
MEDAL
VapRL FBRL R3L
Naive
0.25
0.50
0.75
0.25
0.50
0.75
AUC Normalized Score
Figure 7: Interquartile mean and mean for the area under the curve (AUC) for the learning curve of each run, aggregated across and environments. Higher AUC is better and implies more sample efficient learning.
each baseline, computed rliable metrics with each and used the set with the highest mean, to give as much benefit of the doubt as possible to the baselines. Despite this bias, the difference between the best and the worst was quite small, with the largest difference being approximately .05 for FBRL.
D EARL RESULTS
We present the results for the best policy of each method on the environments in the EARL benchmark for direct comparison with previous methods in Table Note, RISC has the best performance of reset free methods on 3 of 4 environments, and performs somewhat competitively on the last environment.
We also present the interquartile mean and mean for the under the curve (AUC) of the learning curves for each agent, aggregated across seeds and environments (Figure 7 RISC significantly higher than the other methods, implying that is more sample efficient.
E EXPLORING THE MODULATING MECHANISMS
In this section, show the necessity of the modulations introduced in Section 4.2.2 In Figure see that removing the modulations significantly reduces the average trajectory length of the agent. Furthermore, when compare the performance of the RISC with and without modulations in Figure g, we see that without the modulations, RISC underperforms on all environments.
F HYPERPARAMETER ANALYSIS
We first present details of our hyperparameter sensitivity analysis. We took the optimal configurations selected for each environment, and varied the value of hyperparameter at time. The results in
15

---

Published as a conference paper at ICLR 2024
1.0
1.0
0.8
0.8
Return
0.8
Return
0.6
0.2
RISC
0.2
RISC
Cons. Factor Min. Steps
Cons, Factor
Min. Steps
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Training Steps
Training Steps
(a) Tabletop Manipulation
(b) Sawyer Door
Cons, Factor Steps
of Traj.
600
Retum
Return
RISC
Cons, Factor
-1000
Min. Steps
Switch Portion
Traj.
Training Steps
x100
Training Steps
(c) Sawyer Peg
(d) Minitaur
Figure 8: We show the effect of removing the modulations presented in Section 4.2. The perfor- mance of the agent degrades across all environments, with significant deterioration on Minitaur and Sawyer Peg.
16

---

Published as a conference paper at ICLR 2024
200
RISC
300
RISC
Unmodulated
Unmodulated
Trajectory Length
150
forward
Trajectory Length
250
forward
reset
reset
100
150
100
50
0
0.0
0.5
1.0
1.5
2.0
3.0
Training Steps
X10
Training Steps
(a) Tabletop Manipulation
(b) Sawyer Door
200
1000
Trajectory Length
Trajectory Length
120
RISC
RISC
100
Unmodulated
Unmodulated
forward
200
forward
reset
reset
0
Training Steps
x10
Training Steps
x10*
(c) Sawyer Peg
(d) Minitaur
Figure 9: We show the effect of removing the modulations presented in Section|4.2.2. on the trajectory lengths of the agent. As expected, the average trajectory length decreases significantly all environments.
17

---

Published as a conference paper at ICLR 2024
Tabletop Manipulation
Sawyer Door
Sawyer Peg
Minitaur
1.0
1.0
1.0
Cons. Factor
Return
500
0.5
0.5
0.5
1000
0.0
0.0
0.0
0.0
0.9
0.95
0.0
0.9
0.95
0.0
0.9
0.95
0.0
0.9
0.95
1.0
1.0
Min. Traj. Length
Retum
-500
0.5
0.5
0.5
1000
0.0
0.0
0.0
0.0
0.25
0.75
0.25
0.75
0.0
0.25
0.5
0.75
0.0
0.25
0.5
0.75
1.0
1.0
Trajectory Proportion
Return
500
0.5
0.5
1000
0.0
0.0
0.25 0.5
0.75
0.25
0.75
1.0
0.25
0.5
0.75
0.25
0.5
0.75
Hyperparameter sensitivity analysis. We vary the selected configuration one hyperparame- ter at time, and present mean and standard deviation each configuration. The configurations used in the experiments are marked with a green star.
Figure 10:
IQM
Mean
RISC
Only optimize Traj. Proportion Only optimize Min. Steps
Only optimize Cons. Factor
0.00 0.25 0.50 0.75 1.00
0.00 0.25 0.50 0.75 1.00
Normalized Score
Results when only optimizing hyperparameter time.
IQM
Mean
RISC
Don't optimize Traj. Proportion Don't optimize Min. Steps
Don't optimize Cons. Factor
0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00
Normalized Score
(b) Results when only optimizing hyperparameters at time.
Figure 11: Results of partial hyperparameter optimization.
Figure 10 show that our results fairly robust to changes in hyperparameters, with fairly small changes to the final return across all environments.
For second analysis, tried optimizing only hyperparameter time. For each environment, we select the configuration with the best performance for each hyperparameter where the other modulations are turned off, and aggregate across environments. We do similar analysis with optimizing 2 hyperparameters at a time. The results are shown Figure Tia and Figure TTb respectively. The most important hyperparameters proportion of trajectories where the switching happens. They recover most of the performance, but the full method still results in slight performance gain.
to be the minimum trajectory length the
18


Reviews:
[{"review_id": "XjRb4So2N3", "reviewer": "Reviewer_oJUJ", "summary": "The paper presents Reset Free RL with Intelligently Switching Controller (RISC), a novel algorithm for reinforcement learning in reset-free environments. RISC intelligently switches between two agents: a forward agent that learns the task and a backward agent that resets the agent to favorable states. The key ideas are proper bootstrapping when switching controllers and learning when to switch between the agents. The authors demonstrate that RISC achieves state-of-the-art performance on several challenging environments from the EARL benchmark.", "questions": "* How does RISC compare to other reset-free RL algorithms in terms of sample efficiency and generalization?\n* Can RISC be extended to handle environments with irreversible states, where the agent could get stuck?\n* How does RISC perform when combined with other techniques, such as curriculum learning or demonstration-based learning?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* RISC addresses the limitations of episodic RL in real-world applications, where resetting the environment is expensive and difficult to scale.\n* The algorithm intelligently switches between forward and backward agents, maximizing experience generation in unexplored areas of the state space.\n* RISC achieves state-of-the-art performance on several challenging environments from the EARL benchmark.", "weaknesses": "* The paper does not provide a thorough analysis of the theoretical properties of RISC, such as convergence guarantees.\n* The experiments are limited to a small set of environments, and it is unclear how RISC would perform on more complex tasks or in other domains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QteSroP5ia", "reviewer": "Reviewer_J5Lq", "summary": "This paper considers the problem of reset-free RL where automatic reset does not apply. Different from existing work, this paper proposes a new algorithm that switchs between forward and backward controllers intelligently. More specifically, the proposed switching function take the competent of state into account when the specific state has been explored well.  For this purpose, a Q-learning style algorithm is proposed to estimate the competency of a policy in a specific state. Through empirical experiments, the proposed algorithm is shown to achieve better performance comparing to the baselines on benchmarks.", "questions": "In the paper, the authors did not explain details about the hyperparameters needed in the proposed algorithm. In 4.2.2, there is some description. But I did not see any insight on how to tune these three parameters in different scenarios. And these parameters are essential for the algorithm to work properly. I would like to read more on this.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The considered problem in this paper is interesting and has great potential in real applications as episodic RL could be hard to achieve. \nIn terms of different terminal strategies, this paper theoretically analyze different terminal strategies in term of bootstrapping for the final state. As timeout-terminal strategy bring more challenges to the problem, it is more recommended to have timeout-terminal loss when switching controllers. The analysis is rigorous and easy to follow. \nThe idea of defining and learning the competent of state and policy for preventing unnecessary exploration is cogent.", "weaknesses": "Although the idea of switching according to competent makes sense, I have some concerns regarding the limitations of the proposed switching function. In order to have valid result, you need additional mechanisms to modulate the frequency of switching. It could be tricky to tune \\epsilon,\\beta, and the minimum length. There is always a tradeoff here, as you increase the constraints, your proposed method will gain less benefits.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yUujovfjo0", "reviewer": "Reviewer_gA9F", "summary": "Summary:\nThe paper addresses the reset-free reinforcement learning (RL) setting, which assumes the absence of a reset mechanism, making it a more realistic scenario due to the potential cost associated with resets in the real world. Existing RL methods typically train two separate agents: one toward the goal and another toward the initial state, aiming to bring the agent back to the initial state for additional practice. This paper contends that intelligently switching between these two agents, particularly when the agent is competent, is crucial. Furthermore, the addition of bootstrapping for the last state before switching is shown to significantly enhance performance. The proposed method, referred to as RISC, outperforms state-of-the-art reset-free RL methods on four EARL tasks (a common reset-free RL benchmark) and a four-room navigation task.", "questions": "1. Will code be available?\n\n2. Are there any correlations between adding bootstrapping and adding an intelligent switch? For example, will switch be **more** useful when bootstrapping is added? \n\n3. If I understand correctly, a separate critic success function was trained. Why didn't the authors use the critic from SAC (Soft Actor-Critic) instead of training a new one?\n\n4. In EARL paper, they claimed not using bootstrapping can break the long TD chain, in contrast, this paper suggests using bootstrap. Could authors also discuss these two different ideas?\n\n5. Bootstrapping for time-out non-terminal states should always be performed, do you have any intuitions on is it more important in the reset-free setting, since in episodic RL setting, people generally just ignore it? \n\n6. At the beginning of section 5.1, the paper discusses RC and RISC. Could you please elaborate more on RC and the difference between these two agents? Does RC also use bootstrapping? \n\n7. Analysis on Fig.3 (the last paragraph of section 5.1), you mentioned the RISC agent tends to visit areas where Q-values are low. But in the third/fourth column of Fig.3, it seems like the RC agent also tends to visit areas where Q-values are low? In the third column, the RISC agent actually gathers lots of data in the first room where Q-values are already quite high.\n\n8. There’s a performance drop on sawyer peg tasks, do you have any intuitions on the decrease of performance?\n\n9. Which task do you perform ablation study on in Fig.6?\n\n10. You mentioned demo data is put into the replay buffer, do you oversample it? Will RISC perform the same without demo data?\n\n11. There's a recent work called IBC (Demonstration-free Autonomous Reinforcement Learning via Implicit and Bidirectional Curriculum) from ICML 2023. While it is relatively new, it may be worth discussing it in the related work section without the need for experimental comparisons.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written, experiments are well-executed, and details of implementations are reported.\n\n2. The paper raises a significant concern regarding the handling of time-out non-terminal states in RL, which is often overlooked. It emphasizes the importance of correctly handling bootstrapping particularly in the reset-free context.\n\n3. The concept of intelligently switching between two different agents is intriguing, opening the door to further research in this direction.", "weaknesses": "1. One important argument this paper makes is that adding bootstrapping for time-out non-terminal states is important, although it is theoretically well motivated, I think it would be better to see some practical motivations, especially why it is important under the reset-free setting. For example, are value estimations very different?\n\n2. The paper introduces several method-specific hyper-parameters such as M, m, and β. It would be valuable to discuss the method's sensitivity and robustness to these hyper-parameters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LXQnFzPt09", "reviewer": "Reviewer_VmY7", "summary": "The paper presents a novel approach to tackle the challenge of training reinforcement learning agents in real-world scenarios without relying on strong episode resetting mechanisms. The authors propose a method called Reset Free RL with Intelligently Switching Controller (RISC) that learns to switch between controllers intelligently. RISC addresses the crucial aspects of managing state bootstrapping during controller transitions and determining when to switch between controllers. The experimental results demonstrate that the proposed method achieves state-of-the-art performance on various challenging environments for reset-free RL, including sparse and dense reward tasks, though the authors suggest further exploration of new environments or enhanced difficulty levels. However, a limitation of RISC is its potential unsuitability for environments with irreversible states, where safety concerns may arise. Additionally, the paper acknowledges the absence of leveraging demonstrations for agent guidance, which could be an interesting direction for future research.", "questions": "- In the last paragraph of 4.2.1, could the author explain more about why we should set $\\gamma_{sc}$ to be the value of the critic's discount factor?\n- Could the author explain more about how the method makes the learning stable?\n- I suppose the work will benefit the reward-free RL (Yang, Q., & Spaan, M. T. (2023). CEM: Constrained Entropy Maximization for Task-Agnostic Safe Exploration. AAAI-2023). Could the author elaborate a bit on how to use the ideas in reward-free RL for exploration?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper's key strength lies in its innovative approach to addressing the underexplored aspect of learning when switching between controllers in reset-free reinforcement learning. By recognizing the absence of an imposed episode time limit in this setting and considering the duration of controller trajectories as a parameter, the authors introduce a novel concept: dynamically learning when to switch controllers based on the agent's ability to achieve its current goal. This intelligent switching mechanism not only optimizes the agent's experience collection but also facilitates more efficient learning by focusing on unmastered states and goals. The proposed algorithm, Reset Free RL with Intelligently Switching Controller (RISC), is evaluated on a challenging benchmark of robot manipulation and navigation tasks, demonstrating state-of-the-art performance in various reset-free environments. This novel approach to controller switching has the potential to significantly enhance the effectiveness of reset-free reinforcement learning algorithms, addressing an important and previously unexplored aspect of the field.", "weaknesses": "- The paper has no significant theoretical results.\n- The convergence of the proposed method is not guaranteed.\n- Too many hyperparameters to tune, e.g., minimum trajectory length, conservative factor...", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)