Reset-free Reinforcement Learning with World Models
Authors: Zhao Yang
Thomas M. Moerland
Mike Preuss
Aske Plaat
Edward S. Hu
Venue: Accepted by TMLR ()

Abstract: Reinforcement learning (RL) is an appealing paradigm for training intelligent agents, enabling policy acquisition from the agent's own autonomously acquired experience. However, the training process of RL is far from automatic, requiring extensive human effort to reset the agent and environments. To tackle the challenging reset-free setting, we first demonstrate the superiority of model-based (MB) RL methods in such setting, showing that a straightforward adaptation of MBRL can outperform all the prior state-of-the-art methods while requiring less supervision. We then identify limitations inherent to this direct extension and propose a solution called model-based reset-free (MoReFree) agent, which further enhances the performance. MoReFree adapts two key mechanisms, exploration and policy learning, to handle reset-free tasks by prioritizing task-relevant states. It exhibits superior data-efficiency across various reset-free tasks without access to environmental reward or demonstrations while significantly outperforming privileged baselines that require supervision. Our findings suggest model-based methods hold significant promise for reducing human effort in RL. Website: https://yangzhao-666.github.io/morefree

Published in Transactions on Machine Learning Research (02/2025)
Reset-free Reinforcement Learning with World Models
Zhao Yang
The Leiden Institute of Advanced Computer Science
Leiden University
Thomas M. Moerland
The Leiden Institute Advanced Computer Science Leiden University
Mike Preuss
The Leiden Institute of Advonced Computer Science Leiden Universily
Aske Plaat
The Leiden Institute of Advanced Computer Science Leiden University
Edward S. Hu
GRASP Lab University of Pennsyloania
Reviewed on OpenReview: https: // opw.reveee.........mmmo net/ per/forum? id- 2dMIxlt
Abstract
Reinforcement learning (RL) an appealing paradigm for training intelligent agents, enabling policy acquisition from the agent's autonomously acquired experience. However, the training process of RL is far from automatie, requiring extensive human effort to reset the agent and environments. To lackle the challenging reset-free setting, we first demonstrate the superiority of model-based (MB) RL methods in such setting, showing that straightforward adaptation of MBRL can outperform all the prior state-of-tho-art methods while requiring less supervision. We then identify limitations inherent to this direct extension and propose a solution called model-based reset-free (MoReFree) agent, which further enhances the performance. MoReFFree adapts two key mechanisms, exploration and policy learning, to handle reset-free tasks by prioritizing task-relevant states. It exhibits superior data-efficiency across various reset-free tasks without access to environmental reward or demonstrat ions while significantly outperforming privileged baselines that require supervision. Our findings suggest model-based methods hold significant promise for reducing human effort in RL. Website: httpss//yyanghhao 66t1:e2aaao666...... wet.uiithubb.... io/morefree
1 Introduction
Reinforcement learning presents an attractive framework for training capable agents. At first glance, RL training appears intuitive and autonomous once reward is defined, the agent learns from its own automatically gathered experience. However, in practice, RL training often assumes the access to environmental resets that can require significant human effort to setup, which poses significant barrier for real world applications of RL like roboties.
Most RL systems on real robots date have employed various strategics implement resets, all requiring
considerable amount of effort (Levine al.) 2016 Yahya et al. 2017 Zhu al.] 2019 Nagabandi al. 2020 Tn Nagabandi et al. (2020), which trains dexterous hand to rotate balls, the practitioners had to

---

Published in Transactions on Machine Learning Research (02/2025)
(1) position a funnel underneath the hand to catch dropped balls, and (2) deploy separate robot arm to pick up the dropped balls for resets, and (3) script the reset behavior. These illustrate that even for simple behaviors, proper implementation of reset mechanisms can result in significant human effort and time.
Rather than depending on human-engineered reset mechanisms, the agent can operate within a reset- free training scheme, learning to reset itself (Eysenbach et al. 2017 Sharma al. 2021a 2022 Haldar et al., 2023) or train a policy capable of starting from diverse starting states (Zhu et al. 2020). However, the absence of resets introduces unique exploration challenges. Without periodic resets, the agent can squander significant time in irsseirreeevatt regions that require careful movements to escape and may overexplore, never returning from indefinite exploration. Recent unsupervised model-based RL (MBRL) approaches (Mendoncal et al. 2021 Hu et al. 2023) in the episodic setting have shown sophisticated exploration, high data-efficiency tasks. This prompts the question: would MBRL agents eacel in the
and promising results in reset-free R.L setting?
As an initial atlempt, we first evaluate an unsupervised MBRL agent, in reset-free Ant locomotion task. The ant is reset to the center of rectangular arena, and is tasked with navigating to the upper right corner. The agent is reset only once at the start of training. The evaluation is episodic the agent is reset at the start of each evaluation episode.
which was developed to solve hard exploration tasks in the episodic setting. As seen in Figure ! PEG, with minor modifications for the reset-free version, outperforms prior state-of-the-art, model-free agent, IBC (Kim et al., 2023), tailored for the reset-free setting.
For the MBRI. agent, we use PEG HHu et al. 2023
In Figure 1 plot state visitation heatmaps of the agents, where lighter colors correspond to more visitations. The oracle agent, with access to resets, explores the the "task- relevant" area between the initial and top right corner, which is idcal for training a policy that succeods in episodic evaluation. IBC's heatmap (bottom) shows that it fails to explore effectively, never encountering the goal states in the top right region. In contrast, PEG exhaustively explores the entire space, seen through its uniform heatmap. This results in an overexploration problem PFG may devote considerable time on finding irrelevant states rather than concentrating on the task-relovant region of the task. This leads us to ask: how can MBRL agents acquire more task-rolevant data in the reset-free setting to improve its performance?
1.0
agent with resets
reset-free PEG
SOTA model-free (IBC)
0.5
0.0
0.5
3.0
5.5
8.0
Steps (1e5)
Figure 1: Performance and collected data of dif- ferent agents on the reset-free Ant locomotion task.
We propose Model-based. Reset-Free (MoReFree), which improves two key mechanisms in model-based RL, explo- ration and policy optimization, to better handle reset-free training. Following the top row of Figure 2 to gather task-relevant data without resets, we define a training curriculum that alternates between temporally extended phases of task solving, resetting, and exploration. Next, as seen in the bottom row Figure 2 we bias the policy training within the world model towards achicving task-relevant goals such as reaching initial states and evaluation states.
Our key contributions are as follows: (1) We demonstrate the viability of using model-based agents with strong exploration abilities for the reset-free setting as well as their inherent limitations. We address such limitations through the MoReFree framework which focuses exploration and policy optimization on task-relevant states. (2) We evaluate the adapted reset-free version of MBRL baseline and MoReFree against state-of-the- art reset-free methods in 8 challenging reset-free tasks ranging from manipulation to locomotion. Notably, both mosel-based approaches outperform prior ttate-of-the-art baselines in 7/8 tasks in final performance and data efficiency, all the while requiring less supervision (e.g. environmental reward or demonstrations). MoReFree outperforms the model-based baseline in the 3 hardest tasks. (3) We perform in-depth analysis of the MoReFree and baselines behaviors, and show that MoReFree explores the state space thoroughly while
2

---

Published in Transactions on Machine Learning Research (02/2025)
retaining high visitation counts in the task-relevant regions. Our ablations show that the performance gains of MoReFree come from the proposed design choices and justify the approach.
1
(
Real
L: exploration policy
goall-cond. policy
Imagined
Practice Achieving Eval Goal
Practice Achieving Initial State
Practice Achieving Random State
sampled goal
P Po
PE D
Figure 2: MoRcFree is model-based RL agent for solving reset-free tasks. Top row: MoReFree strikes a balance between exploring unseen stales and practicing optimal behavior in lask-relevanl regions by directing the goal-conditional policy to achieve evaluation states, initial state states (emulating a reset), and exploratory goals. Bottom row: MoReFFee focuses the goal-conditioned policy training inside the world model on achieving evaluation states, initial states, and random replay buffer states to better prepare the policy for the aforementioned exploration scheme.
2
Related Work
There is growing interest in researching reinforcement learning methods that can effectively address the complexities of reset-free training. [harmaa et al. (2021b) proposes a reset-free RL benchmark (EARL) and finds that standard RL methods like SAC (Haarnoja et al.) 2018) fail catastrophically
Reset-free RL:
EARL. Multiple approaches have been proposed address reset-free training, which we now summarize.
One approach is to add additional reset policy, to bring the agent back to suitable states for learning
2023). LNT (Eysenbach) Eysenbach et al. 2017 Kim et al.) 2022 Sharma et al., 2021a 2022 Kim et al.
ct al. 2017) and Kim et al. (002)) train a reset policy to bring the agent back to initial state distribution, supervised by dense rewards and demonstrations respectively. MEDAL (Sharma et al.
2022 2023), train a goal-conditioned resct policy and direct it to reset goal states from demonstrations. IBC (Kmm et al. 2023
defines curriculum for both task and reset policies without. requiring demonstrations. VaPRL (Sharma let al.) 2021a) trains single goal-conditioned policy to reach high value states close the initial states. Instead of guiding the agent back to familiar states, R3L (Zhu et al.) 2020) and Xu et al. (2020) learn to reset the policy to diverse initial states, resulting in policy that is more robust to variations in starting states. However, such methods are limited to tasks where exploration is unchallenging. The vast majority of reset-free approaches are model-free, with few exceptions (Lu et al. 2020b a). Other works (Gupta et al.) 2021 Smith et al. 2019) model the reset-free RL training process as a multi-task RL problem and require careful definition of the task distribution such that the tasks reset each other.
Goal-conditioned Exploration: A common theme running through the aforementioned work is the instantiation of a curriculum, often through commanding goal-conditioned policies, to keep the agent in task-relevant portions of the environment while exploring. Closely related is the subfield of goal-conditioned exploration in RL, where a goal-conditioned agent selects its own goals during training time to generate data. There is a large variely of approaches for goal selection, such as lask progress (Baranes & Oudeyer
2013 Veeriah et al. 2018), intermediate difficulty (Florensa et al.] 2018), value disagreement (Zhang et al,
2020 , state novelty (Pong et al. 2019 Pitis et al.) (020)), world model error (Hu et al. 2023 Sckar al.
2020), and more. Many goal-conditioned exploration methods use the Exo-oxxlore" (Ecoffet al.) 2021] strategy, which first selects goal and runs the goal-conditioned policy ("Go"-phase and then switches
3

---

MoReFree Published in Transactions on Machine Learning Research (02/2025)
to exploration policy for the latter half of the episode ("Explore"-phase) PEG Huu et al. 2023). which MoReFree uses, extends Go-Explore to the model-based setting, and utilizes the world model to plan states with higher exploration value as goals. However, such methods are not designed for the reset-free RL setting, and may suffer from over-erploration task-irrelovant states.
Learned Reward Functions: Instead of requiring the environment to provide a reward function, the agent can learn its own reward function from onboard sensors and data. Given human specified example states, e.g. goal image, VICE and C-Learning train reward classifiers over examples (Fu et al.
2018
Eysenbach et al.
2021) and agent data. The learned dynamical distance function (Hartikainen et al. 2019 learns to predict the number of actions between pairs of states. The dynamical distance function is used by unsupervised MBRL approaches like LEXA and PEG (Mendonca et al. 2021 Hu et al. 2023) to train the goal-conditioned policy. MoReFree also employs the dynamical distance function as the reward function to eliminate the need of the environmental reward.
Table 1: A conceptual overview of reset-free methods. Existing methods are model-free, and most of them require other forms of supervision (environmental reward or demonstrations or both). In performance, MoReFree improves over reset-free PEG, which significantly outperforms privileged baselines IBC, MEDAL and R3L.
Approach Model-based Demonstrations Environmental reward
MEDAL
IBC
VaPRL
R3L
reset-free PEG
X
X
X
/
X
V
X
X
X
/
X
X
X
We notice that the majority of all prior work are model-free and may suffer from poor sample efficiency and exploration issues. In contrast, our model-based approaches use world models to efficiently train policies and perform non-trivial goal-conditioned exploration with minimal supervision. See Table | for a conceptual comparison between prior work and two model-based methods (MoReFree and reset-free PEG).
3
Preliminaries
3.1
Reset-free RL
We follow the definition of reset-free RL from EARL (Sharma et al. 2021b); and extend it to the goal-conditioned RI. setting. Consider the goal-conditioned Markov decision process (MDP) M (S, A, Po, 77) At each time step t in the state E S,, a goal-conditioned policy T(-|s,,9) g) under the goal command g E G selects an action a E A and transitions to the next state 8t+1 with the probability p($141l5t, a,); and gets a reward r(st, ar.g), g). Po the initial state distribution, Py is the evaluation goal distribution, and y the discount factor.
The learning algorithm A is defined: (si, at 1 + (ar, me), which maps the transitions collected until the time step & to the action a, the agent should take in the non-episodic training and the best guess m, of the optimal policy T' on the evaluation goal distribution (pg-). ). In reset-free training the agent will only be reset to the initial stale starts from 8p ~ Po, and is asked to achieve g ~ Pg-- The evaluation objective for a policy æ is:
Po few times. The evaluation of agents is still episodic. The agent always
J(m) E Exprpo oy ( p
Pr(ss,
(1)
1=
where T is the total time steps during the evaluation. The goal of algorithm & during the reset-free training is to minimize the performance difference D(A) of the current policy and the optimal policy n²:
D(A) -
(2)
In summary, the algorithm A should output an action a, that the agent should take in the non-episodic data collection and policy n, that can maximize J(xx) at every time step based all previously collected data.
4

---

Published in Transactions on Machine Learning Research (02/2025)
3.2 Model-based RL setup
Recent goal-conditioned MBRL approaches like LEXA (Mendonca el al. 2021 and PEG (Hu el al. 2023 train goal-conditioeed policies purely using synthetic data generated by learned world models. Their robust exploration demonstrates significant success in solving long-horizon goal-conditionod tasks. In the reset-free setting, strong exploration crucial, as the agent can no longer depend on episodic resets to bring it back to task-relevant. areas if it gets stuck. Therefore, we select PFG as the backbone MBRL, agent for its strong exploration abilities and sample efficiency.
PEG (Hu et al. 2023) is a model-based Go- Co-Explore framework that extends LEXA Mendoncaa et al. 2021 unsupervised goal-conditioned variant of DreamerV2 (Hafner et al. 20200. The following components are parameterized by 0 and learned:
world model: (81
goal conditioned policy: Th (arlst.g) exploration policy: n² (at
goal conditioned value: Vo (st,)) g) exploration value: (s+)s
(3)
The world model is recurrent state-space model (RSSM) which is trained to predict future states and is used as learned simulator to train the policies and value functions. The goal-conditioned policy my is trained to reach random states sampled from the replay buffer. The exploration policy Th is trained on an intrinsic motivation reward that rewards world model error, expressed through the variance of an ensemble (Sekar itt al. 2020 (see Appendix C.3 for more details). Both policies are trained on simulated trajectory rollouts in the world model.
Self-supervised Goal-reaching Reward Function: Rather than assuming access to the environmental reward, PEG learns its reward function. PEG uses a dynamical distance function (Hartikainen et al. 2019) as the reward function within world models, which predicts the number of actions between a start and goal state. The distance function is trained on random state pairs from imaginary rollouts of T To trained to minimize the dynamical distance between its states and commanded goal state in imagination. See Appendix C.I for more details.
T is then
Phased Exploration via Go-Explore: For data-collection, PEG employs Algorithm 1 Go- Go-Explore the Go-Explore strategy. In the "Go"-phasc, goal is sampled from some goal distribution p. The goal-conditioned policy, conditioned on the goal is run for some time horizon HG, resulting in trajectory Tg.
Input: gi T m
2: Tg f- {)iTe + 0
3: for 1= 1 to Ha do
Then, in the "Explore"-phasc, starting from the last state in the "Go"-phase, the exploration policy is run for HE steps, resulting in Te- The interleaving of goal- coaliconditioned behavior with exploratory behavior results in more directed exploration and informative data. This in turn improves accuracy of the world model, and the policies that train inside the world model. See Algorithm and A Algorithm 2 for pseudocode. The choice of goal distribution p is important 9:
4:
~ T (- (st,g)
5: St41 ~ T( |st, 0r)
6:
Tg + Tg Us})
7: end for
8: for t= 1 to HE do
a ~ n (. (st)
for Go- Explore In easier tasks, the evaluation goal distribution Pg- may be sufficient. But in longor-horizon tasks, evaluation goals may be too hard to achieve. Instead, intermediate goals from an exploratory goal distribution PE can help the agent explore. We choose PEG, which generates goals by planning 13:
10:
S111 T( |sc 01)
11:
t Te U {8t}
12:
end for
return Ty.
through the world model to maximize exploration value (see Appendix C.2 for details).
4 Method
As motivated in Section and Figure 1 the direct application of PEG to the reset-free setting shows promising performance but suffers from expreexpoonntion of inrelvrreeevant states. To adapt model-based RL to the reset-free sctting, we introduce MoRcFroe, modcl-basod approach that improves PEG to handle the lack of resets and overcome the overexxpooattin problem. MoReFree improves two key mechanisms of MBRL for reset-free training: exploration and policy training.
5

---

Published in Transactions on Machine Learning Research (02/2025)
4.1 Back-and- Forth Go- Go-Explore
First, we introduce MoReFree's procedure for collecting new datapoints the real environment. PEG Hu ct al. 2023) already has strong goal-conditioned exploration abilitics, but was developed for solving episodic lasks. Without resets, PEG's Go-Explore procedure can undesirably linger in unfamiliar but lask-irrelevanl portions of the state space. This generates large amounts of uninformative trajectories, which in turn degrades world model learning and policy optimization.
MoReFree overcomes this by periodically directing the agent to return to the states relevant to the task (i.e. initial and eval- uation goals). We call this exploration procedure "Back-and-Forth Go-Explore", where we sample pairs of initial and eval- uation goals and ask the agent to cycle back and forth between the goal pairs, periodically interspersed with exploration phases (see Figure
Algorithm 2 MBRL Backbone
Input: nf nf. world model To, goal distribution (including: exploratory goal distribution Pg, evaluation goal distribution Pgr, initial state distribution po)
2:
D +
3: while within the resct-free horizon do
4:
# reset-free PEG
5:
sample a goal g ~ PE
2 top row).
Tg. Te +Go (Goo-xpplre(gg) u. m
# MoReFree
Now, we define the "Back-and-Fort" Go- Explore" strategy as scen in Algorithm 3 First, we decide whether to perform ini- tial evaluation state directed explo- 10:
8:
Tg. + Bacck-aan-Forth Go-Explore(ag @ =0
9:
D D+DDu - DUT Te
update To with D update T with Ta in imagination
ration. With probability a, we sample 11:
update Tg with To imagination, cond. on goals g':
goals gg) from Pyt.Pu respectively. 12:
Then, we execute the Go-Explore routine 13:
g~D
for each goal. We name Go-Explore tra- 14: g~ with Pr= a, g pgt, with Pr=a
jectories conditioned on initial state goals 15: end while
"Back" trajectories, and Go-Explore trajectories conditioned on evaluation goals "Forward" trajectories. With probability 1 a, execute exploratory Go-Explore behavior by sampling exploratory goals from PEG. The difference between reset-free PEG and MoReFree can be seen in Algorithm 2, unlike PEG, MoReFreee employs the "Back-and-Forth Go-Explore".
By following this exploration strategy, the agent modulates between various Go-Explore strategies, alternating between solving the task by pursuing evaluation goals, resetting the task by pursuing initial states, and exploring unfamiliar regions via exploratory goals.
4.2 Learning to Achieve Relevant Goals in Imagination
Next, we describe how MoReFree trains the goal-conditioned policy in the world model. To train ng. MoReFree samples various types of goals and executes (- (-1 9) inside the world model to generate "imaginary" trajectories. The trajectory data is scored using the learned dynamical distance reward mentioned in Section 3.2 and the policy is updated to maximize the expected return. This procedure is called imagination (Hafner ct al. 2019), and allows the policy to be trained on vast amounts of synthetic trajectories to improve sample efficiency.
Algorithm 3 Back-and-Forth Go- Go-Explore
Input: # =E Pg Po. PE
Generate a random number r in [0, if
then
4:
g ~ Pgr
5:
To -e Go-Explore(gs" To mT)
6: # Continue from the terminal state
of the previous Go- Explore
Too +? + Go Go-Explore (go. T mg Tg + Ty- U T Ta Te U Te
First, we choose to sample evaluation goals from Pyt since the policy will be evaluated on its evaluation goal-rcaching else
performance. Nexi, recall that. Back-and-Forth Go-Explore 10: 9 PE
procedure also samples initial states from Pa as goals for the 11: Tg. Te + Go T
Go-phase to emulate resctting behavior. Since we would like G to succeed in such cases so that the task is reset, we will also sample from Po- Finally, we sample random states from the
12: end if
13: return Ty. Te

---

Published in Transactions on Machine Learning Research (02/2025)
replay buffer to increase TH's ability to reach arbitrary states. The sampling probability for each goal type is set to a/2, 0/22, 1 respectively. In other words, MoReFree biases the goal-conditioned policy optimization procedure to focus on achieving task-relevant goals (i.e. evaluation and imitial states), as they are used during evaluation and goal-conditioned exploration to condition the goal-reaching policy (sce Figure 2 bottom row).
This leads to additional changes of line 13 Algorithm
4.3 Implementation Details
Our work builds on the top of PEG (Hu et al., 2023). and we use its default hyperparameters for world model, policies, value functions and temporal reward function. We set the length of each phase for Go-Explore (Ha, Hg) half the evaluation episode length for each task. We set the default value of = 0.2 for all tasks (never tuned). See Appendix B.3 for more details and the supplemental for MoReFree code.
5
Experiments
We evaluate three MBRL methods (PEG (Hu et al. 2023 the extension reset-free PEG and our proposed method MoReFree) and four competitive reset-free baselines eight reset-free tasks. We aim to address the following questions: 1) Do MBRL approaches work well in reset-free tasks in terms of sample efficiency and performance? 2) What limitations arise from rumning MBRL in the reset-free setting, and does our proposed solution MoReFree address them? 3) What sorts of behavior do MoReFree and baselines exhibit in such tasks, and are our design choices for MoReFree justified?
Baselines: All baselines except for R3L are implemented using official codebases, see Appendix B.2 for details.
PEG (Hu al. 2023) is the original episodic PEG in which exploratory goals are only sampled once at the beginning of each episode (in the reset-free setting, the episode is extremely long). The goal-conditioned policy and the exploration policy are then executed for the first. half and second half of the episode, respectively.
reset-free PEG is straightforward extension of PEG the reset-free setting. Exploratory goals are sampled every HG + HE steps. Then, the goal-conditioned policy is executed for HG steps followed by the exploration policy being executed for Hg steps.
Dreamer V2 Hafner et al. 2020 commonly used MBRL method. The goal-conditioned policy executed for the whole resel-free episode.
MEDAL (Sharma et al. 2022) requires demonstrations and trains two policies, one for returning to demonstration states and another that achieves task goals.
IBC (Kim et al. 2023 is a competitive baseline that outperforms prior resct-free work (c.g. MEDAL, VaPRL) by defining a bidirectional curriculum for the coal-conditioned forward and backwards (i.e. reset) policies trained using the environmental reward.
R3L (Zhu et al.) 2020) trains two policies, one for achieving task goals and another that perturbs the agent to novel states. Notably, it is the only baseline that operates without any additional assumptions (i.c. environmental rewards, demonstrations, and resets).
Oracle is SAC (Haarnoja et al. 2018 trained under the episodic setting on the environmental reward.
Note that most baselines enjoy some advantage over two MBRL methods: MEDAL, IBC and Oracle use ground truth environmental reward, while MEDAL also uses demonstrations and Oracle uses resets. See Table D for a conceptual comparison between MoReeFee and prior work.
Environments: We evaluate MoReFree and baselines on eight tasks (see Figure 3 We select five tasks from IBC's evaluation suite (Kim et al. 2023) of six tasks; (PointUMave) Tabletop, Sawyer Door, Fetch
7

---

Published in Transactions on Machine Learning Research (02/2025)
PointUMaze
Tabletop
Sawyer Door
Fetch Push and
Push and PP (hard)
Figure 3: We evaluate MoReFree eight reset-free tasks ranging from navigation to manipulation. PP is short for Place
Push and PP, Fetch Reach is omitted because it is trivially solvable). Next, we increased the complexity of the two hardest lasks from IBC, Fetch Push and Fetch Pick&Place, by exlending the size the workspace, replacing artificial workspace limits (which cause unrealistic jittering behavior near the limits, see the website harder goal states (i.c. Pick&&Placo goals only in the air rather
for vidcos) with real walls, and ovaluating than including ones the ground). In addition, contributed difficult locomotion task, Ant, which is adapted from the PEG codebase
(Hu et al. 2023
UMaze:
A point-mass agent navigates U-shape maze through continuous acceleration commands. During evaluation, the agent starts from the bottom-left corner and is tasked to reach the lop-left corner.
The agent needs to grab and move the mug to one of the four goal locations. The initial state always fixed and the goal state is uniformly sampled from four fixed locations.
Tabletop Manipulation:
The agent controls Sawyer robot arm to close the door in an open position. During the feset-free training, it needs to learn to close the door and open the door again to practice. The door is opened 60 degrees for evaluation.
Sawyer Door:
Fetch Push&PP: The agent commands a Fetch robot arm to push pick&place the object initialized at the center of the table to goal locations. The environment taken from IBC's evaluation suite, which modified the original environment from Plappert et al. (2018). To prevent the block from falling off the table, the IBC authors artificially limited the block position with block position constraints.
(rash&PP(harr)) Using block position constraints (in Fetch Push&PPP resulted in unrealistic jittering behavior near the limits. To avoid this, we removed the artificial joint constraints and surrounded the table with physical walls. Furthermore, we enable the usage of the grippers (disabled in IBC's version) to permit picking behaviors (i.e. useful for resetting), the cost of increased action space and exploration difficulty.
Ant: The 4-legged ant agent needs to navigate a square room to given goal, which uniformly located the top-right corner. The initial state at the center point with randomness. is adapted from Hu al. (2023). with changing the U-shape maze into a square room.
Most methods are run with 5 seeds, and the mean performance and standard error are reported. During the ovaluation, the performance on tasks with randomly sampled goals from Pyr is mcasured by averaging over 10 episodes. See Appendix for more experimental details.
5.1 Results
As shown in Fig Iwo reset-free model-based methods (MoReFree and reset-free PEG), without demonstra- tions or access to environmental reward, outperform other baselines with privileged access to supervision in both final performance and sample efficiency 7/8 tasks. We observe that the two reset-free MBRL methods learn good behaviors: the pointmass agent hugs the wall of the UMaze to minimize travel time and the Fetch robot deftly pushes and picks up the block into multiple target locations. MoReFree is always competitive
8

---

Published in Transactions on Machine Learning Research (02/2025)
1.0
1.0
1.0
1.0
MoReFree
Success r220
M
reset-free PEG
0.5
0.5
0.5
PEG
0.2
0.2
1.4
2.0
0.4
1.6
2.8
0.5
3.0 5.5
DreamerV2
Steps (1e5)
Steps (1e5)
Steps (1e5)
Steps (1e5)
PointUMaze
Tabletop
Sawyer Door
Push
IBC
1.0
1.-
MEDAL
R3L
0.5
0.5
0.5
oracle (200k)
0.5 4.5
12.5
0.5
3.0
8.0
0.0
0.0
Steps (le5)
Steps (le5)
Steps
Pick&Place
Push (hard)
Pick&Place (hard)
Ant
Figure 4: Two reset-free MBRL methods (MoReFree and reset-free PEG) significantly outperform baselines in 7/8 tasks. However, directly applying MBRL methods (PEG and DreamerV2) works poorly. In tasks, only MBRL methods are able to learn meaningful behavior, showcasing MBRL's sample efficiency in the reset-free setting. MoReFree outperforms reset-free PEG in the 3 most difficult tasks.
with or out purperfomm reset-free PEG, with large gains in the 3 hardest lasks: Push (hard) by 45%, Pick&&lace (hard) by 13% and Ant (hard) by 36%. We observe that MoReFree learns non-trivial reset behaviors such picking and pushing blocks back into the center of the table for the hard variants of the Fetch manipulation tasks. However, the original PEG performs poorly, suggesting that directly applying episodic MBRL methods in a reset-free setting without adaptations yields suboptimal results. See the website for videos of MoReFree and baselines.
In many tasks, the baselines fail to learn at all. We believe this is due the low sample budget, which may be too low for the baselines to fully explore the environment and learn the proper resetting behaviors necessary to train the actual task policy. In Appendix G increased the training budget by 3x for the IBC bascline and it still fails, underscoring the difficulty of the lasks and the sample-effciinccy gains of MoReFree and MBRL. On the other hand, we noticed that one environment, Sawyer Door, seemed particularly hard for MBRL agents to solve. We hypothosize that the dynamics of the task are hard to model, resulting in performance degradation for model- based approaches (see Appendix for more analysis).
5.2 Analysis
To explain the performance differences between MoReFree and baselines, we closely analyze the exploration behaviors.
MoReFree focuses on task-relevant states. In Figure we visualize the state visitation heatmaps of methods in various environments, and also compute the percentage of "task-relevant" states (initial and goal regions, highlighted with white borders). We highlight two trends. First, the heatmaps show that MoRcFree and reset-free PEG explore thoroughly while baselines have more myopic exploration patterns, seen in the Ant heatmaps at the top.
Next, performance differences between resct-free PEG and MoReFree are intuitively explained by the amount of task-relevant data collected by each agent. In easier environments like Push or Pick&Place where both reset-free PEG and MoReFree encounter similar amounts task-relevant states, the performance roughly similar between reset-free PEG and MoRcFree. But in harder environments (Ant, Push (hard), Plck& (hard)) with larger state spaces and more complicated resetting dynamics, MoReFree collects 1.3 5x more fask-relevant data and has large performance gains over reset-free PFG. By experiencing more task-relevant.

---

Published in Transactions on Machine Learning Research (02/2025)
Ant
6.1%
1.3% O
1.0
MoReFree
reset-free PEG
PEG
Drv2
IBC
MEDAL
R3L
Push
Pick&Place
0.6
83:6%
83.6%
79.7%
74.5%
Push (hard)
Plac& (hard)
0.2
22.7% %
11.4%
13.2%
9.8%
MoReFree
reset-free PEG
MoReFree
reset-free PEG
Figure 5: State visitation heatmaps different agents. White arcas are task-rclevant states (including initial and goal state distributions) and overlay the percentages of task-relevant states, reset-free MBRL methods explore more and in harder environments, MoReFree experiences more task-relevant states.
states and training policies them in imagination, MoRcFree policies more suited towards succeeding at the episodic evaluation criteria. See Appendix D for additional visualizations.
MoReFree effectively resets. Next, we investigate the qualitative behavior of MoReFree's Back-and Bock-and-FFrrte Forth Go-Explore. To see "Back" trajectories help free the agent from the sink states, we analyze the replay buffer of MoReFree for the environments, and plot the starting locations of the agent / object up to 100 timesteps before successful "Back" trajectory is exccuted in Figure The color intensity of the dots correspond state density over the last 100 steps (i.e. dark red means the agent object has rested there for a while). We observe that the starting locations (red dots) of the agent / object in corners or next to walls in all environments. This suggests that these areas act sink states, where the agent object would remain for long and waste time. We obscrve that MoReFree learns reset behaviors like picking the block out of corners and walls in Fetch Push and Fetch Pick& Place. See detailed videos of the reset. behavior on the website.
PointUMaze
Tabletop
Push (hard)
Pick&Place (hard)
Ant
Figure 6: We visualize the start position (red dots) of successful "Back" trajectorics of MoReFree's Back-and- Forth Go-Explore, where T to state density over the last 100 steps.
G is directed to reset the enviromment. The color intensity of the dots correspond
10

---

Published in Transactions on Machine Learning Research (02/2025)
5.3 Ablations
To justify our design choices, we ablate the two mechanisms of MoReFree, the back-and-forth exploration and task-relevant goal-conditioned policy training, and plot the results in Figure
First, removing all mechanisms (MF w/o Explore & Imag.) reduces reset-free PEG, and we can see a large gap performance. Next, MF with Only Task Goals sets 1, which causes an extreme bias towards lask-relevant states in the exploration and policy training. This also degrades performance, due to the need for strong exploration in the reset-free setting. Examinations of more values for be found in Appendix B.3
Finally, we isolate individual components of MoRe- Free. First, we disable Back-and- Forth Go-Explore by disallowing the sampling of imitial or evaluation goals during Go-Explore. Only exploratory goals are used in Go-Explore for this ablation (named MF w/o BF-GE). Next, in MF w/o Imag. turn off the imitial / evaluation goal sampling in imagination, so only random replay buffer goals are used to train Tz. We see that both variants perform poorly. This is somewhat intuitive, as the two components rely on each other. Having both forms a synergistic cycle where 1) the goal-conditioned policy's optimization is more focused towards reaching initial goal stales, and 2) the exploration is biased towards reaching initial goal states by using the goal-conditioned policy we just optimized in step 1. If we remove one without the other, then the cycle breaks down. In MF w/o Imag., Back-and-Forth. Go- Go-Explore will suffer since Th trained on random goals cannot reliably reach initial evaluation goals. In MF w/o BF-GE, the exploration strategy will not seek initial evallaaiinn states, resulting in inaccurate world model and degraded policy optimization. In summary, the ablations show that MoReFree's design is sound and is the major factor behind its success in the reset-free setting, See Appendix for details.
1.0
MoReFrec (MF)
w/o Explore Imag.
0.8
with Only Task Goals
MF w/o Imag.
MF w/o BF-GE
0.6
0.4
0.2
0.0
Figure 7: Ablations on 5 variants of MoReFree over 3 hard environments, Push (hard), Pick&Placce (hard) and Ant, with normalized final performance.
6 Conclusion and Future Work
As step towards reset-frec training, we adapt model-based methods the resct-free setting and demonstrato their superior performance. Specifically, show that with minor modifications, unsupervised MBRL method substantially outperforms the state-of-the-art model-free baselines tailored for the reset-free setting while being more autonomous (requires less supervision like environmental reward or demonstrations). We then identify a limitation of unsupervised MBRL in the reset-free setting (over-exploration on task-irrelevant states), and propose MoReFree to address such limitations by focusing model-based exploration and goal-cond coal-conditioned policy training on task-rclevant states. We conduct a thorough experimental study of MoRcFree and basclines over tasks, and show considerable performance gains over the MBRL baseline and prior stale-of-the-art reset-free methods.
Despite its overall success, MoReFree is not without limitations. Being a model-based approach, it inherits all associated disadvantages. For example, we believe Sawyer Door is a task where learning the dynamics is harder than learning the policy (see Appendix F), disadvantaging MBRL approaches. Next, MoReFree uses a fixed percentage of task-rolevant goals for exploration and imagination, whereas future work could consider an adaptive curriculum. Finally, scaling MoReFree high-dimensional observations and real-world applications would be natural extensions. We hope MoReFree inspires future efforts in increasing autonomy in RI.
11

---

Published in Transactions on Machine Learning Research (02/2025)
References
Adrien Baranes and Pierre-Yvel Oudeyer. Active learning of inverse models with intrinsically motivated goal exploration in robots. Robotics and Autonomous Systems, 61(1):99-73, 2013. ISSN 0921-8890. doi: http:///.......00//oooooo....... p11/8092188902206444
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation.
arXiv preprint arXi::1810 12894, 2018.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth Stanley, and Jeff Clunc. First return, then explore. Nature, 590(7847):580 586, 2021.
Benjamin Eysenbach, Shixiang Gu, Julian Ibarz, and Sergey Levine. Leave no trace: Learning to reset for safe and autonomous reinforcement learning. arXiv preprint arXie:1711.06788, 2017.
Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. C-learning: Learning to achieve goals
via recursive classification. In International Conference Learning Represenlations, 2021. URL https
//openreview.neteo wtt..tteo/oooooiitotciiiiiee.e
Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. In International conference on machine learning, pp. 1515-1528. PMLR, 2018.
Justin Fu, Avi Singh, Dibya Ghosh, Larry Yang, and Sergey Levine. Variational inverse control with events: A general framework for data-driven reward definition. Advances in neural information processing systems, 31, 2018,
Abhishek Gupta, Justin Yu, Tony Z Zhao, Vikash Kumar, Aaron Rovinsky, Kelvin Xu, Thomas Devlin, and Sergey Levine. Reset-free reinforcement learning via multi-task learning: Learning dexterous manipulation behaviors without human intervention. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 6664-6671. IEEE, 2021.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Schoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupla, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint XrXiv: 11X:::111200355 05900 2018.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning bchaviors by latent imagination. arXiv preprint 1rXiv:1912.2 01603, 2019.
Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint 1XXw:2010002193, 2020.
Siddhant Haldar, Jyothish Pari, Anant Rai, and Lerrel Pinto. Teach robot to fish: Versatile imitation from one minute of demonstrations. arXiv preprint 3rXiv:2308.01499 2023.
Kristian Hartikainen, Xinyang Geng, Tuomas Haarnoja, and Sergey Levine. Dynamical distance learning for semi-supervised and unsupervised skill discovery. arXiv preprint 1rxii:11077008 00825. 2019.
Edward S. Hu, Richard Chang, Oleh Rybkin, and Dinesh Jayaraman. Planning goals for exploration. In The Eleventh International Conference on Learning Representations, 2023. URL httpiss///ppenreiieee.eett
forum/id--6eBuuz507P
Jigang Kim, J hycon Park, Dacsol Cho, and H Jin Kim. Automating reinforcement learning with example-basod resets. IEEE Roboties and Aulomation Letters, 7(3):6606 6613, 2022.
Jigang Kim, Daesol Cho, and H Jin Kim. Demonstraatinn-free autonomous reinforcement learning via implicit and bidirectional curriculum. In International Conference on Machine Learning. PMLR, 2023.
Sergey Levine, Chelleaa Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 1737)31334-1373, 2016.
12

---

Published in Transactions on Machine Learning Research (02/2025)
Kevin Lu, Aditya Grover, Picter Abbeel, and Igor Mordatch. Reset-free lifelong learning with skill-space planning. orXiv preprinl arXiv:2012.0055 2020a.
Kevin Lu, Igor Mordatch, and Pieter Abbeel. Adaptive online planning for continual lifelong learning, 2020b. URL
Russell Mendonca, Oleh Rybkin, Kostas Danilidiss, Danijar Hafner, and Deepak Pathak. Discovering and achieving goals via world models, 2021.
Anusha Nagabandi, Kurt Konolige, Sergey Levine, and Vikash Kumar. Deep dynamics models for learning dexterous manipulation. In Conference on Robol Learning, pp. 1101 1112. PMLR, 2020.
Silviu Pitis, Harris Chan, Stephen Zhao, Bradly Stadie, and Jimmy Ba. Maximum entropy gain exploration for long horizon multi-goal reinforcement learning. In International Conference on Machine Learning, pp- 7750-7761. PMLR, 2020.
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas
Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, et al. Multi-goal reinforcement learning: Challenging robotics environments and request for rescarch. arXiv preprint arXiv: 1801.8800444.. 2018.
Vitchyr H Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew-fit: State-covering sule-super reinforcement learning. arXiv preprint Xrxi: 03698, 2019.
Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Halner, and Deepak Pathak. Planning explore via self-supervised world models. In 1CML, 2020.
Archit Sharma, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Autonomous reinforcement learning via subgoal curricula. Advances in Neural Information Processing Systems, 34:18474 18486, 2021a.
Archit Sharma, Kelvin Xu, Nikhil Sardana, Abhishek Gupta, Karol Hausman, Sergey Levine, and Chelsea Finn. Autonomous reinforcement. learning: Formalism and benchmarking. arXiv preprint 2rXiv:2112.9600: 2021b.
Archit Sharma, Rehaan Ahmad, and Chelsea Finn. A state-distribution matching approach to non-episodic reinforcement learning. arXiv preprint Xr:::22005.000 05212, 2022.
Archit Sharma, Ahmed M Ahmed, Rehaan Ahmad, and Chelsea Finn. Self-improving robots: End-to-end autonomous visuomotor reinforcement learning. arXiv preprint arXiv:2303.01488 2023.
Laura Smith, Nikita Dhawan, Marvin Zhang, Pieter Abbeel, and Sergey Levine. Avid: Learning multi-stage 2019.
tasks via pixel-level translation of human vidcos. arXiv preprint
Vivek Veeriah, Junhyuk Oh, and Satinder Singh. Many-goals reinforcement. learning. ArXiu, abs/ 19801006099605 2018.
Kelvin Xu, Siddharth Verma, Chelsea Finn, and Sergey Levine. Continual learning of control primitives: Skill discovery via reset-games. Advances in Neural Information Processing Systems, 33:4999-5011 2020.
Ali Yahya, Adrian Li, Mrinal Kalakrishnan, Yevgen Chebotar, and Sergey Levine. Collective robot rein- forcement learning with distributed asynchronous guided policy search. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 79-86. IEEE, 2017.
Yunzhi Zhang, Pieter Abbeel, and Lerrel Pinto. Automatic curriculum learning through value disagreement. Advances in Neural Information Processing Systems, 33:7648 7659, 2020.
Henry Zhu, Abhishek Gupta, Aravind Rajeswaran, Sergey Levine, and Vikash Kumar. Dexterous manipulation with deep reinforcement learning: Efficient, general, and low-cost. In 2019 International Conference on Robotics and Automation (ICRA), pp. 3651-3657 IEEE, 2019.
Henry Zhu, Justin Yu, Abhishek Gupta, Dhruv Shah, Kristian Hartikainen, Avi Singh, Vikash Kumar, and Sergey Levine. The ingredients of real-world robotic reinforcement learning. arXiv preprint ar 2rxi 2020.
13

---

Published in Transactions on Machine Learning Research (02/2025)
A Broader Impacts
As we increase the autonomy of RL agents, the possibility of them acting in unexpected ways to maxinize reward increases. The unsupervised exploration coupled alongside the learned reward functions further add to the unpredietability; neither mechanisms are very interpretable. As such, we expect research into value alignment, interpretability, and safety to be paramount as autonomy in RL improves.
B Experimental Details
B.1 Environments
Point
The state space is 7D and the action space is 2D. The initial state is (0, 0), which located in the bottom-left corner, and noise sampled from ((-00110.1) is added when reset. The goal during the evaluation is always located in at the top-left corner of the U-shape maze. The maximum steps during the evaluation is 100. Hard reset will happen after every 2c5 steps. In the whole training process we performed, it only reset once at the beginning of the training. Taken from the IBC {Kim et al. 2023) paper.
Tabletop:
The state space is 6D, and the action space is 3D. During the evaluation, four goal locations are sampled in turn, the initial state of the agent is always fixed and located in the center of the table. The maximum steps during the evaluation is 200. Hard reset will happens after every 2e5 steps. In the whole training process we performed, it only reset once at the beginning of the training. Taken from the EARL [hharma el al. 2021b) benchmark and also used in the IBC paper.
The state space is 7D and the action space is 4D. The position of door is initialized to open state (60 degree with noise sampled from 18) degree) and the goal is always to close the door (0 degree). The arm is initialized to fixed location. Maximum number of steps is 300 for the evaluation. Hard reset will happen after every 2e5 steps. In the whole training process we performed, it resets twice. Taken from the EARL (Sharma et al. 2021b) benchmark and also used in the IBC paper.
Sawyer Door:
Fetch Push and Pick&PPlace The state space is 25D and action space 4D. These are taken from the IBC paper. Authors converted the original Fetch environments a reversible setting by defining a constraint on the block position. The initial and goal distributions are identical to the original Fetch Push and Pick& More details can be found in the IBC paper.
Push (hard): Different from the original Fetch Push task, in our case walls are added to prevent the block from dropping out of the table. The workspace of the robot arm is also limited. The block is always initialized to a fixed location, and goal distribution during the evaluation is 21-00.15, 15). Fetch Push used in the IBC paper, the block limited by joint constraint, which shows unrcalistic jittering behaviors near the limits (we observe such phenomenon by running model-based go-explore, the exploration policy prefers to always interact with the block and keep pushing it towards the limit boundary, see videos on our project website D. Meanwhile, the gripper is blocked, which makes the task easier. In our case, we release the gripper and it can now open and close again which add two more dimension of the state space. We found it is important to release the gripper in our version of Push task, when the block is in corners, it will need to operate the gripper to drag the block escape from corners. The maximum steps the agent can take in 50 during the evaluation. Hard reset will happen after every 1e5 steps. In the whole training process we performed, it resets times in total.
Pick&Place (hard): We add walls in the same way as we did for Push (hard). We make it more difficult by only evaluating the agent on goals that are in the air. Then it has to learn to perform picking behavior properly, whereas goals on the ground can just be solved by pushing. The goal will be uniformly sampled from a 5 x 5 x 10 em cubic area above the table. It has the same observation space, action space, initial state and maximum steps with Fetch Push described above. Hard reset will happens after every 1e5 steps. In the whole training process we performed, resets 5 times in total. See the visual difference between our Pick&Placee and IBC's in Figure 3 Since the workspace of the robot is limited within the walls as well in Push (hard) and Pick&PPacee (hard), when the block gets stuck in corners, the robot needs to preciscly move
htttps://yangzhhao-6e 666. 666.66666..88 wwt.lb....t.b......e.. io/norefree
14

---

Published in Transactions on Machine Learning Research (02/2025)
to the corner and bring the block back. In contrast, the robot in IBC's version can move to everywhere, being able to create various circumstance to solve such difficult position.
Ant: We adapt the AntMaze task from environments codebase of PEG and change the shape of the maze to square, also change the evaluation goal distribution to be a uniform distribution 4(2,3) 3) for both x and y location, which lies on the top-left corner of the square. The ant is always initialized to the center point (0, 0) of the square to start from, with uniform noise (01-),,00.1) added. The state space is 29D and the action space is 8D. The maximum steps for evaluation is 500. Hard reset will happen after every 2c5 steps. In the whole training process we performed, it reset 4 times in total.
B.2 Baseline Implementations
PEG: We use the official implementation of PEG and only optimize the exploratory goal distribution once al the beginning each reset-free training episode, i.e. Hg and HE are set to half of the reset-free episode length.
We extend the official implementation of PEG to resct-free setting by 1) set He and HE to hall of the evaluation episode length; 2) optimizing the goal distribution every HG + Hg steps; 3) keeping all other hyperparameters the same as MoReFree.
reset-free PEG:
IBC: We use the official implementation from author- F and keep hyperparameters unchanged.
We use the official implementation of PEG. In order reduce it to DreamerV2 (Hafner et al. 2020), we remove the exploration policy and only execute goal-conditioned policy for the whole reset-free episode. During imagination training, the goal-conditioned policy is only trained the evaluation goal distribution.
Dreamer V2:
MEDAL: We follow the official implementation of MEDAI and use the dcafult setting for experiments. Since MEDAL requires demonstrations, for tasks from EARL benchmark, demonstrationn ratistataios are provided. For other environments, we generate demonstrations by executing the final trained MoReFree to collect data. 30 episodes are generated for each task.
R3L: We implement R3L agent by modifying the FBRL agent from MEDAL codebase. The backward policy is replaced by an exploration policy trained using the random network distillation (RND) objective (Burda
et al. 2018).
The RND implementation we follow is from DI-engine
Oracle: This is a episodic SAC agent, we use the implementation from MEDAL codebase and keep all the hyper-parameters unchanged.
MoReFree: Our agent is built on the model-based go-explore method PEG (Hu et al., 2023): we extend their codebase by adding back-and-forth. goal sampling procedure and training on evaluation initial and goal states in imagination goal-conditioned policy training. See our codebase in the supplemental.
B.3 Hyperparameters
Train ratio (i.e. Update to Data ratio) is an important hyper-parameter in MBRL. It controls how frequently the agent is trained. Every n steps, a batch of data is sampled from the replay buffer, the world model is trained on the batch, and then policies and value functions are trained in imagination. we only vary on different tasks. See the table below for different values on different tasks we used through experiments. MoReFree also introduces
all our experiments,
parameter a, which keep a 0.2 for all tasks and did not tune it at all. All other hyperparameters we keep the same the original code base.
httpp://githuu: wtt.//oo///eeeee edmp//dwhu/mr
https://github.. cttp:/.compeeeeeeeeeeeee
https://github.com/st
httpe///sthuu..c//rrratthaaama/
html
15

---

Published in Transactions on Machine Learning Research (02/2025)
Table 2: Different train ratio we used for different tasks. We keep other hyperparameters the same as default ones.
Maae
Sawyer Door 5
Tabletop
Fetch Push 2
Fetch Pick&Place
2
Push (hard)
2
Pick&Place (hard)
Ant 2
Different values for a. We examine different values of a in MoReFree on Fetch Push task, which affects how much MoReFree focuses on task-relevant goals exploration and imagination. In Figure g we see that introducing a moderate amount of task-relevant goals (a=0.2, a=0.5) results in sensible performance, while too many Lask-relevant goals (o=0.7, a=1.0) degrades performance. We use the value of alpha, 0.2, across all tasks, which showcases MoReFiree 's consistency.
B.4 Results Clarification
In Push and Pick&Placce results, retrieved the final performance of MEDAL directly from the IBC paper (dashed purple lines) and did not have time to run R3L in these two environments. R3L is shown to be lot worse than MEDAL in the MEDAL paper and performs obviously bad in other tasks shown in Figure 4 In Push (hard) and Pick&Place (hard), we ran R3L and MEDAL with less budget since other methods clearly outperform and their learning curves do not show any evidence for going up.
2-60
2-0.5
8-07
0.50
B.5 Resource Usage
0.25
We submit jobs on a cluster with Nvidia 2080, 3090 and A100 GPUs. Our model-based experiments take 1-2 days to finish, and the model-free baselines take half day to one day to run.
0.00
0.5
2.0
3.5
5.0
Steps (1e5)
C Method Details
Figure 8: Performance of MoRe- Free with different values of in Push (hard).
Here, we provide a more in-depth exposition self-supervised goal-reaching reward function (Hartikainen al. 2019) that is used for training the goal-conditioned policy, PEG (Hu et al. 2023) that is used for generating exploratory goal distribution, and P2E
(Sckar et al. 2020 that is used for training the exploration policy.
C.1
Self-supervised Goal-reaching Reward Function
MoReFree does not require environment reward functions, instead learns distance function dy for training the goal-conditioned policy. d, is trained by sampling pairs of states 8t, 8t4k from imagined rollout of the goal-conditioned policy and predicting the distance k/H, where H is the maximum distance equal to the imagination horizon. Then the reward is defined r(st, g) = -d (st,
C.2 Exploratory Goal Distribution
We PEG to generate the exploratory goal distribution pr in MoReFree. PEG generates goals that have high exploration potentials. To evaluate goal g. the goal-conditioned policy is rolled out for K trajectories within the learned world model. Then the terminal state exploration value for each trajectory with the learned exploration value function E (s4) is estimated, where s is the last state of the trajectory тІ- Then the estimates are averaged.
The goal variable g is optimized using model predictive path integral control (MPPI). First, N goal candidates g are sampled from an initial distribution. These candidates are then evaluated as described above. This averaged exploration value acts the "score" for the goal candidate. Once have scores for each goal
16

---

Published in Transactions on Machine Learning Research (02/2025)
MoReFree reset-free PEG
IBC
MEDAL R3L
Oracle
0.0
0.2
0.8 1.0
Figure 9: XY state visitation heatmap of the mug in Tabletop of various approaches. MoReFree's heatmap shows high state diversity while retaining high visitation counts near the task-relevant states (red circles are goal states, the blue circle is the initial state). reset-free PEG also shows diverse exploration, but it over-explores the bottom-right corner which entirely lask-irrelevant. IBC's bi-directional curriculum leads the exploration shuttles between the initial state and goal states, but fails explore well. All other methods fnil to explore, visited states mostly cluster in few spots.
candidate, a Gaussian distribution fit according to the rule:
=e== (gne)
H4
Ex=o(e
(4)
where y is the reward weight hyperparameter. We then sample candidates from the computed Gaussian, and repeat the process for multiple iterations. After the last itcration, Pe is defined as the computed Gaussian.
C.3 Plan2Explore
The world model to consists of the following components:
encoder: e = enc)((+)
posterior: (1
(5)
dynamics: (s(st 91-1)
decoder: (=
The model states 8, contain a deterministic component h, and a stochastic component 24
P2F is the objective we used train the exploration policy, and encourages the agent visit states that can improve the world model the most. We train an ensemble of 1-step models predict the next model state from the current model state:
Ensemble:
fsr. = #t+1
for L...K
(6)
Then the exploration reward is the variance of the ensemble predictions averaged across dimension of the model state, r(s.) = * Σn Var 0x)
More Visualizations on Replay Buffer
We visualize the replay buffer of different agents on more tasks. See Figure 9 for XY location of the mug in Tabletop, Figure T1 for XY location data of the agent in PointUMaze, Figure TO for X7 location of the block in Pick&Place (hard) and Figure 12 for XY location data of the block in Push (hard) and Pick& (hard). Overall, we see MoReFree explores the whole state space better. Meanwhile, due to back-and-forth procedure, MoReFree collects more data near initial / goal states, which are important for the evaluation. However, IBC, MEDAL, R3L and Oracle all fail to explore well; their heatmaps are mostly populated with low visitation cells.
E Detailed Ablations
We report learning curves for each variant agent we ablate Section 5.3 on every task Figure I3 Since MoReFree does not learn at all in Saywer Door task, exclude the ablation for it. In each task, MoReFree
17

---

Published in Transactions on Machine Learning Research (02/2025)
MoReFree
reset-free PEG
IBC
MEDAL
R3L
Oracle
Figure 10: XZ state visitation healmap of the block in Pick&Place (hard). States above the red line are in the air, which are crucial for solving the picking task. Two MBRL methods collect more data diversely in the air, while other reset-free methods barely pick up the block.
MoReFree
MEDAL
Oracle
Figure 11: State visitation heatmap point maze. MoReFree has special focuses both initial state (blue circles) corner and goal state (red circles), while explore much uniformly. MEDAL collects lots of data near the goal state and little data on the initial state. Both MEDAL and Oracle explore less extensively.
MoReFree
IBC
MoReFree
IBC
MEDAL
Oracle
MEDAL
Oracle
Figure 12: Block state visitation heatmap on Fetch Push (left) and Fetch Pick&Place (right) of different agents. MoReFree better explores the whole state space, while IBC and MEDAL do not have too much interactions with the block, thus lighted areas are scattered everywhere.
is better par with all other ablations. Through learning curves, see different components contribute differently on different tasks.
We further analyze the ablation on UMaze as an example by visualizing the replay buller of different. variants, see Figure [4] In the performance on PointUMaze from Figure [3] sampling exploratory goals for data collection important (MF w/o Explore Imag. outperforms other ablations). But we see
MF w/o Explore Imag. does not have focus on the initial goal state which we care about for the evaluation, which makes it slightly worse than MoReFree. MF with Only Task Goals has strong preference
18

---

Published in Transactions on Machine Learning Research (02/2025)
1.0
1.50
1.00
1.00
2.00
0.75
0.75
0.75
0.55
0.75
030
0.50
0.30
0.50
032
025
0.25
0.5
025
0.5
0.00
2.2
0.00
2.0
0.5
2.5
0.02
64
Stepe 13e5;
Steps (1e5)
Steps (le5)
Steps (1e5)
Steps 11e51
PointUMaze
Tabletop
Push (hard)
Pick&Place (hard)
Ant
Figure 13: Learning curves of ablation study on 5 tasks. We see different components contribute differently in different tasks. For instance, in Tabletop, MF w/o Imag, even performs better than MoReFree, maybe because the whole state space can be explored quickly, then randomly sampling states from the replay buffer as goals for training already has good coverage on evaluation initial / goal states.
on initial / goal state, we think it because the later phase of the training when the agent able to solve the task, it goes back-and-forth consistently to collect data. But in the early phase of the training, it might lack exploration which causes the degraded performance compare with MoReFree. MF w/o Explore and MF w/o Imag. only cither go to initial goal state for data collection and do not practice it during the imagination training, or practice without really going, which both does not form the positive cycle, and end with poor performance.
MoReFree (MF)
MF w/o Explore & Imag.
MF with Only Task Goals
MF w/o Explore
MF w/o Imag.
Figure 14: State visitation heatmap on Point task of all ablations. Red circles are evaluation goal states and blues are initial states. We see MoReFree collect good amount of data near initial / goal states while stronger exploration. MF w/o Explore and MF w/o Imag. could not gather task-relative data, which further causes poor performance.
F MBRL on Sawyer Door
We investigate why two MBRL methods fail on Sawyer Door tasks. Note that MoReFree is able to solve intermediate goals such as closing the door in some angles, but is unable to solve the original IBC evaluation goal (see website for more videos).
We simplify Sawyer Door task by limiting the movement range of the robot to box and also having a block holds the door to prevent it from opening it too much, Figure 15 Although MBRI. methods are
19

---

Published in Transactions on Machine Learning Research (02/2025)
trained the simplified environment, see learning curves Sawyer Door are completely flat in Figure compared with other baselines trained on the original task. We wonder why MBRL methods can show the same performance and gain benefits as it does in other environments.
MoReFree and reset-free PEG use DreamerV2 V2 as backbone agents and extend it to reset-free settings. We hypothesize that Dreamer itself, even under the episodic setting with task reward function, would not work well. If that's the case, then MBRL methods in the reset-free setting with self-supervised reward function would almost certainly not work either. For example, if the backbone agent cannot model the dynamics precisely, then policy learning, dynamical distance reward learning, will be degraded.
Figure 15: Simplified version of Sawyer Door. Orange walls show the limited workspace for the robot arm, and a grey wall is added limit the movement of the door. The door can only move to maximum degrees.
0.5
DreamorV2 Breamer33
0.0
0.5
3.5
6.5 9.5
Steps (1e5)
Figure 16: Performance of DreamerV2 V2 and V3 on episodic Sawyer Door task. SAC can solve the task in 200k steps, while after 1 million steps MBRL is still not able to steadily solve the task.
We then run the underlying MBRL backbones under the episodic setting. Figure shows Dreamer V2 5 and Dreamerv3 F struggle solve the task, while model-free method SAC can steadily solve the task after 200k steps. This might be potential reason that MBRL methods do not work on the more difficult reset-free setting. We hypothesize that the combination of the sparse environmental reward and dynamics of the door result in hard prediction problem for world modelling approaches. We leave further investigation for the future work.
G More Analysis on Fetch Environments
Although IBC gains good final performance in Push and Pick& it starts learning late compared with MBRL methods and fails entirely in our harder versions. We suspect IBC might need more computational budget to start learning in harder tasks. Thus we train IBC with two millions environment steps and results in Figure 17 show that it still fails to solve the harder version of Push.
http:://gsthub.coo//aaiin
20

---

Published in Transactions on Machine Learning Research (02/2025)
1.0
1.0
MoReFree
IBC
MoReFree IBC
0.5
0.5
0.0
5.5 10.5 15.5 Steps (1e5)
0.0
0.5
5.5 10.5 15.5 Steps (1e5)
Push (hard)
Pick&Place (hard)
Figure 17: Longer training of IBC in our Fetch tasks, where the state space larger and artificial constraints are replaced with surrounded walls. IBC still can not learn meaningful behaviors.
10
03
08-
08-
04
Push
Push (hard)
Figure 18: XY location of the block collected by IBC on Push (hard) and its original version (Push). IBC covers the whole state space very well in Push while fails in Push (hard), where the block slays for long time in corners or areas next to walls.
Figure 18 shows 600k data the obejet (XY view) collected by IBC on our Push (hard) and IBC's Push. We see the block stays in corners or next to walls lot in Push (hard), while goes everywhere and covers the whole space in IBC's Push, indicating object interaction more difficult in Push (hard) due to the larger state space, surrounded walls and limited work space. In IBC's Push, the block can bounce back when it hits the limit of joint constraints. However, in Push (hard), the block needs to be explicitly brought back from the corner or walls, requiring more sophisticated behaviors. Meanwhile, larger size of the limited area (our version is 3x larger than IBC's.) also increases the difficulty of the task.
Analysis on R3L
R3L trains two policies, one for reaching the goal and another that brings the agent to novel states. The goal-reaching policy is trained using a learned classifier to classify the goal state and other states. Original R3L takes images as inputs, thus the trained classifier can successfully classify goal images from random state images. In our work, we low-dimensional state input. Outputs of the trained classifier on the whole state space of Point is shown in Figure 19 We that the classifier learns to output higher values for states close to the goal state (red dot) and lower values for states further away. Nonetheless, due to the smoothnoss of the output scope, states near the initial state (blue circle) that are numerically closer but spatially further to the goal state also have higher values. R3L agent trained using such reward function will always tend to follow states with higher values to the corner instead of going forward. See the website for more videos. These trained reward functions are misleading for learning reasonable policies which result in poor performance we in Figure
21

---

Published in Transactions on Machine Learning Research (02/2025)
Figure 19: Outputs of the learned classifier on the whole state space. Due to the smoothness of the output scope, states near the initial state (blue circle) also have higher values.
22


Reviews:
[{"review_id": "ImSSnYCdgk", "reviewer": "Reviewer_76Wb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I thank the authors for their great work. The paper is well written and much effort is put into explaining things clearly. The work examines the reset-free RL scenario, which is important for many real-world applications, so I expect it to be impactful. After analysing prior methods either specifically designed for the scenario or adapted for it, the authors propose a novel algorithm which changes both exploration with the new Back-and-Forth Go-Explore methodology, as well as the policy training, by changing the goals that are sampled to train the goal conditioned policy to be biased towards task relevant states. The resulting empirical evaluation demonstrates the improvements over relevant baselines and the analysis & ablations show that it is indeed the changes proposed by the authors that make the difference. \n\nI do not see any major weaknesses of the paper - one could of course include more baselines or environments to make the empirical evaluation even more compelling, however the amount of evidence presented here is enough to make a point.\n\nI have a couple of minor comments / questions:\n- for reviewing, it would be great to have line numbers\n- What confused me a little was the following sentence: \"In reset-free training the agent will only be reset to the initial state s0 ∼ ρ0 a few times\". The name \"reset-free\" kind of implies that this resetting happens 0 times, could elaborate this further? How many times do you reset during training & how often do \"normal\" algorithms reset?\n- In fig. 2 I am missing an explanation for the purple dot / state (bottom row in the middle) - since it says initial state, I would imagine the cooler should be turquoise?\n- Typo: \"Their robust exploration demonstrate significant ...\" (demonstrates)\n- Typo: \"conduct a through\" (thorough)\n- One thing I was wondering while reading: You assume to have access to the goal & start state distributions p_g & p_0 so that you can sample them during policy training. I'm wondering whether that is a common & justified assumption in the reset-free setting? In the normal case one could at least get the starting distribution by sampling from the reset function, but we cannot in this case. Also the goal distribution is not directly visible maybe in some scenarios. Could you elaborate a little? Why do you e.g. not use environment reward but do use those two pieces from the original env?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors examine the role of world models in reset-free reinforcement learning scenarios, identify their limitations, propose a new algorithm to mitigate those limitations, evaluate model based as well as model free baselines on a number of challenging benchmarks, and analyse the algorithms' performances by inspecting the resulting state space visitations.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See comments above, please fix fig.2 / add an explanation.", "broader_impact_concerns": "-", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LwvNfxAsOg", "reviewer": "Reviewer_crQe", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n\n- The introduction is well-written, and the motivation for reset-free reinforcement learning is clearly presented.\n- The empirical results are comprehensive. The authors compare their method with five different baselines (one model-based and four model-free) in 8 different tasks.\n- The authors include an ablation study, where they discuss and justify the use of the different components of their method.\n\n\n### Weaknesses:\n\n- There is no comparison with another model-based RL method except from the adapted reset-free PEG.\n- Although it seems like reset-free PEG leads to over-exploration based on Figure 1, the results in Figure 4 show that reset-free PEG is on par with MoReFree for most tasks – the gap looks convincing mainly for Push (hard) and Ant environment.\n- Missing descriptions and information about the environments in the main paper. This information is in the appendix, but a basic description and some information about each environment's initial and goal location would be helpful to be in the main paper.\n\nRegarding weaknesses, I have these additional points for discussion and clarification:\n\n1. In Algorithm 3, line 6, for the Go-Explore process where the goal is sampled from the initial state distribution, it needs to be clarified what is the starting state of that trajectory. Is it the previous goal state or the terminal state of the previous exploration? Could you specify that in the algorithm to make it clear?\n\n2. What happens when $\\rho_g*$ and $\\rho_0$ are identical to the state space $S$, i.e., every state can be a plausible goal location or initial state. Do the Back-and-Forth Go-Explore and the sampling strategy become trivial in that case? Does that mean that this method makes sense in cases where $\\rho_g*$ and $\\rho_0$ are distinct subsets of state $S$?\n\n3. It is unclear why, in Algorithm 2, line 5, the goal for reset-free PEG is sampled from $\\rho_E$. To me, the equivalent for the reset-free PEG is to be sampled from $\\rho_g*$. If the exploration goal distribution $\\rho_E$ is different from the evaluation goal distribution $\\rho_g*$, reset-free PEG never observes the evaluation goal distribution as MoReFree does. It would be nice if the authors could clarify this point. \n\n4. Could you explain how the choice of the task relevant states is being made in order to get the percentages in Figure 5 (section 5.2 analysis)?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors study the setting of reset-free reinforcement learning. They build upon PEG, a Go-Explore style exploration strategy used alongside a goal-conditioned model-based reinforcement learning framework. \n- The authors empirically demonstrate that directly applying PEG in a reset-free setting leads to over-exploration. \n- They adapt the Go-Explore strategy used in PEG with their Back-and-Forth Go-Explore strategy, which additionally incorporates trajectories with goals sampled from the initial distribution of the environment. \n- When updating the goal-conditioned policy by using the learned world model, the authors use a sampling strategy to alternate between sampling goals from the goal, initial, and exploration distribution. By selecting goals from these distributions, the claim is that the goal-conditioned policy learns to reach the goals but also learns to reset the agent to the initial states during the reset-free training procedure. \n- In their experiments, the authors demonstrate that the two model-based methods, i.e., reset-free PEG and their method MoReFree, outperform in terms of efficiency model-free baselines in 7 out of 8 tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "First, I would like to ask the authors to address and clarify the aforementioned weaknesses and open questions/points.\n\nAdditionally, to strengthen the paper and make it easier to follow, I recommend the authors to consider the following changes in the submission:\n\n- I recommend that the authors not refer the reader to the two other works (Mendonca et al. (2021) and Hu et al. (2023)) for more details, as done in 3.2. Instead, it is better to include in the paper all the relevant details that we need to know from these two works. Since these details are important to follow and to understand the method, adding these details would make the paper self-contained.\n\n- In Algorithm 2, lines 11 and 12, the policies are updated to maximize some rewards. It would be good to clarify and explain in the paper what are the rewards that the policies try to maximize.\n\n- Clarify further how distribution $\\rho_E$ is defined and include it in the paper.\n\n- Include a short description of each environment along with information about each environment's initial and goal location in the main paper.", "broader_impact_concerns": "No concerns here. The authors include a broader impact paragraph in the appendix.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PIIa9r96Af", "reviewer": "Reviewer_ybee", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n- This paper points out an important problem of reducing human effort in RL training, which is a significant bottleneck for real-world applications.  The focus on reset-free RL is very relevant to this challenge.\n\n- Combining a tailored exploration strategy (Back-and-Forth Go-Explore) with targeted policy training within a learned world model seems to be a good idea to address the over-exploration problem observed in prior MBRL adaptations for reset-free tasks.\n\n- MoReFree achieves good performance without relying on environment rewards or expert demonstrations, unlike many existing reset-free methods.  This increased autonomy is a significant advantage for practical applications.\n\n- This paper provides a clear analysis of MoReFree's behavior, including visualizations of state visitations and targeted investigations of the \"Back\" trajectories.  In addition, the ablation studies evaluate the contributions of MoReFree's individual components, providing clear evidence for the effectiveness of the proposed two strategies.\n\n\n## Weaknesses\n\n\n- It seems the proposed method exhibit limitations when handling tasks with *irreversible consequences*. While in the Ant task, the agent can freely relocate to achieve different goals, many real-world scenarios do not offer such flexibility. For instance, in table-top manipulation tasks, if a robot kick an object out of the the workspace, learning becomes impossible until the environment is reset by someone else. The core challenge lies in moving between arbitrary current states and goals, which may be infeasible in numerous scenarios. The paper's reported failures in the Sawyer Door task likely stem from similar difficulties. While the proposed method works well in many tasks, I feel its applicability is constrained in environments where actions can lead to irreversible consequences.\n\n\n- The proposed method requires access to a goal distribution, which limits its applicability to tasks with a single fixed goal. For instance, in tasks like plugging a charger into a socket, where only one specific goal state exists, it remains unclear how the method could be effectively implemented. \n\n- Based on Figure 4, the performance difference between MoReFree and the reset-free PEG baseline appears marginal.\n\n- The current implementation uses a fixed parameter $\\alpha$ to control the balance between task-relevant and exploratory goals.  This fixed curriculum might not be optimal for all environments or stages of learning. An adaptive curriculum that adjusts $\\alpha$ based on the agent's progress could potentially improve performance.\n\n- While the motivation is rooted in reducing human effort for real-world applications, the experiments are still conducted in simulated environments. Demonstrating the effectiveness of MoReFree on real-world robotic tasks would significantly improve the work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces MoReFree (Model-based Reset-Free), an RL agent designed for reset-free tasks. This paper first demonstrates the potential of model-based RL in reset-free settings, by adapting an existing MBRL method (PEG) and showing it outperforms prior state-of-the-art reset-free methods on a locomotion task.  Then, it propose MoReFree to address the over-exploration issue in PEG by proposing two key mechanisms: 1) Back-and-Forth Go-Explore; 2) Task-Relevant Goal-Conditioned Policy Training.\n\nThe authors evaluate MoReFree and several baselines (including the adapted PEG, IBC, MEDAL, and R3L) on eight reset-free tasks, ranging from manipulation to locomotion.  MoReFree demonstrates good performance and sample efficiency compared to the baselines in many tasks. \n\nAnalysis of the agent's behavior reveals that MoReFree explores the state space effectively while maintaining a higher concentration of experience in task-relevant regions.  Ablation studies confirm the importance of both the Back-and-Forth Go-Explore and the task-focused policy training.\n\nThe key contributions of this paper are:\n-  Demonstrate the viability of using model-based agents with strong exploration abilities for the reset-free setting as well as their inherent limitations. \n-  Address such limitations through the MoReFree framework which focuses exploration and policy optimization on task-relevant states.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weaknesses.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper investigates the role that world models play in the context of reset-free reinforcement learning (RL). The paper identifies the limitations of existing Go-Explore-like exploration (namely PEG) and propose changes to both the exploration strategy (namely, Back-and-Forth Go-Explore that utilizes trajectories with goals sampled from the environment) as well as to policy training (by using a sampling different goal types) in an effort to address some of these limitations.

The reviewers are in agreement regarding the paper's key strengths as well as some of the questions/concerns with the paper as initially submitted. Among the strengths, all three reviewers agree that the paper addresses an important problem (namely, reset-free RL) that is of interest to many in the community. Additionally, the reviewers emphasize the comprehensive nature of the experimental evaluation and the value of the ablation study, which provides evidence of the benefits of the proposed changes to the exploration strategy and policy training. The reviewers similarly agree that the paper is well written and that the presentation is clear and easy to follow. Some reviewers questioned the seemingly small gain in improvement over the PEG baseline, while at least one reviewer suggested the importance of comparing to an additional model-based RL method other than the adapted reset-free PEG baseline. As part of the rebuttal process, the authors included an additional comparison to DreamerV2. Meanwhile, the authors effectively addressed other questions and concerns that the reviewers raised, which were acknowledged in comments to the AE.
Decision: Accept (poster)