A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoals
Authors: Grace Liu
Michael Tang
Benjamin Eysenbach
Venue: ICLR 2025 Poster (2025)
Keywords: exploration
emergent skills
contrastive reinforcement learning
open-ended learning
Primary area: reinforcement learning
TL;DR: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoals

Abstract: In this paper, we present empirical evidence of skills and directed exploration emerging from a simple RL algorithm long before any successful trials are observed. For example, in a manipulation task, the agent is given a single observation of the goal state (see Fig. 1) and learns skills, first for moving its end-effector, then for pushing the block, and finally for picking up and placing the block. These skills emerge before the agent has ever successfully placed the block at the goal location and without the aid of any reward functions, demonstrations, or manually-specified distance metrics. Once the agent has learned to reach the goal state reliably, exploration is reduced. Implementing our method involves a simple modification of prior work and does not require density estimates, ensembles, or any additional hyperparameters. Intuitively, the proposed method seems like it should be terrible at exploration, and we lack a clear theoretical understanding of why it works so effectively, though our experiments provide some hints.

Published as a conference paper at ICLR 2025
A SINGLE GOAL IS ALL YOU NEED: SKILLS AND EXPLORATION EMERGE FROM CONTRASTIVE RL WITHOUT REWARDS, DEMONSTRATIONS, OR SUBGOALS
Grace Liu" Princeton University
Michael Tang
Benjamin Eysenbach
ABSTRACT
In this paper, we present empirical evidence of skills and directed exploration emerging from a simple RL algorithm long before any successful trials are observed. For example, in manipulation task, the agent is given single observation of the goal state (see Fig. T and learns skills, first for moving its end-effector, then for pushing the block, and finally for picking up and placing the block. These skills emerge before the agent has ever successfully placed the block at goal location and without the aid of any reward functions, demonstrations, manually-specified distance metrics. Once the agent has learned to reach the goal state reliably, exploration is reduced. Implementing prior work and does not require density estimates, ensembles, or any additional hyperparameters. Intuitively, the proposed method seems like it should be terrible at exploration, and we lack a clear theoretical understanding of why it works so effectively, though our experiments provide some hints.
method involves simple modification
Videos and code:
github. 1o/ co/sger1/
matal brrtfteit
Mave hard Welte
Knock noject bon
Pick the noject
Training progress
Openiblaue pour
Yraheleech
Pickand
Figure Skills and Directed Exploration Emerge. this task, we provide the agent with a single goal observation where the green block is in the left bin. The agent never receives any rewards (not even sparse rewards). Throughout training, the agent learns skills that increase in complexity. Easier skills seem to enable the agent to unlock more complex skills; moving the hand is prerequisite for pushing the object; closing the gripper prerequisite for picking up the object, which is prerequisite for moving the object the left bin.
INTRODUCTION
Exploration one grand challenges in reinforcement learning (RL) TTrun 1992). Effective exploration algorithms would enable RL agents solve long-horinn horizon, sparse reward problems with minimal human supervision: no need for dense reward functions, demonstrations, or hierarchical RL. While there is long history of exploration methods, even today's best methods fail to explore in settings with sufficiently sparse rewards, and the complexity of sophisticated exploration techniques means that most researchers today employ limited exploration methods (e.g., adding random noise to actions (Heess et al.) 2015 Lillicrap et al. 2015; Mnih et al. 2013 Fujimoto et al. 2018)).
In this paper, focus a specific type of RL problem (Kaelbling 1993
Chane-Sane al. 2021 Liu et al. 2022): : the agent is given an observation of the single desired goal state, which
"Correspondence Grace Liu egliu2@andreed aanui@@@nnneeec.ecc.t

---

Published as a conference paper at ICLR 2025
it tries to reach. This problem setting captures many practical problems, from cell biology (grow a certain cell type) to chemical engineering (create a specific molecule) to video games (navigate to the final room). However, this problem setting is exceedingly challenging for standard RL methods, as the agent does not receive any reward feedback about how it should solve the task. In continuous settings, the agent will never reach the goal exactly, so no reward signal is ever observed. Because of the difficulty of this exploration problem, prior work typically assumes that a human user can provide dense reward function (Yu et al. 2020 HasssnnEsstruch et al., 2022) (or distance et 2019 P lappert et al. 20018a] Chane-Sane et al. (021)) or a set of easier training goal (Eysenbach et al. 2022). However, constructing these reward functions
metric/threshold /Vnnattaaaaaauj
or easier goals is challenging (Liu et al. 20222] Hadfield et al.) (011)) and stymies potential applications of RL: a chemist who wants to synthesize one particular molecule would have to write down several "easier" molecules for the RI agent to reach. Our paper lifts the assumptions of prior work by considering a setting that is casier for human users but significantly more challenging for RL agents: single goal state is provided and is used for both training and evaluation.
We present a simple RL algorithm where skills and directed exploration emerge long before any successful trials ever observed. For example, in manipulation task where the agent is given a single observation of the goal state (see Fig. [ the agent ends up learning skills for moving its end-effector, then for pushing the block, then for lifting the block, and finally for picking up the block. These skills are learned before the agent has ever succeeded at placing the block in the correct location distance metrics. Once the
and without any reward functions, demonstrations, or agent has learned to reliably reach the goal state, it slows exploration. Implementing this method involves a simple modification of prior work and does not require density estimates, ensembles, or any additional hyperparameters.
Our method works by learning goal-conditioned value function via contrastive RL (CRL) (Eysen bach et al. (2022) and using that value function to train a coal-conditioned policy. The key ingredient embarrassingly simple: when doing exploration, always condition the goal-conditioned policy on the single target goal. There are several intuitive reasons why we initially thought this method should work poorly:
(i) Before the single target goal is reached, the value function will predict bogus values for that goal, so it should be unable to train the policy.
(ii) There is no mechanism to drive exploration. Sampling currieulum of goals that includes easy goals and leads to the target goal should perform much better.
This intuition turned out to be fallacious.
Empirically, we evaluate our approach on tasks ranging from bin picking to peg insertion to maze navigation, finding that significantly outperforms prior methods that use a manually-designed curricula of subgoals (Eysenbach et al., 2022), methods that automatically propose subgoals for training (Chane-Sane et al.) 2021), and even methods that use dense rewards (Haarnoja et al., 2018). While do not claim that this is the best exploration method, it outperforms all alternative exploration strategies we have tried. Not only do we observe emergent skills during training, we find that different random seed initializations learn divergent strategies for solving the problem. While we still lack a theoretical understanding of why this approach is so effective, experiments highlight that (I) the contrastive representations used to express the value function are important, and that (2) the gains are not caused by "overfitting" the policy or the value function to the single target goal.
RELATED WORK
Our work builds on a long line of prior work in exploration methods for reinforcement learning (Jin
et al. 2020; Thrun 1992 Tokic
2010 Stadie et al. 22055] Tang al. 2017 Asmuth et al. 2009
Kearns & Singh McGoven & Barto| 2001). ), and will study this problem in the specific setting
of goal-condittoned RI. (GCRL) KKelbiing 1993 Schaul et al. 2015 Annrycchwicc al.
2017
Lin et al. 2019 Eysenbach et al. 2021 Rudner et al. 2021 (avvinvv et al. al. 2018) Ding et al.
2019
Sun al.
2019
Ghosh et al. 2020 Lynch et al.) 2020 Dosovitskiy & Koltun 2016 (Schmeckepper
Some prior methods automatically propose training goals (Florensa et al. [217 2018
Sukhbaatar et al.
2018][OpenAl al.] (202]) 1). yet these methods require additional machinery and are primarily evaluated on
settings where subgoals lie on 2-dimensional manifold.
2

---

Published as a conference paper at ICLR 2025
et al. 2020 Nachum et al. 2018 Savinov et al. 2018: Srinivas et al. 2018 Nasiriany al. 2019 Eysenbach et al. 2019). This section reviews three types of strategies for exploration. Our proposed method aims to lift the limitations associated with these prior methods.
Rewards and demonstrations. One of the key challenges with GCRL is the sparsity of the learning problem, many prior GCRL methods assume access to dense, hand-crafted reward function (Plappert et al., 2018a Schaul et al. 2015) or distance metric (Trott et al. (2019): Tian et al. (222)) Hartikkinen et al. (2020); Wu et al. (2019). Other methods attempt make GCRL more tractable by using expert demonstrations Paul et al. (201)); Ding et al. (2019) to guide learning and planning. Although well-designed reward functions and expert demonstrations are useful for training, these components add complexity, and collecting demonstrations can be challenging. Our method builds upon growing collection of GCRL algorithms that require neither a reward function nor demonstrations (Lin et al. 2019; Eysenbach et al., [022] 2021: Zheng et al. 2024] Sun et al. 2019 Ghosh et al. 2020) specifically, we consider variant of GCRL where only single goal is provided for training and evaluation. As such, we could treat it as single-task problem. but we find
that treating it a multi-task problem crucial to achieving good performance.
Exploration and subgoal sampling. Without a dense reward function or expert demonstrations, the primary challenge of GCRL is effective exploration. One class of exploration strategies adds noise to the actions (Fujimoto et al.] 20188] [illicrap et al. 2015 Heess et al. [015] Haarnoja et al. 2018) or policies (Plappert et al./ |0188] Fortunato et al., 2018). While these methods are simple to implement, they typically fail to perform directed exploration (Osband et al. (0166)) A second class
of methods formulates an intrinsic exploration reward (Machado et al.) 2020 Pathak et al. 2017 7
et al. 2020; Eysenbach et al., 2018 Conti et al. [0188] [Bugiee & Ichise (020). which the agent aims to optimize in addition to the rewards provided by the environment. While these methods can work effectively, they can be challenging to scale to high-dimensional and long-horizon tasks, A third class of methods use probabilistic techniques, including ensembles (Osband et al. 200 Chen 2017: Yao et al., | Chen et al.) 2018 Pearce et al.) (018)), posterior sampling (Osband 2016b 2018 Dann et al. 2021 Fan & Ming 2021). and uncertainty propagation O'Donoghuc 2018 Tosatto al. 2019) these methods can excel at directed exploration, though challenges
ct al.
ct al.
al.
inc include tuning the prior and dealing with large ensembles. A fourth class of methods modifies the goals that are used in training. For example, some methods automatically propose subgoals, breaking
down a hard task into a sequence of easier tasks (Chane-Sane et al. 2021 Zhang et al. 2022 Savinov
al. 2018 Shah al.
2022] Zhang al. 2024). We compare against one prototypical subgoal
sampling method (RIS (Chane-Sane et al., 2021) D. Other methods automatically adjust the goal
distribution (Pong et al.
2020 Florensa al. [0188] Venkattaramanujam et al., (019)) or initial state
distribution (Florenss et al.] 2017), so that the difficulty of learning increases throughout training.
Despite excellent results in certain settings, scaling these methods beyond 21) navigation remains
challenging. and the algorithms remain complex.
Multi-task learning for single-task problems The last strategy for exploration is so ubiquitous it is easy to forget: training on multiple related tasks, even when we only care about performance on a single difficult task. For example, many prior GCRL methods command a range of goals during exploration. Intuitively, the easy tasks can be learned with little exploration, and learning those tasks should enable the agent to solve more challenging tasks (similar to curriculum learning (Bengio et al.) 2009, Matüienn M et al., 2019, Campero et al.) 2021)). 1). However, actually constructing these multiple training tasks or goals requires additional human supervision: the human often lays out "trail of breadcrumbs". and the agent learns how to navigate to each (Eysenbach et al. 2022). Our paper will study the setting where only a single goal is provided for training, yet our experiments will compare against baselines that have access to training goals with range of difficulties.
3 SINGLE-GOAL EXPLORATION WITH CONTRASTIVE RL
3.1 PRELIMINARIES
Notation. We consider a controlled Markov process (i.e., an MDP without a reward function) defined by time-indexed states 8t and actions aj. Our experiments will use continuous states and actions. The initial state is sampled so (s() and subsequent states are sampled from the Markovian dynamics 5 ~ p(8 st, a.). Without loss of generality, we assume that episodes have infinite horizon; finite horizon problems can be handled by augmenting the dynamics with an
3

---

Published as a conference paper at ICLR 2025
absorbing state. We assume that the algorithm is given input the single target goal state s" and aims to learn a policy π Ta $t) by interacting with the environment. Unlike prior work, we do not assume that distribution of goals for exploration is given; we do not assume that either dense or sparse reward function is given.
Following prior work (Eysenbach et al.
Schroecker & Isbell, 2020), we define the objective as maximizing the probability of reaching the goal. Formally, define the 7-ddiscounted state occupancy
2021
measure (Ho & Ermon 2016 Syed et 2008
Dayan 1993)
p²(ss) (1 7)
Σ t=0
V - 8y), 88),
where pi (s, = 85) is the probability of being at state sy at time step L. In continuous settings, P (sy) probability density. The objective is to find a policy that maximizes the likelihood of the single target goal under this occupancy measure:
max p (s)===")
In discrete settings, this objective is equivalent to the standard discounted reward objective with a reward function r(s,, at) (ss = s"); in continuous settings, it is equivalent to using a reward function r(st, ax) = p(s' s+ 8t, az). Intuitively, this corresponds to maximizing the time spent at the goal. The hyperparameter y E [0, 1) is part of the problem description.
Contrastive RL. Our method builds on contrastive RL (Eysenbach et al.) 2022), prior work that uses temporal contrastive learning solve goal-conditioned RL problems. This method was designed for a slightly different setting, where the input is a distribution over goals p(g). and the aim was to learn conl-conditioned policy x (a 8,g) for reaching each of these goals. Contrastive RL is an actor-critic method. The critic C(s, a, sy) ) is learned so that it outputs the (relative) likelihood that an agent starting state and taking action a will visit state sy. Following prior work, we parameterize the critic as the dot product between two learned representations, e(s, ((s,)) The representations are not normalized. We will write the loss function in terms of these representations, which will turn out to be key for achieving good exploration.
To define the learning objective, we introduce a few distributions. Define p(s, a) as the marginal distribution over state-action pairs in the replay buffer, and define p(sy s, a) as the empirical discounted state occupancy measure, conditioned on a state s and action a. Define p(s,) as the corresponding marginal distribution over future states. Contrastive RL uses these distributions train the critic with a contrastive learning objective. Following prior work, we learn these representations using the infoNCE contrastive objective (Oord et al., 2018) together with a LogSumExp regularization that prior analysis (Eysenbach al. 2022) has shown is necessary when using the infoNCE objective for control:
log e Maa.)) (1)
e (s.a) ((o
E (f ((sr)).p(rs)
00.01 log
mplsylaa) of (2.) ~pter)
Σ (al
infoNCE
LogSumExp rogularization
(3)
In practice, this loss is implemented by sampling a random (st. 0z) pair from the replay buffer and then sampling future state s 8 by looking A ~ GEOM(1 -n) steps ahead. The negative examples are obtained by shuffling the future states (i.e., sampling from the product two marginal distributions).
Once learned, the representations encode a Q-value (Eysenbach et al. 2022 o(s, (sy)) = log Q(s, a, - p(sy), where the value is defined with respect to the reward function introduced above. The policy is learned to maximize this (log) Q-value:
max Ep(e)p(s))
[Ø(x,a) (b(r)))()))))))))))))))())))(( + aH(
where a is an adaptive entropy coefficient. Intuitively, the actor loss chooses the action a that maximizes the alignment between 6(s, a) and (ss"..
4

---

Published as a conference paper at ICLR 2025
Algorithm Single-goal Exploration with Contrastive RI. difference from most prior methods that exploration is done by commanding a single difficult goal s", rather than sampling goals with a range of difficulties.
1: Initialize policy ap(a s. s.9). replay buffer B, classifier with logits o(s.a)) (b))))))))).......))))))))))))
2: while not converged do
3: Collect one trajectory of experience using (a 8,8y = s"), add to buffer B.
4: Update representations ((8, a), (by8) and policy xaaa 8, 88y)) using contrastive RL.
5: Return policy <a 8, 8*).
3.2 OUR APPROACH
We now describe our approach to tackling the problem of learning to reach a single goal state. Our approach is simple modification of contrastive RL: rather than asking the human user to provide many training goals for exploration, we always command the policy to collect data with the single hard goal s* (see Fig. 2). No other modifications are made the algorithm. We use the same single goal state and success criteria as prior work (Yu et al.) 2020);; in each environment, the single goal corresponds to a state representative of successful task completion (e.g. closing box inserting peg). The critic loss is the same as contrastive RL (Eq. The actor loss is the same (Eq. 4). We will call this method "single [hard] goal CRL." Note that the single-goal in the algorithm name refers to the goal used for data-collection, not the goals used for policy updates. For training the actor, we use goals sampled from future states in the replay buffer. (Eq. 4 Ablation experiments (Fig. 10) show that only training the actor on the single goal decreases performance. We summarize the approach in Alg.
Collecting during traininng witha singie gool (ours)
Goads collection training range aifhoutiess (most prior methocs)
start
start
data collection goals
goal (data collection +
evalustion)
evaluation goal
Figure 2: Single-goal exploration. (Left) Our method uses a single difficult goal for both data col- lection and evaluation. is exceedingly unlikely that a random policy would ever reach this goal. (Right) Typical methods for goal-conditioned RL use range of different goals for data collection, even if the user only cares about success reach- ing single difficult goal. These different goals be provided by the user (Eysenbach al.) 2022) or generated with a GAN (Florensa ct al. 2018 8)
VAE (Nasiriany et al, 2019), planning (Cl Chanc
Sane et al. 202 Savinov al. 2018 Zhang al. 2022).
4 EXPERIMENTS
The main aim of our experiments is evaluate the performance of single-goal contrastive RL compared to its multi-goal counterpart as well as prior baselines. We do so on four exploration-hear goal-reaching tasks, involving robotic manipulation and maze navigation. All experiments were run with five random seeds, and error bars in the plots depict the standard error. Hyperparameters can be found in Appendix B and code to reproduce our results is available online:
comm /ra fuce/ sger (tree/maaa
Tasks. We measure the efficacy of our method four goal-reaching tasks taken from prior work (Eysenbach et al. 2024), which are chosen to measure long horizon exploration. The tasks include three robotic manipulation environments (Yu al. (020) and one maze navigation environ- ment (Eysenbach et al. 2019). The robotic manipulation tasks require controlling a sawyer robot to grasp an object (e.g.. block, lid, or peg) and accurately place it in predetermined location (e.g.. in bin, top of box, or inside hole). The point spiral task is a 2D maze navigation task. We quantify success in each episode by whether the agent reached sufficiently close the goal in least one state in an episode. This 0/1 sparse reward signal is used by a few of our baselines but is not needed by contrastive RI. Note that while (Eysenbach et al., 2019) measured success by evaluating on goals with range of difficulties, we evaluate only on the single, hard goal, which corresponds to successful task completion in the Metaworld benchmark (Yu et al. 2020).
These tasks present exceedingly difficult exploration challenges. To quantify the difficulty, we measured the success rate under a uniform random policy: for each of the sawyer environments, we did not a single success state in 75,000,000 environment steps (600,000 episodes); for the point spiral environment, we did not see single success state in 40,000,000 environment steps (400,000 episodes). Previous methods solve these tasks with the aid of dense rewards ( (Yu et al. 2020). or
5

---

Published as a conference paper at ICLR 2025
Sawyer Bin
Sawyer Box
Success (Single Hard Goall
1.0
1.0
Sawyer Peg
Point Spiral
1.0
tSinglc Hard Giaall
Success
0.5
0.0
Steps
167
Steps
1e6
CRL (Single Hard Goal)
CRL (Range of Difficulties)
Figure Single goal Exploration Highly Effective. We compare single hard goal exploration (command the single hard goal in every trial) to "range of difficulties" exploration (sampling uniformly from a human-provided set of easy/mediun/hard goals). In each of the four environments, single-goal exploration yields considerably higher success rates, all while being easier for the human user.
with a rich curriculum of subgoals (Bysenbach et al. 2019), but the single-goal CRI agents must accomplish these tasks with no hand-crafted rewards, demonstrations, subgoals.
4.1 SINGLE-GOAL EXPLORATION IS EXCEEDINGLY EFFECTIVE.
Reaching
awayy
up
Nudge place
Training progress
Tourhh lid
Punchvild flig maat
PEV and placosld
Figure 4: Skills and Directed Exploration for Putting Lid Box:
This manipulation task contains placed neatly on top of the center box. The images
open box and lid. The single fixed goal has above show skills acquired throughout the course of learning. Note that some skills unlock subsequent skills (e.g., reaching is prerequisite for picking. which is prerequisite for placing) while others look like open-ended "play" (flipping the lid over, pushing the lid away from the box).
single goal works well. We find that Contrastive RL effectively solves these four tasks: equipped only with single target goal, the agent automatically explores the environments and learss complex manipulation skills (see Fig. 3}. We compare this method to an "oracle" variant that is trained -usmandeeiinned goals that vary in difficulty, ranging from easy goals the single hard goal. Surprisingly, our proposed method significantly outperforms this "range of difficulties" method.
While have success within thousands of trials: 3,197 trials for sawyer box, 9,329 trials for sawyer bin, 15,895 trials for sawyer peg, and 11 11,724 trials for the spiral task
random policy solve any of these tasks, method achieves its first
"These numbers averaged across the random seeds.

---

Published as a conference paper at ICLR 2025
reaching
Grasp pes
Knock against
Recover fromm mistakess
Training
progress
pigg trwery
pag
Insert neg into hale
Figure 5: Skills and Directed Exploration for Peg Insertion: This manipulation task contains peg and box with narrow hole; the single fixed goal is state where the peg is inside the hole. The agent acquires sequence of increasingly complex skills throughout training, some of which important for solving the task (e-g,, reaching, grasping) while others more "playful" (e-g., knocking the peg against the box). The agent also learns to recover from mistakes (sce Fig.
#0000
Total Grid Bins Visited
60000
30000
Actor Steps
Sawyer Bin
Sawyer Box
Sawyer Peg
Figure 6: Quantifying Exploration. We analyze "single hard goal" exploration by discretizing the object's XY position and counting the cumulative number of unique positions visited throughout training. Exploration starts to plateau after the agent can successfully reach the goal (compare with Fig. 3.
(a) "Push and flick" (sced = 3)
(b) "gently pick" (seed = 4)
Figure 7: Different random seeds learn different strategies. Other seeds learn a policy whose strategy depends the initial position of the block.
Early training: agent develops emergent curriculum of skills. Not only does single- goal CRL consistently achieve high success rates on manipulation tasks, but also demonstrates complex and directed exploration techniques during early training. To observe the agent's behavior throughout training, we saved learning checkpoints at fixed intervals and visualized the agent's 's behavior at each checkpoint (see Figures
and (5. We found that the agent learns simple skills before complex ones. For example, in all three environments, the agent (/) first learns how to move its end-effector to varying locations, (2) then learns how to nudge and slide the object, and (3) finally learns to pick up and direct the object. We observe wide array of exploratory behavior in early training that is not directly connected to the goal: from punching the box lid to flip it over (Fig. far away in random direction (Fig.
pushing the block
First successes: agent trades off exploration for exploitation. As the agent learns to reach the goal more consistently, the agent's behavior becomes less exploratory, qualitatively similar UCB exploration (Guo et al.) 2020) Osband et al. 2016b 2018 Dann et al. 2021 Fan & Ming 2021). To quantify exploration, we discretized the state space of the robotic manipulation environments and recorded the cumulative number of unique positions visited throughout training. Fig. shows that the growth rate this exploration metric decreases as the success rate increases (compare with Fig. 3) For the sawyer box and sawyer peg environments, the agent achieves high success rate earlier in training, which corresponds the earlier plateau of the unique grid cells curve. For the sawyer bin environment, the agent takes longer to reach high success, and the exploration metric does not start leveling out until the end of training. This trend highlights how single-goal CRL develops self-directed exploration strategy that automatically trades off between exploration and exploitation.
Consistent successes: agent finds diverse paths to the goal. Not only is the performance of single-goal CRL reproducible (all random seeds solve the manipulation tasks), but policies trained

---

Published as a conference paper at ICLR 2025
with different random seeds learn qualitatively different goal-reaching strategies. For example, Fig. 7 shows the strategies learned by different random seeds on the sawyer bin task. One seed consistently moves the block flush against the wall of the red bin and flicks it into the blue bin. Another seed tends to grasp the block, lift it, and gently drop it in the blue bin. A third seed chooses between these strategies depending
whether the block starts near the wall or away from the wall. Without explicit human guidance in the form of rewards, demonstrations, or subgoals, we see multiple creative and divergent strategies emerge for solving the same problem.
agent develops robustness and self-recovery. During later stages of training, we observe that the agent demonstrates robustmess and learns recover from mistakes. For example, in the sawyer peg environment, when the agent drops the peg, it is able to recover by bending down and grasping the peg again. To quantify robustness, we ran perturbation experiments in the sawyer peg and sawyer box environments, in which we randomly perturbed the target object's 's location between 0 and 0.05 meters along each of the three axes. We tested two settings: (/) perturbation at the start of the episode ("static perturbations") and (2) in the middle of an episode ("dynamic perturbations": t 20 for sawyer box and t 50 for sawyer peg). As shown in Fig. X single goal exploration is robust to static perturbations and somewhat robust to dynamic perturbations, notably outperforming multi-goal CRL three out four scenarios. We hypothesize that this is also the result of better exploration, which leads to learned representations that generalize better across unusual unseen states.
Further training:
Succoss ISing hard 6.0
Sanyel Pag
Sovee Poy
Sunver fox
Semva Bor
Wiiiii Here Deal Pertrrent
CAL Change Deficattes Feturnent
Figure 8: Robustness to perturbations: Single (Hard) Goal exploration results in policies that are more robust to environment perturbations, as com- pared to policies trained with goals ranging in dif- ficulty. The success rate remains high even when the object is perturbed the start ("static") or in the middle ("dynamic") of an episode, likely because its effective exploration means that it has seen wide range of states during training.
4.2 THE RL ALGORITHM KEY
We compare against number of algorithms to investigate the importance of the underlying RL algorithm: is single goal exploration useful for other goal-conditioned RL algorithms?
Baselines. We compare single-goal CRL against prior methods that aim to address the sparse reward problem by making additional assumptions or employing additional machinery for exploration. Reinforcement learning with imagined subgoals (RIS) (Chane-Sane et al. 2202 maintainn high-level policy that predicts subgoals halfway to the end goal and learns the behavioral policy to reach both the subgoal and the end goal. We also employ a few variants Soft Actor-Critic (SAC) with additional assumptions: SAC with sparse rewards, SAC+HER with sparse rewards, and SAC with dense rewards. In the sparse rewards setting, the agent receives a reward of near the goal and O otherwise. In the dense reward setting, the agent receives continuous reward tailored to the environment, using the distance to the goal for the spiral task and the Metaworld (Yu et al. 2020) reward function for sawyer tasks. Table summarizes the assumptions for each method.
Results. As shown in Fig. 9 single--gor goal contrastive RL significantly outperforms these alternative methods, showing that the underlying RL algorithm is important for single goal exploration. Notably, prior methods rarely reach the goal at all, with the exception of RIS the simplest task (po nt spi r) To verify that our implementation of SAC was correct, we also plotted the reward function throughout training (recall that SAC has access to reward function, while other methods do not) and observed that it increases throughout the course of learning.
4.3 WHY DOES SINGLE-GOAI EXPLORATION WORK?
In this section, we present ablation experiments that provide insight into the workings of single-gon CRL. We find that using single-goal data collection and inner product critic are both key to the
8

---

Published as a conference paper at ICLR 2025
Table 1: Baselines: Assumptions for methods used in the experiments below.
Requirements Dense Rewards
Algorithm
Exploration
Exploration Goals
Distance Threshold
single goal (ours)
Contrastive RL
multiple goals
x
SAC (sparse rewards)
single goal
x
SAC (dense rewards)
single goal
SAC (sparse rewards) + HER
single goal
X
RIS
single goal
X
X
Sawyer Bin
Sawyer Box
1.
(Singlo Hard Goal)
Success
0.5
0.0
Sawyer Peg
107
Point Spiral
107
1.0
(Singie Hard Goall
Success
0.5
0.8
9.0
0.5
1.0
1.5
Actar Steps
167
Actor Steps
Jes
CRL
SAC+HER
RIS
SAC
SAC (dense rewards)
Figure 9: The RL Algorithm Matters. We compare several underlying RL algorithms all using single-goal exploration, with Fable[[highlightiee the assumptions of each method. CRL outperforms these prior methods, showing that (i) single-goal exploration is only effective with the right underlying RL algorithm and that (ü) with this algorithm, can achieve considerably higher performance while making fewer assumptions (no rewards).
Sawyer Bin
Sauyye
Sespyer Box
coal)
Staps
Merokiiiccric (angl Hard
Mondlinic
single poa collection policy tranning mutiigral policy tiaging
Eingleg goi palicy tra ning singl-pa
The importance representations. Single goal exploration less effective with using a monolithic critic architecture (as opposed to an inner product architecture), suggesting that the contrastive representations may drive exploration.
Figure 11:
Ablation experiments. While our method uses an actor loss that uses many goals (Eq, 4 alternatives that train the policy with single goal per- form no better.
Figure 10:
effectiveness of the method. Additionally, the performance boost of single-goal data collection does not seem to be due to overfitting of the policy parameters the task.
The effectiveness of single-goal exploration is not explained by overfitting. One possible explanation for the method's 's success is that the algorithm overfits its policy parameters on the single- goal task. That is, if the agent only collects data conditioned on the single goal, the algorithm does not just overfit the policy parameters to reach the single goal, but also learns useful representations for states along the path from the starting point to the goal. To test this hypothesis, we compared our method (which randomly samples different goals (see Eq 4) with a variant that always uses the single hard goal in the actor loss. Note that this single hard goal is the overfitting were occurring, we would expect that modifying the actor loss to only train on the single goal would boost performance. However, the results in Fig. 10 show that this is not the case.
that is used for evaluation.

---

Published as a conference paper at ICLR 2025
When data collected with single goal, using single goal in the actor loss degrades performance (purple vs. blue (ours)). When data are collected with multiple goals, using a single goal in the actor loss gives only slight boost performance (red vs. black). In short, the performance of single goal exploration is not explained by overfitting.
Representations are important. Our next set of experiments study how the representations might drive exploration. To do this, we replaced the inner product critic function (ø(s, a) a)"T')( (b(s)) with a monolithic critic function (Q(s, a, 8 St)), which takes as input a concatenated array of the state, action, and goal The results, shown in Fig. 11 show that single-godl exploration is not effective with using monolithic critic network. We found that using this modified critic function resulted no performance boost for the single-goal strategy, and in fact the single-goal method performed worse than the multi-goal method. This experiment suggests that the contrastive representations d(s, a)? and (s(r) could be important in driving single-goal exploration, though the precise mechanism for how they drive exploration remains unclear. In summary, single-goal exploration only effective when combining the right algorithm (CRL) with the right critic architecture. In Appendix A we show that environment dynamics are reflected in the contrastive representations early in training.
CONCLUSION
In this paper, we showed that skills and directed exploration emerges from straightforward RL algorithm: contrastive RL where every trajectory is collected by trying to reach a single fixed goal. The resulting method has many appealing properties of prior exploration methods: each episode the agent seems to try to visit some new state or attempt some new behavior. We do not see the random dithering that is common with naive exploration methods like e-greedy. Moreover, this method does not require any additional hyperparameters: this is in contrast to even the simplest exploration algorithms today, which require a scale parameter (e.g.. Gaussian noise in TD3 (Fujimoto) et al. 2018)). And, while prior exploration algorithms include schedule (more hyperparameters!) for gradually decreasing the degree of exploration throughout the course of learning, such behavior emerges automatically from our proposed method.
There remain two important outstanding questions raised by our experiments. First, we lack a clear understanding of why skills and directed exploration emerge. Our experiments provide some hints (representations are important; it is not explained by overfitting), yet much theoretical work remains be done to understand exactly what is driving the exploration. A rich theoretical understanding of the mechanisms driving the exploration here is important not only for explaining the success of this method, but it may also provide insights into how adapt the method here to other settings. Second, how can we leverage the success of the proposed method to address exploration in other problem settings (e.g.. a reward function were given or if not even a single fixed goal were available). The "impossible goal" experiment in Appendix proovides one simple approach, but there likely exist significantly better methods of exploiting the emergent exploration properties that we have observed in the single-goal setting.
The emergence of divergent strategies for solving the bin picking task points towards broader research questions about autonomous capabilities. The RL research community has built much scaffolding to help agents succeed. However, we have shown that without this scaffolding, agents develop unexpected and unique methods for problem-solving. Perhaps this seemingly creative behavior emerges not in spite of but because of the lack of human guidance. We encourage future research to explore what RL can accomplish in the absence of human intervention.
The primary limitation of our work is a lack of theoretical analysis explaining why skills and directed exploration emerge. Empirically, our experiments are focused primarily on manipulation tasks; we encourage future work to study applications to other settings.
Limitations.
Reproducibility Statement. We provide the source code (https://githur) com/ gral wwweeuueeeee wwwiiiegeeeeteeeeeeeeeeeeeeeeeee /ttgr//ttr///aainnnn/// for reproducing the primary results of the paper, shown in Figure The codebase supports experiments on all four environments, for both the single-goal and
In this experiment only, we decreased the hatch size from 256 to 32 for both methods, otherwise we encountered out-of-memory errors for the monolithic method.
10

---

Published as a conference paper at ICLR 2025
multi-goal versions of the CRL algorithm. The hyperparameters used for the CRL algorithm and baselines are given in Appendix
Acknovledgments. Thanks the members of the Princeton RL lab for feedback on preliminary versions of the work. We are also grateful to Elliot Chane-Sane for corresponding with us on the strengths and limitations of the RIS algorithm. The experiments in this paper were substantially performed using the Princeton Research Computing resources at Princeton University, which is consortium of groups led by the Princeton Institute for Computational Science and Engineering (PICSciE) and Office of Information Technology's 's Research Computing.
REFERENCES
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advances in Neural Information Processing Systems, 30, 2017.
John Asmuth, Lihong Li, Michael L Littman, Ali Nouri, and David Wingate. A bayesian sampling approach to exploration in reinforcement learning. In Conference on Unceriainty in Artificial Intelligence, pp. 19-26. 2009.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In International Conference on Muchine Learning, pp. 41-48, 2009.
Nicolas Bougie and Ryutaro Ichise. Skill-hased curiosity for intrinsically motivated reinforcement learning. Machine learning, 109:493-512, 2020.
Andres Campero, Roberta Railcanu, Heinrich Kuttler, Joshua Tenenbaum, Tim Rocktäschel, and Edward
Grefenstette. Learning with AMIGo: Adversarially motivated intrinsic goals. 2021. URL httpp::7
wwwarreviee.neet/Eeee - d-BT3c_MIMgox.
Elliot Chanc- Sane, Cordelia Schmid, and Ivan Laptev. Goal-conditioned reinforcement learning with imagined subgoals, In International Conference on Machine Learning, pp. 4430-1440, PMLR, 2021.
Gang Chen, Yiming Peng, and Mengjie Zhang. Effective exploration for deep reinforcement learning via bootstrapped q-ensembles under tsallis entropy regularization. arXiv preprint arXiv: 1rXiv:1809.00000 809000000 2018.
Richard Y Chen, Szymon Sidor, Pieter Abbeel, and John Schulman. UCB exploration via q-ensembles. arXiv preprint arXiv:1706.0 01502, 2017.
Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth Stanley, and Jeff Clune. Improving exploration in evolution strategies for deep reinforcement learning via a population of novelty- seeking agents. Advances Neural Information Processing Systems, 31, 2018.
Christoph Dann, Mehryar Mohri, Tong Zhang, and Julian Zimmert. A provably efficient model-free posterior sampling method for episodic reinforcement learning. Advances in Neural Information Processing Systems, 34: 34:12040-12051, 1021.
Peter Dayan. Improving generalization for temporal difference learning: The successor representation. Neural Computation, 5(4):613-624, 1993.
Yiming Ding, Carlos Florensa, Pieter Abbeel, and Mariano Phielipp. Goal-conditioned imitation learning. Advances in Neural Information Processing Systems, 32, 2019.
Alexey Dosovitskiy and Vladlen Koltun. Learning to act by predicting the future. In International Conference on Learning Representations, 2016.
Ben Eysenbach, Russ R Salakhutdinov, and Sergey Levine. Search the replay buffer: Bridging planning and reinforcement learning. Advances in Neural Information Processing Systems, 32, 2019.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representations, 2018.
Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. C-learning: Learning to achieve goals via recursive classification. In International Conference on Learning Representations, 2021. URI. httpp 7/open net net./oorumii--t.sa. so0---s55ss5--00

---

Published as a conference paper at ICLR 2025
Benjamin Eysenbach, Tianjun Zhang. Sergey Levine, and Russ Salakhutdinov. Contrastive learning as goal- conditioned reinforcement learning. Advances Neural Information Processing Systems, 35:35603-35620 2022.
Benjamin Eysenbach, Vivek Myers, Ruslan Salakhutdinov, and Sergey Levine. Inference via interpolation:
Contrastive representations provably enable planning and inference. arXiv preprint arXiv:2403.04000 2024.
Ying Fan and Yifei Ming. Model-basee reinforcement learning for continuous control with posterior sampling. In International Conference on Machine Learning, pp. 3078-3087. PMI.R, 2021.
Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. Reverse curriculum generation for reinforcement learning. In Conference Robot Learning, pp. 482 495. PMLR, 2017.
Carlos Florensa, David Held, Xinyang Geng, and Picter Abbeel. Automatic goal generation for reinforcement learning agents. In International Conference Machine Learning, pp. 1515-1528. PMLR, 2018.
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot. Jacob Menick, Matteo Hessel, lan Osband, Alex Graves, Volodymyr Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg. Noisy networks for exploration. In Internationaal Conference Learning Representations, 2018. URL
httpps://cpeneviiii
net/ forum? ieammiid-ryaae (d-rywwHcPAA)
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International Conference Machine Learning, pp. 1587-1596. PMLR, 2018
Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Manon Devin, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals via iterated supervised learning. In International Conference on Learning Representations, 2020.
Zhaohan Daniel Guo, Bernardo Avila Pires, Bilal Piot, lean-Bastien Grill, Florent Altché, Rémi Munos, and Mohammad Gheshlaghi Azar. Bootstrap latent predictve representations for multitask reinforeement Icarning. In International Conference on Machine Learning, pp. 3875-3886. PMLR, 2020.
Tuomas Haarnoja. Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning. pp. 1861-1870. PMLR, 2018.
Dylan MedneeldMenell, Smitha Milli, Pieter Abbeel, Stuart J Russell, and Anca Dragan, Inverse reward design. Advances Neural Information Processing Systems, 30, 2017.
Philippe
Amy Zhang, Ashvin Nair, Patrick Yin, and Sergey Levine. Bisimulation makes analogies in conlicondi reinforcement learning. In International Conference Machine Learning, pp. PMLR, 2022.
Kristian Hartikainen, Xinyang Geng, Tuomas Haarnoja. and Sergey Levine. Dynamical distance learning for semi-supervised and unsupervised skill discovery. In International Conference on Learning Representations, 2020. URL httpttttt://ppreeeeeeeeee
net foormm///mmmmem id= InhaVeve
Nicolas Heess, Gregory Wayne, David Silver, Timothy Lillicrap. Tom Frez, and Yuval Tassa. Learning continuous control policies by stochastic value gradients. Advances Neural Information Processing Systems, 28, 2015.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in Neural Information Processing Systems, 29, 2016.
Chi Jin, Akshay Krishnamurthy, Max Simchowitz, and Tiancheng Yu. Reward-free exploration for reinforcement learning. In International Conference Muchine Learning, pp. 4870-4879. PMLR, 2020.
Leslie Pack Kaelbling. Learning to achieve goals. In JJCAI, volume 2, pp. 1094-8. Citeseer, 1993.
Michael Kearns and Satinder Singh. Near-optimal reinforcement learning in polynomial time. Machine Learning, 49:209-232, 2002.
Jing Li, Xinxin Shi, Jiehao Li, Xin Zhang, and Junzheng Wang. Random curiosity-driven exploration in deep reinforcement learning. Neurocomputing, 418:139 147, 2020.
Timothy P Lillicrap. Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver. and Daan Wienstra. Continuous control with deep reinforcement learning. arXiv preprint 1xXvv:55000097 2015.
Xingyu Lin, Harjatin Singh Baweja, and David Held. Reinforcement learning without ground-truth state. arXiv preprint arXiv: 0076666 2019.
12

---

Published as a conference paper at ICLR 2025
Minghuan Liu, Menghui Zhu, and Weinan Zhang. Goal-conditioned reinforcement learning: Problems and solutions. arXiv preprint arXiv:2001.0829, 2022.
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learrnng latent plans from play. In Conference on Robot Learning, pp- 1113-1132. PMI., 2020.
Marlos Machado, Mare G Bellemare, and Michael Bowling. Count-based exploration with successor representation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 5125-5133, 2020.
Tambet Matüsen, Avital Oliver, Taco Cohen, and John Schulman. Seacher-student curriculum learning. IEEE Transactions Neural Networks and Learning Systems, (1(9):3732 3740. 2019.
Amy McGovem and Andrew G Barto. Automatic discovery of subgoals in reinforcement learning using diverse donsity. In International Conference on Machine Learning, pp. 361-368, 2001.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Joannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 1312. 2013.
Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. Advances in Neural Information Processing Systems, 31, 2018.
Soroush Nasiriany, Vitchyr Pong, Steven Lin, and Sergey Levine. Planning with goal-conditioned policies. Advances in Neural Information Processing Systems, 32, 2019.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: A.kiv:8807.03745 2018
OpenAl OpenAL, Matthias Plappert, Raul Sampedro, Tao Xu, Ilge Akkaya, Vineet Kosaraju, Peter Welinder, Ruben D'Sa, Arthur Petron, Henrique P d O Pinto, et al. Asymmetric self-play for automatic goal discovery in robotic manipulation. arXiv preprint arXiv:2101.0483 044882, 2021.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped DQN, Advances in Neural Information Processing Systems, 29, 2016a.
Tan Osband, Benjamin Van Roy, and Zheng Wen. Generalization and exploration via randomized value functions. In International Conference on Machine Learning, pp. 2377-2386. PMLR, 2016b.
Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement learning. Advances in Neural Information Processing Systems, 31, 2018.
Brendan O' D'onogguee Ian Osband, Remi Munos, and Volodymyir Mnih. The uncertainty bellman equation and exploration. In International Conference on Machine Learning, pp. 3836-3845, 2018.
Deepak Pathak, Pulkit Agrawal, Alexci A Efros, and Trevor Darrell. duriosity-driver exploration by self- supervised prediction. International Conference on Machine Learning, pp. 2778 2787, PMLR, 2017.
Sujoy Paul, Jeroen Vanbaar, and Amit Roy-Chowdhury. Learning from trajectories via subgoal discovery. Advances Neural Information Processing Systems, 32, 2019.
Tim Pearce, Nicolas Anastassacos, Mohamed Zaki, and Andy Neely. Bayesian inference with anchored ensembles of neural networks, and application to exploration in reinforcement learning. arXiv preprint WxXiv:1805.11324 2018
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Macick Chociej, Peter Welinder, et al. Multi-goal reinforcement learning: Challenging roboties environments and request for research. arXiv preprint wxXv:1802.009466 2018a. 8a.
Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowiez. Parameter space noise for exploration. In International Conference on Learning Representations, 2018b.
Vitchyr Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew-fit: State- covering cuff-superviee reinforcement learning. In International Conference on Machine Learning, pp. 7783-7792. PMLR, 2020.
Tim GJ Rudner, Vitchyr Pong, Rowan McAllister, Yarin Gal, and Sergey Levine. Outcome-driven reinforcement learning via variational inference. Advonces in Neural Information Processing Systems, 34, 2021.
13

---

Published as a conference paper at ICLR 2025
Nikolay Savinov, Alexey Dosovitskiy, and Vladlen Koltun. Semi- Semi-parametric topological memory for navigation. arXiv preprint 1003::v8800005333 2018.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In International Conference on Machine Learning, pp. 1312-1320. PMIR, 2015.
Karl Sehmeckpeper, Annie Xie, Oleh Rybkin, Stephen Tian, Kostas Daniilidis, Sergey Levine, Chelsea Finn. Learning predictive models from observation and interaction. In Luropean Conference on Computer Vision, 125. Springer, 2020.
Yannick Schmeeker and Charles Isbell. Universal value density estimation for imitation learning and goal- conditioned reinforcement learning. arXiv preprint arXiv:2002.06473, 2020.
Dhruv Shah, Benjamin Eysenbach, Nicholas Rhinehart, and Sergey Levine. Rapid exploration for open-world navigation with latent goal models. In Conference on Robot Learning, pp. 674-684. PMLR, 2022.
Aravind Srinivas. Allan Jabri, Pieter Abbeel, Sergey Levine. and Chelsea Finn. Universal planning networks: Learning generalizable reprcsentations for visuomotor control. In International Conference on Machine Learning, pp. 4732-4741. PMLR, 2018.
Bradly C Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing exploration reinforcement learning with deep predictive models. arXiv preprint 150.0001500000144 2015.
Sainbayar Sukhbaatar. Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Rob Fergus. Intrinsic motivation and automatic curricula via asymmetric self-play. In International Conference on Learning Representations, 2018
Hao Sun, Zhizhong Li, Xiaotong Liu, Bolei Zhou, and Dahua Lin. Policy continuation with hindsight inverse dynamics. Advances Neural Information Processing Systems, 32: 10265-050025,, 2019.
Umar Syed, Michael Bowling, and Robert E Schapire. Apprenticeship learning using linear programming. In Proceedings of the International Conference on Machine Learning, pp. 1032-1039, 2008.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAl Xi Chen, Yan Duan, John Schulman, Filip DeTurek, and Pieter Abbeel. exploration: study of baset- exploration for deep reinforcement learning. Advances in Neural Information Processing Systems, 30, 2017.
Sebastian B Thrun. Efficient exploration in reinforcement learning. Carnegie Mellon University, 1992.
Stephen Tian, Suraj Nair, Frederik Ebert, Sudeep Dasari, Benjamin Eysenbach, Chelsea Finn, and Sergey Levine. Model-based visual planning with self-supervised functional distances. In International Conference on Learning Representations, 2021. URLLstttss://ppeeeeeee
net http://rrorn/otooxtoteete-e
Michel Tokic. Adaptive e-greedy exploration in reinforcement learning based on value differences. In Annual Conference on Artificial Intelligence, pp. 203-210. Springer, 2010.
Samuele Tosatto, Carlo D' O'Eramo, Joni Pajarinen, Marcello Restelli, and Jan Peters. Exploration driven by an optimistic bellman equation. International Joint Conference on Neural Networks (IJCNN), pp. 1-8. IEEE, 2019.
Alexander Trott, Stephan Zheng, Caiming Xiong, and Richard Socher. Keeping your distance: Solving sparse reward tasks using self-balancing shaped rewards. Advances in Neural Information Processing Systems, 32, 2019.
Srinivas Venkattaramanujan, Eric Crawford, Thang Doan, and Doina Precup. Self-supervised learning of distance functions for goal-conditioned reinforcement learning. arXiv preprint arXiv:1907.02998. 2019.
Yifan Wu, George Tucker, and Ofir Nachum. The laplacian in RL: Learning representations with effi- cient approximations. In International Conference on Learning Representations, 2019. URL Ittpp!
Yao Yao, Xiao, Zhicheng An, Wanpeng Zhang, and Dijun Luo. Sample efficient reinforcement learning via model-ensemble exploration and exploitation. In International Conference on Robotics and Automation (ICRA). pp. 4200-4200000 2021.
Tianhe Yu, Deirdre Quillen. Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A henchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094-1 100. PMLRR, 2020.
14

---

Published as a conference paper at ICLR 2025
Tianjun Zhang, Benjamin Eysenbach. Ruslan Salakhutdinov, Sergey Levine, and Joseph E. Gonzalez. C- planning: An automatic curriculum for learning goal-reaching tasks. In International Conference on Learning Representations, 2022. URL
Zichen Zhang, Yunshuang Osbert Bastani, Abhishek Gupta. Dinesh Jayaraman. Yecheng Jason Ma. and Luca Weihs. Universal visual decomposer: Long-horizon manipulation made casy. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 6973-6980. IEEE, 2024.
Chongyi Zheng. Ruslan Salakhutdinov, and Benjamin Eysenbach. Contrastive difference predictive coding. In International Conference on Learning Representations, 2024, URL httppss//opeeeeeccc..ooo
15

---

Published as a conference paper at ICLR 2025
A ENVIRONMENT DYNAMICS ARE VISIBLE IN THE NORMS OF CONTRASTIVE REPRESENTATIONS
To further investigate our predictions about targeted exploration and robustness arising from the building of rich representations early training, we visualized the norms of the contrastive goal encoder (5g)())) 1 at early checkpoint, shown in Figure 12 We target goal encoder norms since we observe that mean goal encoder norms over the training distribution rollout positions closely correlate with training loss, and hypothesize that these norms reflect environment-leamiing
Specifically, we fix the end-effector distance to the target distance for gripping. uniformly randomly sample many states x; corresponding both the agent end-effector and object being Iú. and plot the corresponding values [[<)( [ from an early (pre-first-suceess) encoder checkpoint on the sawyer bin task.
As shown in Figure 12 we find interpretable patterns corresponding to a map of environment dynamics, such as high goal encoder norms at the location of the impassable bin walls and bin bottom. We hypothesize that these of higher-level skills and ultimately behaviors such as perturbation robustness.
environment features contribute to the development
0.08
rorm
0.12
555
0.10
6.0
655
0.5
0.06
0.04
0.02
-0.25
-0.20
-0.10
-0.05
0.00
0.30
Figure 12: Contrastive representations capture environment dynamics (impassable bin walls and floor) in an interpretable way. We plot the log of the goal encoder norms, (sx)((+ss,) for various observations where the end effector and block are at the same location. We show a top-down view at box- box-height (left) and side view (right). We find that the impassable bin wall is visible both from the top view (thin strip of lighter blue) and side view (vertical spike of red), represented by relatively higher norms. Example policy rollouts (colored lines) from checkpoints at the same stage in training overlaid for reference, with their starting positions marked as red diamonds and the goal marked as gold star.
16

---

Published as a conference paper at ICLR 2025
B EXPERIMENTAL DETAILS
Table 2: Hyperparameters for our method and the baselines.
hyperparameter
value
Contrastive RL (CRL)
(Eysenbach et al. 2022
batch size
256
learning rate
3e-4
discount
0.99
actor target entropy
hidden layers sizes (policy, critic, representations)
(256, 256)
initial random data collection
10,000 transitions
replay buffer size
le6
samples per insert'
256
representation dimension (dim(((s,a)). a)), dim(1/(s,)))
64
actor minimum std dev
1c-6
SAC (Haarnoja al. 2018)
batch size
256
learning rate
3e-4
discount
0.99
hidden layers sizes (policy, critic)
(256, 256)
target EMA term
5c-3
initial random data collection
10,000 transitions
replay buffer size
le6
samples per insert"
256
actor minimum std dev
1e-6
RIS (Chane-Sane et al. 2021
batch size
256
learning rate
1e-3 (critic), 1e-4 (policy)
high-level policy learning rate
le-4
discount
0.99
hidden layers sizes (policy, high-level policy, critic)
(256, 256)
initial random data collection
10,000 transitions
replay buffer size
1e5
Polyak coefficient for target networks
5e-3
valid state KI. constraint (e)
le-4
subgoal KL penalty (a)
0.1
high-level policy weight regularization (A)
0.1
How many times transition used for training before being discarded.
We collect N traniitoos,, add them to the buffer, and then do N gradient steps using the experience sampled randomly from the beffer.
17

---

Published as a conference paper at ICLR 2025
C
IMPOSSIBLE GOALS
impossible goal
initial state
Figure 13: Single-goal exploration with an impossible goal.
C.I IMPOSSIBLE GOALS.
To further probe single goal exploration, we tried commanding goal that was impossible to reach in maze navigation task. One might expect to see completely random behavior, or see no behavior at all. Visualizing the trajectories visited throughout training (Fig. 13), observe that the agent seems to try to navigate to that impossible goal but then gets stuck. This pattern suggests that commanding an impossible goal is plausible strategy for improving exploration. However, it fails to explore fair number states, suggesting that there are likely more effective ways of inducing exploration in settings without single fixed goal.
18

---

Published as a conference paper at ICLR 2025
D VISUALIZING EMERGENT EXPLORATION IN POLICY ROLLOUTS
of Action Rollouts
t-SNE Action Rollouts
-9
9
Principal Component
t-SNE Component 2
Chockpoint Number
10
-6
6
Checkpoiing
10
5
-10
1..0
3
-30
-1.5
2
2
-40
-1.5
-1.0
-0.5
1.5
-20
Principal Component
t-SNE Component
PCA of State Rollouts
t-SNE of State Rollouts
10
10
40
9
Principal Component 2
t-SNE Component 2
0.2
Creckpoint Number
7
Chockpaint Numborr
6
0.2
-20
3
3
-0.4
0.2
0.0
0.2
0.4
20
20
40
Principal Component
t-SNE Component
Figure 14: Rollouts at early checkpoints indicate distinct exploratory behavior. Here plot action and state rollouts after PCA and t-SNE dimensionality reduction the first checkpoints the model Sawyer Peg environment.
To further analyze the progression of exploration and skill learning, we apply dimensionality reduction on the states encountered and the actions taken for one episode the first 10 checkpoints of the single-goal CRL model on the sawyer peg environment. The checkpoints were saved at fixed time intervals (e.g. every 15 minutes), which corresponded 3,300 trials the environment. We observe that the action and state rollouts for early checkpoints (2,3,4) tend to be more spatially separated in PCA and T-SNE plots (Fig. [4], indicating unique exploratory behavior, whereas the rollouts for later checkpoints (8,9,10) have more overlap, indicating the algorithm has converged to similar behavior patterns once the single-goal has been reached.
19

---

Published as a conference paper at ICLR 2025
E EVALUATION OF SINGLE-GOAL METHOD ON MULTIPLE GOALS
0.75
(Single Hard Goal)
Success
0.50
0.25
0.00
Sawyer Box
Sawyer Bin
Sawyer Peg
CRL (Single Hard Goal) evaluated on Single Hard Goal
CRL (Single Hard Goal) evaluated Range of Goal Difficulties
CRL (Range Difficulties) evaluated on Single Hard Goal
CRL (Range of Difficulties) evaluated Range of Goal Difficulties
Figure 15: Generalization to multiple goals. Single-goal CRL can still reach goals along the path from the starting point the goal.
We evaluate single-goal CRL and multi-goal CRL on the distribution goals used for data collection in the multi-goal algorithm in addition to the single, hard goal used for evaluation in the main text. Figure [5]shows that single-goal CRL retains the ability to reach these goals (i.e. accomplish multiple tasks) without being explicitly guided to learn those tasks. The finding is most prominent in the Sawyer Box and Sawyer Peg tasks. This evaluation, in combination with the ablation experiment (see Fig. [0). suggests that single-goal CRL not overfitting its policy parameters on a single task. Instead it performs multi-task learning, which surprisingly improves performance on the single task as well compared to prior methods.
20

---

Published as a conference paper at ICLR 2025
F
SENSITIVITY OF REPRESENTATIONS TO STATE DIMENSIONS
Representation sensitivity
400
300
200
100
10
20
Training checkpoint
Hand Y Pos leasy)
Obj Pos (medium)
Obj Pos (hard)
Figure 16: Representation sensitivity to state dimensions evolves throughout training. We probe single-goal CRL representations trained the Sawyer Box environment for sensitivity thrce state dimensions the hand position, object y position, and the object z position (height). For each checkpoint, we measure representation sensitivity by the 12 distance between the initial state representation before (st) and after (s6) small perturbation of 0.1 to the given state dimension: []ő(sb, au) 6(s0. an)]
We further probe the single-gen goal CRL representation for sensitivity three state dimensions in the sawyer box environment: The position of the hand (end-effector), the y position of the object (box lid position on table), and the z position of the object (box lid height). These dimensions because changing these dimensions requires increasing levels of skill; moving the end-effector position is easy, moving the object y position medium difficulty, and changing the object height is hard. For each training checkpoint, we measure the representation sensitivity by the 12 distance between the initial state representation before (s/) and after (s a small perturbation of 0.1 the given state dimension: []Ø(s), ao) Φ( oo)ll2. We find that the representations are more sensitive to the object y position during earlier stages of training and more sensitive to the object z position during later stages of training. This shift in sensitivity to the difficulty of the state dimension suggests that the method automatically develops reasonable curriculum for exploration.
chosen
21

---

Published as a conference paper at ICLR 2025
G
TRANSFERRING REPRESENTATIONS BETWEEN TASKS
10
(Single Hard Goal)
Success
0.0
Actor Steps
1e7
Figure 17: Transferring representations between tasks may improve sample efficiency. After training the single-goal CRL representation for 15M environment steps to place a block in a box, we then use these representations to initialize agent for solving the Sawyer Bin task.
In this experiment, we evaluate whether transferring single-goal CRL representations between tasks can improve training sample efficiency. Because single-goal CRL representations learn interpretable environment dynamics (see Fig. [2),, one could expect these learned properties to transfer between tasks. Over three random seeds, train CRL representations for 15M environment interactions task that involves placing a block in a box. We then use these representations to initialize an agent for solving the Sawyer Bin task. We find that this process leads to better sample efficiency (compare Fig. I7 and Fig. top left), though the difference is not drastic. Overall, this experiment provides some preliminary indication that single-goal CRL representations transferrable between tasks.
22

---

Published as a conference paper at ICLR 2025
H SINGLE-GOAL AND MULTI-GOAL CRL SHOW SIMILAR PERFORMANCE ON AN EASY GOAL
1.0
(Single Easy Goal)
Success
200000
400000
Actor Steps
CRL (Single Hard Goal)
CRL (Range of Difficulties)
Figure 18: Single-goal and multi-goal CRL show similar performance on an easy goal. We train multi-goal CRL and single-goal CRL on the Point Spiral environment but the single goal for single-goal CRL to be an easy goal close to the starting state. The starting position is marked as the red diamond and the single, easy goal is marked gold star.
To verify our implementation of multi-goal CRL, run experiment where we use single, easy goal instead of a single, hard goal in the Point Spiral environment. We observe that single-gon goal and multi-goal CRL show similar training progress and both are able to completely solve the environment (always reach the easy goal) within 500,000 actor steps. This experiment suggests that the performance difference that observe between single-goal CRL and the other methods is due to the difficulty of the given task.
23


Reviews:
[{"review_id": "QXeLxc2UHK", "reviewer": "Reviewer_Gqvh", "summary": "This work aims to solve goal-conditioned RL (GCRL) tasks where only one goal is desired without access to rewards (dense or sparse), curriculums, or demonstrations. It proposes modifying standard Contrastive RL (CRL) by using only a single goal in rollouts while still using multiple goals while training. The paper runs experiments to show that the specific combination of single-goal rollouts, multi-goal training, and contrastive representations are needed to achieve good results. The paper also includes some additional analysis to examine single-goal CRL under impossible goals and to explore the representations learned by CRL.", "questions": "How are the “difficult goals” sampled, how many are sampled, etc? Why are thy chosen in this way?\n\nAre there other experiments done (e.g. something like the perturbation test, or a test where the single-goal policy shows useful representations or reasonable success on goal states outside of those in the goal path) which would alleviate concerns that this method is simply overfitting to solve for this one goal?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Under the conditions set by the work, i.e. learning single difficult goals without rewards, demonstrations, curriculums, etc, the single-goal CRL clearly outperforms other methods. Furthermore, this method reduces assumptions compared to standard RL. Finally, an important finding of this paper is that a policy can be trained under contrastive learning to find a difficult goal without providing intermediate easier goals.", "weaknesses": "Firstly, it is unclear how the authors selected or defined “difficult goals”. Are the difficult goals sampled by failure cases of CRL? How many difficult goals are the results evaluated on? For example, if this method is effective on one particular goal, but not others, then it would not be effective in practice.\n\nSecondly, the authors claim that the first ablation disproves the hypothesis that single-goal CRL “learns useful representations and an effective policy only for states along that goal path” (line 476). However, since the single-goal CRL fills the replay buffer with rollouts where the policy aims to reach goal $s^*$, and the actor is only being updated with states from the replay buffer, how does this prove that the single-goal rollout + multi-goal training policy is not overfit to/does not only contain good representations for states along the goal path?\n\nFinally, Fig. 11 shows that a monolithic critic CRL (Range of Difficulties) can reach approximately 60% success rate on the single hard goal on Sawyer Box after 100,000 steps. However, Fig. 3 shows that standard CRL (range of difficulties) can only achieve up to approximately 40% success rate on Sawyer Box after 1,000,000 steps. It seems this might contradict the assertion that the separate representations are important? (It is possible that the methodology for selecting the “difficult goals” is impacting these results.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nQBMRcdWKi", "reviewer": "Reviewer_zVRJ", "summary": "The authors describe a curious phenomenon observed in some simulated robot manipulation environments used for RL. When using contrastive RL, using a single hard goal (one far away from the robot corresponding to task success) led to better learning outcomes than using a human designed curriculum of easy and hard goals.\n\nThis is observed across 4 environments, a Sawyer bin placing task, box placing task, peg insertion, and a 2d maze navigating task.\n\nThe authors provide a number of ablation experiments trying to study why this phenomenon occurs.", "questions": "Do the authors have any statistics on eval performance over the same distribution of goals used to generate the replay buffer? One natural argument is that, in so far as each goal can be viewed as a separate task, the model will be best at the goal distribution that appears in the replay buffer. I believe this is distinct from the overfitting experiments, because in those experiments the final evaluation number is still only the final hard eval goal.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The fact that a single hard goal is sufficient with contrastive RL is certainly surprising, and the paper is upfront in not having a good explanation why this occurs. The fact it occurs over a few environments provides some evidence it is not simply a one-off occurrence.", "weaknesses": "I am suspicious that the results are too good - that there was no environment where using multiple goals performed best. Broadly my concern is that maybe these environments are too easy. If the model is able to succeed at the task given just the final hard goal, perhaps it's too hard to design a good dense reward or curriculum of goals to speed up learning. This wouldn't be too surprising, it's often been remarked that good partial reward design is difficult.\n\nI'm also not sure how \"single goal\" the final method is. In particular, Figure 10's caption was confusing to me. It seemed to suggest that in their method, the actor loss uses multiple goals, rather than a single goal? If so, this doesn't really seem like \"1 goal is all you need\". Essentially I think the authors may be overgeneralizing their conclusions.\n\nMy understanding so far is this:\n\n* A contrastive critic is learned via contrastive RL, defining reward by $\\phi(s,a)^T \\psi(s_f)$, where $s_f \\sim Geom(1-\\gamma)$ steps in the future\n* When generating data, we use a single hard goal $s^*$ and act according to $\\pi(a|s,s^*)$\n* When updating the actor, we sample a trajectory from the replay buffer of data generated according to $\\pi(a|s,s^*)$. But, for each initial $s$, we then sample $s_f \\sim Geom(1-\\gamma)$ within the trajectory, and apply gradient updates as if we collected data according to $\\pi(a|s,s_f)$, even though we actually collected data according to $\\pi(a|s,s^*)$.\n\nIn which case, at most you could say 1 goal is the only requirement for data collection, but the policy still needs to be trained on every intermediate goal observed to achieve good performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kyTDLhMzz8", "reviewer": "Reviewer_gayD", "summary": "This work builds upon previous work in contrastive reinforcement learning. What's unique to this work is how their learning algorithm conditions on a single goal; this has implications on the distribution of data within the replay buffer which ultimately impacts the learning dynamics and exploration behaviour of the agent. The authors argue that this approach results in emergent skill learning without explicitly having to define a reward function or learning curriculum for the task being solved. This argument is supported with simulated experiments on robotic manipulation and maze traversing benchmarks. The performance of their approach is compared to contrastive reinforcement learning using a range of human-specified goal difficulties and existing approaches (SAC+HER and RIS). The authors justify their claim with empirical results, leading to the insight that contrastive reinforcement learning with conditioning on a single goal demonstrates advantageous exploration and skill learning behaviours relative to existing approaches that seek to learn effective exploration and skill learning strategies.", "questions": "Thank you for contributing this work. When reading the paper the emergent skill learning argument felt like it needed further justification. \n\n\n(1) I wished to clarify if the following pattern of skill emergence always held? The paper suggests yes but I wished to confirm this point. \n\n\"For example, in all three environments, the agent (1) first learns how to move its end-effector to\nvarying locations, (2) then learns how to nudge and slide the object, and (3) finally learns to pick up\nand direct the object. \"\n\n(2) I also wished to ask when you mention fixed sampling intervals for checkpoints, what fixed interval was used? Were there any exceptions to the skill learning trends identified in (1) (e.g. checkpoints with purely random behaviours) and if so how frequent were exceptions?\n\nIn answering (1) and (2) it would be useful to provide:\n\n(a) Video's indicating the progression of skill learning with details of the checkpoints, this would be helpful for validating the qualitative behaviours claimed within the paper. \n\n(b) Not entirely necessary but it would be nice to have quantitative metrics for characterising the various skills. For instance when you mention learning how to move the end-effector, distinguishing between purely random movements and directed movement would be useful, there are certain quantitative metrics that could be used, such as correlations between actions. Contrasting this with naive exploration approaches such as SAC may yield insights and further validate the skill learning claim. Similarly for learning to interact with objects and eventually pick them, characterising these skills quantitatively would be useful. \n\n(c) It may be possible to perform clustering on policy rollouts data to identify distinct exploratory behaviours as learning progresses. This would be a valuable addition.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors provide valuable insights into the effectiveness of training a policy conditioned on a single goal with contrastive reinforcement learning in the single-task setting. \n\n- The authors contribute to the discussion on skill learning through empirically demonstrating the emergence of skill learning with contrastive objective functions.\n\n- The authors validate each of their claims and provide discussions on counter arguments (e.g. overfitting single-task) that support their empirical results. \n\n- The authors provide sufficient information and high-quality materials to reproduce their results.", "weaknesses": "- There could be more analysis of emergent skill learning, in the current draft the authors mention that fixing a checkpoint and examining the qualitative behaviour of the policy they see evidence of skill learning, further justification of this claim is required in my opinion. \n\n- The authors highlight an awareness of this point but further analysis on why this strategy works would be useful. \n\n- It would be interesting to see if this approach results in exploration strategies that are capable of solving more complex tasks. For example tasks that encounter Sussman's anomaly, where naive subgoal design may not work well. I am unsure how this approach would perform but it would be promising if it did well and might further justify the strength of this more general approach to learning to solve tasks with reinforcement learning. Also tasks with significant bottlenecks in which naive exploration strategies are often incredible sample inefficient, does this approach help improve performance on such tasks?\n\n- While the authors address the concern of overfitting to the individual task being solved, they do not explore how well the learnt representations generalise. While there are examples of solving tasks with minor perturbations there isn't a discussion of more significant changes to the task environment. From a practical standpoint, it would be interesting to understand how well this approach can be used to learn value representations that generalise well across task environments. \n\n- This work suffers from ailments of reinforcement learning approaches more generally, most especially sample efficiency. It would be interesting to understand if the representations learnt in this work can be leveraged to address the issue of sample efficiency when learning multiple tasks (e.g. on the metaworld benchmark).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)