A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoals

## Abstract

Abstract In this paper, we present empirical evidence of skills and directed exploration emerging from a simple RL algorithm long before any successful trials are observed. For example, in a manipulation task, the agent is given a single observation of the goal state (see Fig. 1 ) and learns skills, first for moving its end-effector, then for pushing the block, and finally for picking up and placing the block. These skills emerge before the agent has ever successfully placed the block at the goal location and without the aid of any reward functions, demonstrations, or manually-specified distance metrics. Once the agent has learned to reach the goal state reliably, exploration is reduced. Implementing our method involves a simple modification of prior work and does not require density estimates, ensembles, or any additional hyperparameters. Intuitively, the proposed method seems like it should be terrible at exploration, and we lack a clear theoretical understanding of why it works so effectively, though our experiments provide some hints. Videos and code: https://graliuce.github.io/sgcrl

## 1 Introduction

Exploration is one of the grand challenges in reinforcement learning (RL) []. Effective exploration algorithms would enable RL agents to solve long-horizon, sparse reward problems with minimal human supervision: no need for dense reward functions, demonstrations, or hierarchical RL.
While there is a long history of exploration methods, even today’s best methods fail to explore in settings with sufficiently sparse rewards, and the complexity of sophisticated exploration techniques means that most researchers today employ limited exploration methods (e.g., adding random noise to actions [, , , ]).

In this paper, we focus on a specific type of RL problem [, , ]: the agent is given an observation of the single desired goal state, which it tries to reach. This problem setting captures many practical problems, from cell biology (grow a certain cell type) to chemical engineering (create a specific molecule) to video games (navigate to the final room). However, this problem setting is exceedingly challenging for standard RL methods, as the agent does not receive any reward feedback about *how* it should solve the task. In continuous settings, the agent will never reach the goal *exactly*, so no reward signal is ever observed. Because of the difficulty of this exploration problem, prior work typically assumes that a human user can provide a dense reward function [, ] (or distance metric/threshold [, , ]) or a set of easier training goals(^1^11Some prior methods automatically propose training goals [, , , ], yet these methods require additional machinery and are primarily evaluated on settings where subgoals lie on a 2-dimensional manifold.) []. However, constructing these reward functions or easier goals is challenging [, ] and stymies potential applications of RL: a chemist who wants to synthesize one particular molecule would have to write down several “easier” molecules for the RL agent to reach. Our paper lifts the assumptions of prior work by considering a setting that is easier for human users but significantly more challenging for RL agents: a single goal state is provided and is used for both training and evaluation.

We present a simple RL algorithm where skills and directed exploration emerge long before any successful trials are ever observed. For example, in a manipulation task where the agent is given a single observation of the goal state (see Fig. 1), the agent ends up learning skills for moving its end-effector, then for pushing the block, then for lifting the block, and finally for picking up the block. These skills are learned before the agent has ever succeeded at placing the block in the correct location and without any reward functions, demonstrations, or manually-specified distance metrics. Once the agent has learned to reliably reach the goal state, it slows exploration. Implementing this method involves a simple modification of prior work and does not require density estimates, ensembles, or any additional hyperparameters.

Our method works by learning a goal-conditioned value function via contrastive RL (CRL) [] and using that value function to train a goal-conditioned policy. The key ingredient is embarrassingly simple: when doing exploration, always condition the goal-conditioned policy on the single target goal.
There are several intuitive reasons why we initially thought this method should work poorly:

- *(i)*
Before the single target goal is reached, the value function will predict bogus values for that goal, so it should be unable to train the policy.
- *(ii)*
There is no mechanism to drive exploration. Sampling a curriculum of goals that includes easy goals and leads to the target goal should perform much better.

This intuition turned out to be fallacious.

Empirically, we evaluate our approach on tasks ranging from bin picking to peg insertion to maze navigation, finding that it significantly outperforms prior methods that use a manually-designed curricula of subgoals [], methods that automatically propose subgoals for training [], and even methods that use dense rewards []. While we do not claim that this is the best exploration method, it outperforms all alternative exploration strategies we have tried. Not only do we observe emergent skills during training, we find that different random seed initializations learn divergent strategies for solving the problem. While we still lack a theoretical understanding of *why* this approach is so effective, experiments highlight that (1) the contrastive representations used to express the value function are important, and that (2) the gains are not caused by “overfitting” the policy or the value function to the single target goal.

## 2 Related Work

Our work builds on a long line of prior work in exploration methods for reinforcement learning [ladosz2022exploration, , , , , , , , ], and will study this problem in the specific setting of goal-conditioned RL (GCRL) [, , , , , , , , , , , , , , , , , ]. This section reviews three types of strategies for exploration. Our proposed method aims to lift the limitations associated with these prior methods.

#### Rewards and demonstrations.

One of the key challenges with GCRL is the sparsity of the learning problem, so many prior GCRL methods assume access to a dense, hand-crafted reward function [, ] or a distance metric [, , , ]. Other methods attempt to make GCRL more tractable by using expert demonstrations [, ] to guide learning and planning. Although well-designed reward functions and expert demonstrations are useful for training, these components add complexity, and collecting demonstrations can be challenging. Our method builds upon a growing collection of GCRL algorithms that require neither a reward function nor demonstrations [, , , , , ] – specifically, we consider a variant of GCRL where only a single goal is provided for training and evaluation. As such, we could treat it as a single-task problem, but we find that treating it as a multi-task problem is crucial to achieving good performance.

#### Exploration and subgoal sampling.

Without a dense reward function or expert demonstrations, the primary challenge of GCRL is effective exploration.
One class of exploration strategies adds noise to the actions [, , , ] or policies [, ]. While these methods are simple to implement, they typically fail to perform directed exploration [].
A second class of methods formulate an intrinsic exploration reward [, , , , , ], which the agent aims to optimize in addition to whatever rewards (dense or sparse) that are provided by the environment. While these methods can work effectively, they can be challenging to scale to high-dimensional and long-horizon tasks.
A third class of methods use probabilistic techniques, including ensembles [, , , , ], posterior sampling [, , , ], and uncertainty propagation [, ] – these methods can excel at directed exploration, though challenges include tuning the prior and dealing with large ensembles.
A fourth class of methods modifies the goals that are used in training.
For example, some methods automatically propose subgoals, breaking down a hard task into a sequence of easier tasks [, , , ]. Other methods automatically adjust the goal distribution [, , ] or initial state distribution [], so that the difficulty of learning increases throughout training. Despite excellent results in certain settings, scaling these methods beyond 2D navigation remains challenging, and the algorithms remain complex. We compare against one prototypical subgoal sampling method (RIS []).

#### Multi-task learning for single-task problems

The last strategy for exploration is so ubiquitous it is easy to forget: training on multiple related tasks, even when we only care about performance on a single difficult task. For example, many prior GCRL methods command a range of goals during exploration. Intuitively, the easy tasks can be learned with little exploration, and learning those tasks should enable the agent to solve more challenging tasks (similar to curriculum learning [, , ]). However, actually constructing these multiple training tasks or goals requires additional human supervision: the human often lays out a “trail of breadcrumbs”, and the agent learns how to navigate to each []. Our paper will study the setting where only a single goal is provided for training, yet our experiments will compare against baselines that have access to training goals with a range of difficulties.

## 3 Single-Goal Exploration with Contrastive RL

This section outlines the problem statement and the approach used in our experiments.

Figure: Figure 2: Single-goal exploration. (Left) Our method uses a single difficult goal for both data collection and evaluation. It is exceedingly unlikely that a random policy would ever reach this goal. (Right) Typical methods for goal-conditioned RL use a range of different goals for exploration, even if the user only cares about success at reaching a single difficult goal. These different goals can be provided by the user [] or generated with a GAN [], VAE [], or planning [, , ].
Refer to caption: x2.png

### 3.1 Preliminaries

#### Notation.

We consider a controlled Markov process (i.e., an MDP without a reward function) defined by time-indexed states $s_{t}$ and actions $a_{t}$.
Our experiments will use continuous states and actions.
The initial state is sampled $s_{0}\sim p_{0}(s_{0})$ and subsequent states are sampled from the Markovian dynamics $s_{t+1}\sim p(s_{t+1}\mid s_{t},a_{t})$. Without loss of generality, we assume that episodes have an infinite horizon; finite horizon problems can be handled by augmenting the dynamics with an absorbing state. We assume that the algorithm is given as input the single target goal state $s^{*}$ and aims to learn a policy $\pi(a_{t}\mid s_{t})$ by interacting with the environment. *Unlike prior work, we do not assume that a distribution of goals for exploration is given; we do not assume that either a dense or sparse reward function is given.*

Following prior work [, ], we define the objective as maximizing the probability of reaching the goal. Formally, define the $\gamma$-discounted state occupancy measure [, , ] as

$$ $\displaystyle\rho^{\pi}(s_{f})\triangleq(1-\gamma)\sum_{t=0}^{\infty}\gamma^{t }p_{t}^{\pi}(s_{t}=s_{f}),$ (1) $$

where $p_{t}^{\pi}(s_{t}=s_{f})$ is the probability of being at state $s_{f}$ at time step $t$. In continuous settings, $p_{t}^{\pi}(s_{f})$ is a probability *density*. The objective is to find a policy that maximizes the likelihood of the single target goal under this occupancy measure:

$$ $\displaystyle\max_{\pi}\rho^{\pi}(s_{f}=s^{*}).$ (2) $$

In discrete settings, this objective is equivalent to the standard discounted reward objective with a reward function $r(s_{t},a_{t})=\mathbbm{1}(s_{t}=s^{*})$; in continuous settings, it is equivalent to using a reward function $r(s_{t},a_{t})=p(s^{\prime}=s^{+}\mid s_{t},a_{t})$. Intuitively, this corresponds to maximizing the time spent at the goal.
The hyperparameter $\gamma\in[0,1)$ is part of the problem description.

#### Contrastive RL.

Our method builds on *contrastive RL* [], prior work that uses temporal contrastive learning to solve goal-conditioned RL problems. This method was designed for a slightly different setting, where the input is a distribution over goals $p(g)$, and the aim was to learn a goal-conditioned policy $\pi(a\mid s,g)$ for reaching each of these goals.
Contrastive RL is an actor-critic method. The critic $C(s,a,s_{f})$ is learned so that it outputs the (relative) likelihood that an agent starting at state $s$ and taking action $a$ will visit state $s_{f}$. Following prior work, we parameterize the critic as the dot product between two learned representations, $\phi(s,a)^{T}\psi(s_{f}))$. The representations are not normalized. We will write the loss function in terms of these representations, which will turn out to be key for achieving good exploration.

To define the learning objective, we introduce a few distributions.
Define $p(s,a)$ as the marginal distribution over state-action pairs in the replay buffer, and define $\rho(s_{f}\mid s,a)$ as the empirical discounted state occupancy measure, conditioned on a state $s$ and action $a$. Define $\rho(s_{f})$ as the corresponding marginal distribution over future states.
Contrastive RL uses these distributions to train the critic with a contrastive learning objective. Following prior work, we learn these representations using the infoNCE contrastive objective [] together with a LogSumExp regularization that prior analysis [] has shown is necessary when using the infoNCE objective for control:

$$ $\displaystyle\max_{\phi(s,a),\psi(s_{f})}\mathbb{E}_{\begin{subarray}{c}(s,a) \sim p(s,a),s_{f}^{(1)}\sim\rho(s_{f}\mid s,a)\\ s_{f}^{(2:N)}\sim\rho(s_{f})\end{subarray}}\Bigg{[}\underbrace{\log\bigg{(} \frac{e^{\phi(s,a)^{T}\psi(s_{f}^{(1)})}}{\sum_{j=1}^{N}e^{\phi(s,a)^{T}\psi(s _{f}^{(j)})}}\bigg{)}}_{\text{infoNCE}}-0.01\cdot\underbrace{\log\bigg{(}\sum_ {j=1}^{N}e^{\phi(s,a)^{T}\psi(s_{f}^{(j)})}\bigg{)}^{2}}_{\text{LogSumExp regularization}}\Bigg{]}.$ (3) $$

In practice, this loss is implemented by sampling a random $(s_{t},a_{t})$ pair from the replay buffer and then sampling a future state $s_{f}=s_{t+\Delta}$ by looking $\Delta\sim\textsc{Geom}(1-\gamma)$ steps ahead. The negative examples are obtained by shuffling the future states (i.e., sampling from the product of two marginal distributions).

Once learned, the representations encode a Q-value []: $\phi(s,a)^{T}\psi(s_{f})=\log Q(s,a,s_{f})-\log\rho(s_{f})$,
where the Q value is defined with respect to the reward function introduced above.
The policy is learned to maximize this (log) Q-value:

$$ $\displaystyle\max_{\pi}\mathbb{E}_{p(s)p(g)\pi(a\mid s,g)}\left[\phi(s,a)^{T} \psi(s_{f})+\alpha\mathcal{H}(\pi(\cdot|s,s_{f}))\right],$ (4) $$

where $\alpha$ is an adaptive entropy coefficient. Intuitively, the actor loss chooses the action $a$ that maximizes the alignment between $\phi(s,a)$ and $\psi(s^{*})$.

### 3.2 Our Approach

We now describe our approach to tackling the problem of learning to reach a single goal state. Our approach is a simple modification of contrastive RL: rather than asking the human user to provide many training goals for exploration, we always command the policy to collect data with the single hard goal $s^{*}$ (see Fig. 2). No other modifications are made to the algorithm. The critic loss is the same as contrastive RL (Eq. 3). The actor loss is the same (Eq. 4). We will call this method “single [hard] goal CRL.” However, note that this is a multi-task algorithm because multiple goals are used for training the actor (Eq. 4), even though it only ever collects data conditioned on a single goal and success is evaluated on a single goal. Ablation experiments (Fig. 11) show that only training the actor on the single goal decreases performance. We summarize the approach in Alg. 1.

Figure: Algorithm 1 Single-goal Exploration with Contrastive RL. The difference from most prior methods is that exploration is done by commanding a single difficult goal $s^{*}$, rather than sampling goals with a range of difficulties.

## 4 Experiments

The main aim of our experiments is to evaluate the performance of single-goal contrastive RL compared to its multi-goal counterpart as well as prior baselines. We do so on four exploration-heavy, goal-reaching tasks, involving robotic manipulation and maze navigation.
All experiments were run with five random seeds, and error bars in the plots depict the standard error.
Hyperparameters can be found in Appendix B and code to reproduce our results is available online: [https://graliuce.github.io/sgcrl](https://graliuce.github.io/sgcrl)

#### Tasks.

We measure the efficacy of our method on four goal-reaching tasks taken from prior work [], which are chosen to measure long horizon exploration. The tasks include three robotic manipulation environments [] and one maze navigation environment []. The robotic manipulation tasks require controlling a sawyer robot to grasp an object (e.g., a block, lid, or peg) and accurately place it in a predetermined location (e.g., in a bin, on top of a box, or inside a hole). The point spiral task is a 2D maze navigation task. We quantify success in each episode by whether the agent reached sufficiently close to the goal in at least one state in an episode. This 0/1 sparse reward signal is used by a few of our baselines but is not needed by contrastive RL.

These tasks present exceedingly difficult exploration challenges. To quantify the difficulty, we measured the success rate under a uniform random policy: for each of the sawyer environments, we did not see a single success state in 75,000,000 environment steps (600,000 episodes); for the point spiral environment, we did not see a single success state in 40,000,000 environment steps (400,000 episodes). Previous methods solve these tasks with the aid of dense rewards [], or with a rich curriculum of subgoals [], but the single-goal CRL agents must accomplish these tasks with no hand-crafted rewards, demonstrations, or subgoals.

### 4.1 Single-goal Exploration is Exceedingly Effective.

Figure: Figure 3: Single goal Exploration is Highly Effective. We compare single hard goal exploration (command the single hard goal in every trial) to “range of difficulties” exploration (sampling uniformly from a human-provided set of easy/medium/hard goals). In each of the four environments, single-goal exploration yields considerably higher success rates, all while being easier for the human user.
Refer to caption: x3.png

Figure: Figure 4: Skills and Directed Exploration for Putting a Lid on a Box: This manipulation task contains an open box and a lid. The single fixed goal has the lid placed neatly on top of the center of the box. The images above show skills acquired throughout the course of learning. Note that some skills unlock subsequent skills (e.g., reaching is a prerequisite for picking, which is a prerequisite for placing) while others look like open-ended “play” (flipping the lid over, pushing the lid away from the box).
Refer to caption: x4.png

#### A single goal works well.

We find that Contrastive RL effectively solves these four tasks: equipped only with a single target goal, the agent automatically explores the environments and learns complex manipulation skills (see Fig. 3). We compare this method to an “oracle” variant that is trained on human-designed goals that vary in difficulty, ranging from easy goals to the single hard goal. Surprisingly, our proposed method significantly outperforms this “range of difficulties” method.

While we have never seen a random policy solve any of these tasks, our method achieves its first success within thousands of trials: 3,197 trials for sawyer box, 9,329 trials for sawyer bin, 15,895 trials for sawyer peg, and 11,724 trials for the spiral task.(^2^22These numbers are averaged across the random seeds.)

#### Early training: agent develops an emergent curriculum of skills.

Not only does single-goal CRL consistently achieve high success rates on manipulation tasks, but it also demonstrates complex and directed exploration techniques during early training. To observe the agent’s behavior throughout training, we saved learning checkpoints at fixed intervals and visualized the agent’s behavior at each checkpoint (see Figures  1,  4 and 5). We found that the agent learns simple skills before complex ones. For example, in all three environments, the agent (1) first learns how to move its end-effector to varying locations, (2) then learns how to nudge and slide the object, and (3) finally learns to pick up and direct the object. We observe a wide array of exploratory behavior in early training that is not directly connected to the goal: from punching the box lid to flip it over (Fig. 4) to pushing the block far away in a random direction (Fig. 1).

Figure: Figure 5: Skills and Directed Exploration for Peg Insertion: This manipulation task contains a peg and box with a narrow hole; the single fixed goal is a state where the peg is inside the hole. The agent acquires a sequence of increasingly complex skills throughout training, some of which are important for solving the task (e.g., reaching, grasping) while others are more “playful” (e.g., knocking the peg against the box). The agent also learns to recover from mistakes (see Fig. 8).
Refer to caption: x5.png

Figure: Figure 6: Quantifying Exploration. We analyze “single hard goal” exploration by discretizing the object’s XY position and counting the cumulative number of unique positions visited throughout training. Exploration starts to plateau after the agent can successfully reach the goal (compare with Fig. 3).
Refer to caption: x6.png

#### First successes: agent trades off exploration for exploitation.

As the agent learns to reach the goal more consistently, the agent’s behavior becomes less exploratory, qualitatively similar to UCB exploration [, , , , ]. To quantify exploration, we discretized the state space of the robotic manipulation environments and recorded the cumulative number of unique positions visited throughout training. Fig. 7 shows that the growth rate of this exploration metric decreases as the success rate increases (compare with Fig. 3). For the sawyer box and sawyer peg environments, the agent achieves a high success rate earlier in training, which corresponds to the earlier plateau of the unique grid cells curve. For the sawyer bin environment, the agent takes longer to reach high success, and the exploration metric does not start leveling out until the end of training. This trend highlights how single-goal CRL develops a self-directed exploration strategy that automatically trades off between exploration and exploitation.

#### Consistent successes: agent finds diverse paths to the goal.

Not only is the performance of single-goal CRL reproducible (all random seeds solve the manipulation tasks), but policies trained with different random seeds learn qualitatively different goal-reaching strategies. For example, Fig. 7 shows the strategies learned by different random seeds on the sawyer bin task. One seed consistently moves the block flush against the wall of the red bin and flicks it into the blue bin. Another seed tends to grasp the block, lift it, and gently drop it in the blue bin. A third seed chooses between these strategies depending on whether the block starts near the wall or away from the wall. Without explicit human guidance in the form of rewards, demonstrations, or subgoals, we see multiple creative and divergent strategies emerge for solving the same problem.

#### Further training: agent develops robustness and self-recovery.

During later stages of training, we observe that the agent demonstrates robustness and learns to recover from mistakes. For example, in the sawyer peg environment, when the agent drops the peg, it is able to recover by bending down and grasping the peg again. To quantify robustness, we ran perturbation experiments in the sawyer peg and sawyer box environments, in which we randomly perturbed the target object’s location between 0 and 0.05 meters along each of the three axes. We tested two settings: (1) perturbation at the start of the episode (“static perturbations”) and (2) in the middle of an episode (“dynamic perturbations”: $t=20$ for sawyer box and $t=50$ for sawyer peg). As shown in Fig. 8, single goal exploration is robust to static perturbations and somewhat robust to dynamic perturbations, notably outperforming multi-goal CRL in three out of four scenarios. We hypothesize that this is also the result of better exploration, which leads to learned representations that generalize better across unusual or unseen states.

Figure: Figure 8: Robustness to perturbations: Single (Hard) Goal exploration results in policies that are more robust to environment perturbations, as compared to policies trained with goals ranging in difficulty. The success rate remains high even when the object is perturbed at the start (“static”) or in the middle (“dynamic”) of an episode, likely because its effective exploration means that it has seen a wide range of states during training.
Refer to caption: x9.png

### 4.2 The RL Algorithm is Key

We compare against a number of algorithms to investigate the importance of the underlying RL algorithm: is single goal exploration useful for other goal-conditioned RL algorithms?

Figure: Table 1: Baselines: Assumptions for methods used in the experiments below.
Refer to caption: x10.png

#### Baselines.

We single-goal CRL against prior methods that aim to address the sparse reward problem by making additional assumptions or employing additional machinery for exploration.
Reinforcement learning with imagined subgoals (RIS) []maintains a high-level policy that predicts subgoals halfway to the end goal and learns the behavioral policy to reach both the subgoal and the end goal. We also employ a few variants of Soft Actor-Critic (SAC) with additional assumptions: SAC with sparse rewards, SAC+HER with sparse rewards, and SAC with dense rewards. In the sparse rewards setting, the agent receives a reward of 1 near the goal and 0 otherwise. In the dense reward setting, the agent receives a continuous reward tailored to the environment, using the distance to the goal for the spiral task and the Metaworld [] reward function for sawyer tasks. Table 1 summarizes the assumptions for each method.

#### Results.

As shown in Fig. 9 single-goal contrastive RL significantly outperforms these alternative methods, showing that the underlying RL algorithm is important for single goal exploration. Notably, prior methods rarely reach the goal at all, with the exception of RIS on the simplest task (point spiral).
To verify that our implementation of SAC was correct, we also plotted the reward function throughout training (recall that SAC has access to a reward function, while other methods do not) and observed that it increases throughout the course of learning.

Figure: Figure 10: Ablation experiments. While our method uses an actor loss that uses many goals (Eq. 4), alternatives that train the policy with a single goal perform no better.
Refer to caption: x11.png

### 4.3 Why does Single-Goal Exploration Work?

In this section, we present ablation experiments that provide insight into the workings of single-goal CRL. We find that using single-goal data collection and an inner product critic are both key to the effectiveness of the method. Additionally, the performance boost of single-goal data collection does not seem to be due to overfitting of the policy parameters on the task.

#### The effectiveness of single-goal
exploration is not explained by overfitting.

One possible explanation for the method’s success is that the algorithm overfits its policy parameters on the single-goal task. That is, if the agent only collects data conditioned on the single goal, the algorithm learns useful representations and an effective policy only for states along that goal path. To test this hypothesis, we compared our method (which randomly samples different goals (see Eq. 4)) with a variant that always uses the single hard goal in the actor loss. Note that this single hard goal is the one that is used for evaluation.
If overfitting were occurring, we would expect that modifying the actor loss to only train on the single goal would boost performance. However, the results in Fig. 11 show that this is not the case. When data are collected with a single goal, using a single goal in the actor loss degrades performance (red vs. blue (ours)). When data are collected with multiple goals, using a single goal in the actor loss gives only a slight boost performance (purple vs. black).
In short, the performance of single goal exploration is not explained by overfitting.

#### Representations are important.

Our next set of experiments study how the representations might drive exploration.
To do this, we replaced the inner product critic function ($\phi(s,a)^{T}\psi(s_{f})$) with a monolithic critic function ($Q(s,a,s_{f})$), which takes as input a concatenated array of the state, action, and goal.(^3^33In this experiment only, we decreased the batch size from 256 to 32 for both methods, as otherwise we encountered out-of-memory errors for the monolithic method.)
The results, shown in Fig. 11, show that single-goal exploration is not effective with using a monolithic critic network.
This experiment suggests that the contrastive representations $\phi(s,a)^{T}$ and $\psi(s_{f})$ drive exploration, though the precise mechanism for *how* they drive exploration remains unclear. In summary, single-goal exploration is only effective when combining the right algorithm (CRL) with the right critic architecture. In Appendix A, we show that environment dynamics are reflected in the contrastive representations early in training.

Figure: Figure 12: Single-goal exploration with an impossible goal.
Refer to caption: extracted/5785779/figures/impossible_goal.png

### 4.4 Impossible goals.

To further probe single goal exploration, we tried commanding a goal that was impossible to reach in a maze navigation task. One might expect to see completely random behavior, or see no behavior at all. Visualizing the trajectories visited throughout training (Fig. 12), we observe that the agent seems to try to navigate to that impossible goal but then gets stuck. This pattern suggests that commanding an impossible goal is a plausible strategy for improving exploration. However, it fails to explore a fair number of states, suggesting that there are likely more effective ways of inducing exploration in settings without a single fixed goal.

## 5 Conclusion

In this paper, we showed that skills and directed exploration emerges from a straightforward RL algorithm: contrastive RL where every trajectory is collected by trying to reach a single fixed goal.
The resulting method has many appealing properties of prior exploration methods: in each episode the agent seems to try to visit some new state or attempt some new behavior. We do not see the random dithering that is common with naïve exploration methods like $\epsilon$-greedy. Moreover, this method does not require *any* additional hyperparameters: this is in contrast to even the simplest exploration algorithms today, which require a scale parameter (e.g., Gaussian noise in TD3 []). And, while prior exploration algorithms include a schedule (more hyperparameters!) for gradually decreasing the degree of exploration throughout the course of learning, such behavior emerges automatically from our proposed method.

There remain two important outstanding questions raised by our experiments.
First, we lack a clear understanding of why skills and directed exploration emerge. Our experiments provide some hints (representations are important; it is not explained by overfitting), yet much theoretical work remains to be done to understand exactly what is driving the exploration. A rich theoretical understanding of the mechanisms driving the exploration here is important not only for explaining the success of this method, but it may also provide insights into how to adapt the method here to other settings.
Second, how can we leverage the success of the proposed method to address exploration in other problem settings (e.g., if a reward function were given or if not even a single fixed goal were available). The “impossible goal” experiment in Fig. 12 provides one simple approach, but there likely exist significantly better methods of exploiting the emergent exploration properties that we have observed in the single-goal setting.

The emergence of divergent strategies for solving the bin picking task points towards broader research questions about autonomous capabilities. The RL research community has built much scaffolding to help agents succeed. However, we have shown that without this scaffolding, agents develop unexpected and unique methods for problem-solving. Perhaps this seemingly creative behavior emerges not in spite of but because of the lack of human guidance. We encourage future research to explore what RL can accomplish in the absence of human intervention.

#### Limitations.

The primary limitation of our work is a lack of theoretical analysis explaining *why* skills and directed exploration emerge. Empirically, our experiments are focused primarily on manipulation tasks; we encourage future work to study applications to other settings.

#### Acknowledgments.

Thanks to the members of the Princeton RL lab for feedback on preliminary versions of the work. We are also grateful to Elliot Chane-Sane for corresponding with us on the strengths and limitations of the RIS algorithm.
The experiments in this paper were substantially performed using the Princeton Research Computing resources at Princeton University, which is a consortium of groups led by the Princeton Institute for Computational Science and Engineering (PICSciE) and Office of Information Technology’s Research Computing.

## Appendix A Environment dynamics are visible in the norms of contrastive representations

#### Approach.

To further investigate our predictions about targeted exploration and robustness arising from the building of rich representations early in training, we visualized the norms of the contrastive goal encoder $||\psi(s_{g})||_{2}^{2}$ at an early checkpoint, shown in Figure 13. We target goal encoder norms since we observe that mean goal encoder norms over the training distribution rollout positions closely correlate with training loss, and hypothesize that these norms reflect environment-learning.

Specifically, we fix the end-effector distance to the target distance for gripping, uniformly randomly sample many states $x_{i}$ corresponding to both the agent end-effector and object being at $x_{i}$, and plot the corresponding values $||\psi(x_{i})||_{2}^{2}$ from an early (pre-first-success) encoder checkpoint on the sawyer bin task.

#### Results.

As shown in Figure 13, we find interpretable patterns corresponding to a map of environment dynamics, such as high goal encoder norms at the location of the impassable bin walls and bin bottom. We hypothesize that these strongly-represented environment features contribute to the development of higher-level skills and ultimately behaviors such as perturbation robustness.

Figure: Figure 13: Contrastive representations capture environment dynamics (impassable bin walls and floor) in an interpretable way. We plot the goal encoder norms $\psi(g)\|$ for various observations where the end effector and block are at the same location. We show a top-down view at box-height (left) and side view (right). We find that the impassable bin wall is visible both from the top view (thin strip of lighter blue) and side view (vertical spike of red), represented by relatively higher norms. Example policy rollouts (colored lines) from checkpoints at the same stage in training are overlaid for reference, with their starting positions marked as red diamonds and the goal marked as a gold star.
Refer to caption: extracted/5785779/figures/psi_norm_crl_final_seed6_ckpt20_slicez.png

## Appendix B Experimental Details

**Table 2: Hyperparameters for our method and the baselines.**
| hyperparameter | value |
| --- | --- |
| Contrastive RL (CRL) [] |  |
| batch size | 256 |
| learning rate | 3e-4 |
| discount | 0.99 |
| actor target entropy | 0 |
| hidden layers sizes (policy, critic, representations) | (256, 256) |
| initial random data collection | 10,000 transitions |
| replay buffer size | 1e6 |
| samples per insert^1 | 256 |
| representation dimension (dim($\phi(s,a)$), dim($\psi(s_{g})$)) | 64 |
| actor minimum std dev | 1e-6 |
| SAC [] |  |
| batch size | 256 |
| learning rate | 3e-4 |
| discount | 0.99 |
| hidden layers sizes (policy, critic) | (256, 256) |
| target EMA term | 5e-3 |
| initial random data collection | 10,000 transitions |
| replay buffer size | 1e6 |
| samples per insert^1 | 256 |
| actor minimum std dev | 1e-6 |
| RIS [] |  |
| batch size | 256 |
| learning rate | 1e-3 (critic), 1e-4 (policy) |
| high-level policy learning rate | 1e-4 |
| discount | 0.99 |
| hidden layers sizes (policy, high-level policy, critic) | (256, 256) |
| initial random data collection | 10,000 transitions |
| replay buffer size | 1e5 |
| Polyak coefficient for target networks | 5e-3 |
| valid state KL constraint ($\epsilon$) | 1e-4 |
| subgoal KL penalty ($\alpha$) | $0.1$ |
| high-level policy weight regularization ($\lambda$) | 0.1 |
| ^1 How many times is each transition used for training before being discarded. |  |
| ^2 We collect $N$ transitions, add them to the buffer, and then do $N$ gradient steps using the experience sampled randomly from the buffer. |  |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: How are the “difficult goals” sampled, how many are sampled, etc? Why are thy chosen in this way?

Are there other experiments done (e.g. something like the perturbation test, or a test where the single-goal policy shows useful representations or reasonable success on goal states outside of those in the goal path) which would alleviate concerns that this method is simply overfitting to solve for this one goal?
soundness: 2
strengths: Under the conditions set by the work, i.e. learning single difficult goals without rewards, demonstrations, curriculums, etc, the single-goal CRL clearly outperforms other methods. Furthermore, this method reduces assumptions compared to standard RL. Finally, an important finding of this paper is that a policy can be trained under contrastive learning to find a difficult goal without providing intermediate easier goals.
summary: This work aims to solve goal-conditioned RL (GCRL) tasks where only one goal is desired without access to rewards (dense or sparse), curriculums, or demonstrations. It proposes modifying standard Contrastive RL (CRL) by using only a single goal in rollouts while still using multiple goals while training. The paper runs experiments to show that the specific combination of single-goal rollouts, multi-goal training, and contrastive representations are needed to achieve good results. The paper also includes some additional analysis to examine single-goal CRL under impossible goals and to explore the representations learned by CRL.
weaknesses: Firstly, it is unclear how the authors selected or defined “difficult goals”. Are the difficult goals sampled by failure cases of CRL? How many difficult goals are the results evaluated on? For example, if this method is effective on one particular goal, but not others, then it would not be effective in practice.

Secondly, the authors claim that the first ablation disproves the hypothesis that single-goal CRL “learns useful representations and an effective policy only for states along that goal path” (line 476). However, since the single-goal CRL fills the replay buffer with rollouts where the policy aims to reach goal $s^*$, and the actor is only being updated with states from the replay buffer, how does this prove that the single-goal rollout + multi-goal training policy is not overfit to/does not only contain good representations for states along the goal path?

Finally, Fig. 11 shows that a monolithic critic CRL (Range of Difficulties) can reach approximately 60% success rate on the single hard goal on Sawyer Box after 100,000 steps. However, Fig. 3 shows that standard CRL (range of difficulties) can only achieve up to approximately 40% success rate on Sawyer Box after 1,000,000 steps. It seems this might contradict the assertion that the separate representations are important? (It is possible that the methodology for selecting the “difficult goals” is impacting these results.)

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: Do the authors have any statistics on eval performance over the same distribution of goals used to generate the replay buffer? One natural argument is that, in so far as each goal can be viewed as a separate task, the model will be best at the goal distribution that appears in the replay buffer. I believe this is distinct from the overfitting experiments, because in those experiments the final evaluation number is still only the final hard eval goal.
soundness: 2
strengths: The fact that a single hard goal is sufficient with contrastive RL is certainly surprising, and the paper is upfront in not having a good explanation why this occurs. The fact it occurs over a few environments provides some evidence it is not simply a one-off occurrence.
summary: The authors describe a curious phenomenon observed in some simulated robot manipulation environments used for RL. When using contrastive RL, using a single hard goal (one far away from the robot corresponding to task success) led to better learning outcomes than using a human designed curriculum of easy and hard goals.

This is observed across 4 environments, a Sawyer bin placing task, box placing task, peg insertion, and a 2d maze navigating task.

The authors provide a number of ablation experiments trying to study why this phenomenon occurs.
weaknesses: I am suspicious that the results are too good - that there was no environment where using multiple goals performed best. Broadly my concern is that maybe these environments are too easy. If the model is able to succeed at the task given just the final hard goal, perhaps it's too hard to design a good dense reward or curriculum of goals to speed up learning. This wouldn't be too surprising, it's often been remarked that good partial reward design is difficult.

I'm also not sure how "single goal" the final method is. In particular, Figure 10's caption was confusing to me. It seemed to suggest that in their method, the actor loss uses multiple goals, rather than a single goal? If so, this doesn't really seem like "1 goal is all you need". Essentially I think the authors may be overgeneralizing their conclusions.

My understanding so far is this:

* A contrastive critic is learned via contrastive RL, defining reward by $\phi(s,a)^T \psi(s_f)$, where $s_f \sim Geom(1-\gamma)$ steps in the future
* When generating data, we use a single hard goal $s^*$ and act according to $\pi(a|s,s^*)$
* When updating the actor, we sample a trajectory from the replay buffer of data generated according to $\pi(a|s,s^*)$. But, for each initial $s$, we then sample $s_f \sim Geom(1-\gamma)$ within the trajectory, and apply gradient updates as if we collected data according to $\pi(a|s,s_f)$, even though we actually collected data according to $\pi(a|s,s^*)$.

In which case, at most you could say 1 goal is the only requirement for data collection, but the policy still needs to be trained on every intermediate goal observed to achieve good performance.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: Thank you for contributing this work. When reading the paper the emergent skill learning argument felt like it needed further justification. 


(1) I wished to clarify if the following pattern of skill emergence always held? The paper suggests yes but I wished to confirm this point. 

"For example, in all three environments, the agent (1) first learns how to move its end-effector to
varying locations, (2) then learns how to nudge and slide the object, and (3) finally learns to pick up
and direct the object. "

(2) I also wished to ask when you mention fixed sampling intervals for checkpoints, what fixed interval was used? Were there any exceptions to the skill learning trends identified in (1) (e.g. checkpoints with purely random behaviours) and if so how frequent were exceptions?

In answering (1) and (2) it would be useful to provide:

(a) Video's indicating the progression of skill learning with details of the checkpoints, this would be helpful for validating the qualitative behaviours claimed within the paper. 

(b) Not entirely necessary but it would be nice to have quantitative metrics for characterising the various skills. For instance when you mention learning how to move the end-effector, distinguishing between purely random movements and directed movement would be useful, there are certain quantitative metrics that could be used, such as correlations between actions. Contrasting this with naive exploration approaches such as SAC may yield insights and further validate the skill learning claim. Similarly for learning to interact with objects and eventually pick them, characterising these skills quantitatively would be useful. 

(c) It may be possible to perform clustering on policy rollouts data to identify distinct exploratory behaviours as learning progresses. This would be a valuable addition.
soundness: 3
strengths: - The authors provide valuable insights into the effectiveness of training a policy conditioned on a single goal with contrastive reinforcement learning in the single-task setting. 

- The authors contribute to the discussion on skill learning through empirically demonstrating the emergence of skill learning with contrastive objective functions.

- The authors validate each of their claims and provide discussions on counter arguments (e.g. overfitting single-task) that support their empirical results. 

- The authors provide sufficient information and high-quality materials to reproduce their results.
summary: This work builds upon previous work in contrastive reinforcement learning. What's unique to this work is how their learning algorithm conditions on a single goal; this has implications on the distribution of data within the replay buffer which ultimately impacts the learning dynamics and exploration behaviour of the agent. The authors argue that this approach results in emergent skill learning without explicitly having to define a reward function or learning curriculum for the task being solved. This argument is supported with simulated experiments on robotic manipulation and maze traversing benchmarks. The performance of their approach is compared to contrastive reinforcement learning using a range of human-specified goal difficulties and existing approaches (SAC+HER and RIS). The authors justify their claim with empirical results, leading to the insight that contrastive reinforcement learning with conditioning on a single goal demonstrates advantageous exploration and skill learning behaviours relative to existing approaches that seek to learn effective exploration and skill learning strategies.
weaknesses: - There could be more analysis of emergent skill learning, in the current draft the authors mention that fixing a checkpoint and examining the qualitative behaviour of the policy they see evidence of skill learning, further justification of this claim is required in my opinion. 

- The authors highlight an awareness of this point but further analysis on why this strategy works would be useful. 

- It would be interesting to see if this approach results in exploration strategies that are capable of solving more complex tasks. For example tasks that encounter Sussman's anomaly, where naive subgoal design may not work well. I am unsure how this approach would perform but it would be promising if it did well and might further justify the strength of this more general approach to learning to solve tasks with reinforcement learning. Also tasks with significant bottlenecks in which naive exploration strategies are often incredible sample inefficient, does this approach help improve performance on such tasks?

- While the authors address the concern of overfitting to the individual task being solved, they do not explore how well the learnt representations generalise. While there are examples of solving tasks with minor perturbations there isn't a discussion of more significant changes to the task environment. From a practical standpoint, it would be interesting to understand how well this approach can be used to learn value representations that generalise well across task environments. 

- This work suffers from ailments of reinforcement learning approaches more generally, most especially sample efficiency. It would be interesting to understand if the representations learnt in this work can be leveraged to address the issue of sample efficiency when learning multiple tasks (e.g. on the metaworld benchmark).

Meta-review: This paper presents an interesting phenomena in contrastive RL where a single hard goal lead to better performing policies than learning with curricula. Empirical results show that diverse skills emerge during learning long before the agent reaches the goal state and receives a good reward. 

Reviewers agree this is an interesting observation that is potentially impactful for its simplicity and effectiveness. The authors are suggested to revise their presentation of the work (potentially the title) to avoid misunderstandings.

Decision: Accept (Poster)