TLDR: Unsupervised Goal-Conditioned RL via Temporal Distance-Aware Representations
Authors: Junik Bae
Kwanyoung Park
Youngwoon Lee
Venue: CoRL 2024 (2024)
Keywords: Unsupervised Goal-Conditioned Reinforcement Learning
Temporal Distance-Aware Representations

Abstract: Unsupervised goal-conditioned reinforcement learning (GCRL) is a promising paradigm for developing diverse robotic skills without external supervision. However, existing unsupervised GCRL methods often struggle to cover a wide range of states in complex environments due to their limited exploration and sparse or noisy rewards for GCRL. To overcome these challenges, we propose a novel unsupervised GCRL method that leverages TemporaL Distance-aware Representations (TLDR). Based on temporal distance, TLDR selects faraway goals to initiate exploration and computes intrinsic exploration rewards and goal-reaching rewards. Specifically, our exploration policy seeks states with large temporal distances (i.e. covering a large state space), while the goal-conditioned policy learns to minimize the temporal distance to the goal (i.e. reaching the goal). Our results in six simulated locomotion environments demonstrate that TLDR significantly outperforms prior unsupervised GCRL methods in achieving a wide range of states.

TLDR: Unsupervised Goal-Conditioned RL via
Temporal Distance-Aware Representations
Junik Bae
Kwanyoung Park Youngwoon Lee
Yonsei University
https://heatz123. gwth.b.ii//ei
Unsupervised coal-condtiooned reinforcement learning (GCRL) is promising paradigm for developing diverse robotic skills without external supervi- sion. However, existing unsupervised GCRL methods often struggle to cover wide range of states in complex environments due to their limited exploration and sparse or noisy rewards for GCRL. To overcome these challenges, we propose a novel unsupervised GCRL method that leverages Temporal. Distance-awar Representa- tions (TLDR). Based on temporal distance, TLDR selects faraway goals to initiate exploration and computes intrinsic exploration rewards and goal-reaching rewards. Specifically, our exploration policy seeks states with large temporal distances (i.e. covering large state space), while the coal-conditioned policy learns to minimize the temporal distance to the goal (i.e. reaching the goal). Our results in six simu- lated locomotion environments demonstrate that TLDR significantly outperforms prior unsupervised GCRL methods in achieving wide range of states.
Abstract:
Keywords: Unsupervised Goal-Conditioned Reinforcement Learning, Temporal Distance-Aware Representations
Introduction
Human babies can autonomously learn goal-reaching skills, starting from controlling their bodies and gradually improving their capabilities to achieve more challenging goals, involving longer-horizon behaviors. Similarly, for intelligent agents like robots, the ability to reach a large set of states-including both the environment states and agent states-is crucial. This capability not only serves as foundational skill set by itself but also enables achieving more complex tasks.
Can robots autonomously learn such long-horizon goal-reaching skills like humans? This is partic- ularly compelling as learning goal-reaching be-
LC Ei
haviors in robots is task-agnostic and does not require any external supervision, offering a seal- able approach for unsupervised pre-training of robots [3, 4, 5, 6, 7, 9]. However, prior unsu- pervised goal-conditioned reinforcement learning (GCRL) [10, and unsupervised skill discovery [1] methods exhibit limited coverage of reachable states in complex environments, as shown in Figure 1.
(a) TLDR (ours)
(b) METRA
(c) PEG
Figure 1: Trajectories (red) of an ant robot in complex maze trained by TLDR, METRA [1], and PEG [2]. While prior methods yield limited exploration, TLDR explores the entire maze.
The major challenges in unsupervised GCRL are twofold: (1) exploring diverse states that the agent learn achieve, and (2) effectively learning goal-reaching policy. Prior unsupervised GCRL methods focus exploring novel states [1 l] or states with high uncertainty in next state prediction [10, 2]. However, discovering unseen states or state transitions may not lead to meaningful states.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Additionally, training goal-reaching policy to maximize sparse [8] or heuristic [10, 12] goal- reaching rewards is often insufficient for long-horizon goal-reaching behaviors in complex environments.
In this paper, we propose novel unsupervised GCRL method that leverages TemporaL. Distance- aware Representations (TLDR) improve both goal-directed exploration and goal-conditioned policy learning. TLDR uses temporal distance (i.e. the minimum number of environment steps between two states) induced by temporal distance-aware representations [1, 13, 14] for (1) selecting faraway goals to initiate exploration, (2) learning an exploration policy that maximizes temporal distance, and (3) learning a goal-conditioned policy that minimizes temporal distance to a goal.
TLDR demonstrates superior state coverage compared prior unsupervised GCRL and skill dis- covery methods in complex AntMaze environments, shown in Figure 1. Our ablation studies confirm that our temporal distance-aware approach enhances both goal-directed exploration and goal-conditioned policy learning. Furthermore, our method outperforms prior work across diverse locomotion environments, underscoring its general applicability.
2
Related Work
Unsupervised goal-conditioned reinforcement learning (GCRL) aims to learn a goal-conditioned
policy that can reach diverse goal states without external supervision [15, 16, 10, 2]. The major chal- lenges of unsupervised GCRL can be summarized two aspects: (1) optimizing a goal-conditioned policy and (2) collecting trajectories with novel goals that effectively enlarge its state coverage.
To improve the efficiency of goad-conditioned policy learning, hindsight experience reply (HER) [8] and model-based policy optimization [10, 12] have been widely used. However, learning complex, long-horizon goal-reaching behaviors remains difficult due to sparse (e.g. whether it reaches the goal [8] or heuristic rewards (e.g. cosine similarity between the state and goal [10, 12]).
Instead, temporal distance, defined the number of environment steps between states estimated from data, can provide more dense and grounded rewards [17, 10, 18, 19J. LEXA [10] and PEG [2] use the expected temporal distances regarding the current policy as goal-reaching rewards [19]. However, this does not reflect the "shortest temporal distance" " between states, often leading to sub-optimal goal-reaching behaviors. In this paper, we propose to use the estimated shortest temporal distance as reward signals for GCRL, inspired by QRL [14] and HILP [13]. We apply the learned representations to compute goal-reaching rewards rather than directly learning the value function in QRL or using for skill-leaming rewards in HILP.
Exploration in unsupervised GCRL relies heavily on selecting exploratory goals that lead an agent to novel states and expand state coverage. Exploratory goals can be simply sampled from a replay buffer as in LEXA [10], or can be selected from less visited states [20]. states with low-density in state distributions [21, 11], and states with high uncertainty in dynamics [2]. Instead of sampling uncertain or less visited states goals, we select states temporally distant from the visited state distribution goals, encouraging the discovery of temporally farther away states.
In addition to exploratory goal selection, an explicit exploration policy [22, 20] can further encourage
exploration by maximizing intrinsic rewards, such as uncertainty in dynamics used in LEXA and PEG. For better exploration, our approach opts for maximizing temporal distance from the visited states, continuously seeking novel and faraway states.
Unsupervised skill discovery [23, 24, 25, 26, 27, 28, 29, 1] is another approach to learning diverse behaviors without supervision, yet often lacks robust exploration capabilities [29], requiring manual feature engineering or limiting to oow-dimessional state spaces. METRA [1] addresses these limi- tations by computing skill-learning rewards with temporal distance distrnce-aware aware representations. While achieving remarkable exploration and zero-shot goal-reaching capabilities, METRA exhibits limited coverage in complex environments, as depicted in Figure 1, We find that METRA tends focus on reaching the known farthest states rather than exploring less visited states, whereas our exploration strategy explicitly encourages reaching unseen farther states.

---

(a) Learn representations
(b) Select goal
(c) Run GC policy
(d) Run exploration policy
Encode temporal distances into representations
Select temporally farthest goal from visited
Learn to move temporally closer goal
Learn to move temporally farther from visited states
Go-phase
Explore-phase O
(())
visited
x goal O starting state ( new
Figure 2: Overview of TLDR algorithm. TLDR leverages temporal distance-aware representations for unsupervised GCRL. (a) We start by learning state encoder e(s) that maps states to temporal distance-aware representations. With the temporal distance-aware representations, TLDR (b) selects the temporally farthest state from the visited states goal using goal-conditioned policy, which learns to minimize temporal distance to the goal, and (d) collects exploratory trajectories using exploration policy that visits states with large temporal distance from the visited states.
exploratory goal, (c) reaches the chosen
Approach
In this paper, we introduce TemporaL Distance-aware Representations (TLDR), an unsupervised goal-conditioned reinforcement learning (GCRL) method, integrating temporal distance-aware representations (Section 3.2) into every facet of the Go-Explore strategy [20] (Section 3.3), as illustrated in Figure 2. TLDR first chooses a goal from experience (Section 3.4), reaches the selected goal via the goal-conditioned policy, and executes the exploration policy to gather diverse experiences. Both the exploration policy (Section 3.5) and goal-conditioned policy (Section 3.6) are then trained on the collected data and rewards computed using the temporal distance-aware representations. We describe the full algorithm Algorithm and implementation details in Appendix A.
3.1 Problem Formulation
We formulate the unsupervised GCRL problem with goal-conditioned Markov decision process, defined the tuple M A,, S and A denote the state and action spaces, respectively.
p: S A + ((S) denotes the transition dynamics, where ( denotes the set of probability distributions over X'. The goal of the agent is to learn an optimal goal-conditioned policy л G + A, where (a s, g) outputs an action E A that can navigate to the goal g E G from state within minimum steps. In this paper, we set G S, allowing any state as potential goal for the agent.
3.2 Learning Temporal Distance-Aware Representations
Temporal distance, defined the minimum number of environment steps between states, can provide more dense and grounded rewards for goal-conditioned policy learning well as exploration. For GCRL, instead of relying sparse and binary goal-reaching rewards, the change in temporal distance before and after taking an action can be an informative learning signal. Moreover, exploration in unsupervised GCRL can be incentivized by discovering temporally faraway states.
Therefore, in this paper, we propose use temporal distance for unsupervised GCRL. We first estimate the temporal distance by learning temporal distance- representations, inspired by Park et al. [13], Wang et al. [14]. The learned representation S - ZZ encodes the temporal distance

---

Algorithm TLDR: unsupervised goal-conditioned reinforcement learning algorithm
1: Initialize goal-conditioned policy n exploration policy nz. temporal distance-aware represen-
tation 6, and replay buffer ②
2: while not converged do
3: ~ p((0)
4: Sample minibatch
D
5:
g - arg m ( (FLDR(()
D. Select state with the highest TLDR reward (Eq. (2))
6:
for O..... - do
7:
ift To then
8:
a, = 8t,
b Follow goal-conditioned policy no for To steps
9:
else
a ( st)
Explore using exploration policy m
11:
St+1 p(- St, at)
12:
+ DU {s,, ar $c+1}
13:
Train representations & to minimize La in Eq. (1)
14:
Train exploration policy zo to maximize Eq. (3)
15:
Train goal-conditioned policy uy using HER with dense reward in Eq. (4)
between two states into the latent space 2, where (l ((82) represents the temporal distance between i and $2. This representation is then used across the entire unsupervised GCRL algorithm: exploratory goal selection, intrinsic reward for exploration, and reward for goal-conditioned policy.
To train temporal distance representations, adopt QRL's constrained optimization [14]:
max Egpp.gg- [f()((s) (lg) s.t. B(s.a.s') [l] (0ss) )ll s 1, (1)
where is an frfine-tranffomme softplus function that assigns lower weights larger distances [[((s) - (g))l. We optimize this constrained objective using dual gradient descent with Lagrange multiplier X, and we randomly sample and g from minibatch during training.
3.3 Unsupervised GCRL with Temporal Distance-Aware Representations
With temporal distance-aware representations, we can integrate the concept of temporal distance into unsupervised GCRL. Our approach is built upon the Go-Fxplore procedure [20], a widely- used unsupervised GCRL algorithm comprising two phases: (I) the "Go-phase," where the goal- conditioned policy - (a s, g) navigates toward goal g, and (2) the "Explore-phase"" where the exploration policy π (a s) gathers new state trajectories refine the goal-conditioned policy.
While Go-Explore relies on task-specife information for goal selection and executes random actions for exploration, our method uses task-agnostic temporal distance metrics induced by temporal distance-aware representations. The subsequent sections detail how our method leverages the temporal distance-aware representations for selecting goals the Go-phase (Section 3.4), enhancing the exploration policy (Section 3.5), and facilitating the GCRL policy training (Section 3.6).
3.4 Exploratory Goal Selection
For unsupervised GCRL, selecting low-density (less visited) states exploratory goals can enhance goal-directed exploration [15, 16]. However, the concept of the "density" of state does necessarily indicate how rare or hard it is to reach the state. For example, while robotic arm might actively seek out unseen (low-density) joint positions, interacting with objects could offer more significant learning opportunities [29]. Thus, we propose selecting goals that are lemporally distant from states that are already visited (i.e. in the replay buffer) to explore not only diverse but also hard-to-reach states.
To sample faraway goal at the start of each episode, we employ the non- conamaraettiic particle-based entropy estimator [27] top of our temporal distance-aware representations, Among states in

---

Ant
HaliChestah
Humanoid Run
Quadruped Escape
Amıaze Large
AmMaze Ultra
Figure 3: We evaluate our method on 6 state-based robotic locomotion environments.
minibatch, we choose goals with high entropy and collect N corresponding trajectories using goal-reaching policy. The entropy can be estimated follows, which refer TLDR reward:
(S) log 1+
Σ
(
where (.) denotes the k-nearest neighbors around d(s) within a minibatch.
3.5
Learning Exploration Policy
After the goal-conditioned policy navigates towards the chosen goal g for To steps, the exploration policy =g is executed to discover states more distant from the visited states. This objective of the exploration policy can be simply defined as:
(s, s') (TLDR(S') (T).
Similar to LEXA [10], we alternate between goal-reaching episodes and exploration episodes. For goal-reaching episodes, we execute the goal-conditioned policy until the end of the episodes. For exploration episodes, we sample the timestep TG Unif(0, 1) the beginning of each episode and execute the exploration policy if the current timestep > Ta-
3.6 Learning Coal-Conditioned Policy
The goal-conditioned policy aims to minimize the distance to the goal. However, defining "distance" to the goal often requires domain knowledge. Instead, we propose leveraging a task-agnostic metric, temporal distance, as the learning signal for the goal-conditioned policy:
(s, g) [lo(s) æ(g)) - [lø(s') é(g)].-
(4)
If representations accurately capture temporal distances between states, optimizing this reward in greedy manner becomes sufficient for learning optimal goal-reaching policy.
4 Experiments
In this paper, we propose TLDR, a novel unsupervised GCRL method that utilizes temporal distance- aware representations for both exploration and optimizing a coal-conditioned policy. Through our experiments, we aim answer the following three questions: (1) Does TLDR explore better compared other exploration methods? (2) our goal-conditioned policy better than prior unsupervised GCRL methods? (3) How crucial is TLDR for goal-conditioned policy learning and exploration?
4.1 Experimental Setup
Tasks. As illustrated Figure 3, we evaluate TLDR in state-based environments: Ant and HalfCheetah from OpenAl Gym [30], Humanoid-Run and Souaruped-EEscapp from DeepMind Control Suite (DMC) [31], AntMaze-Large from D4RL [32], and AntMaze-Ultra [33]. For Humanoid-Run and Quadruped-Escapp Escape, we include the 3D coordinates of the agents in their obser- vations. In addition, also evaluate two pixel-based environments: Quadruped (Pixel) from METRA [1] and Kitchen (Pixel) from D4RL [32], with the 64 x 64 3 image observation.

---

(Ours)
METRA
Disagreement
APT
30000
300
6000
State Coverage
Coverage
State Coverage
10000
100
20000
0
3
9
6
Time (hrs)
Time (hrs)
Time (hrs)
(a) Ant
(b) HalfCheetah
(c) Humanoid-Raun
State Coverage
Coverage
State Coverage
300
600
1000
150
300
500
12
Time (hrs)
Time (hrs)
Time (hrs)
(d) Quadruped-Eseeee -sccape
(c) AntMazze-Larre Large
AntMaze-Ultee
Figure 4: State coverage in state-based environments. We measure the state coverage of unsuper- vised exploration methods. Our method consistently shows superior state coverage compared to other methods, except in HalfCheetah compared against METRA.
Comparisons. We compare our method with prior unsupervised GCRL, skill discovery, and exploration methods. For state-based environments, we compare with METRA, PEG, APT, RND, and Disagreement. For pixel-based environments, compare with METRA and LEXA.
METRA [1]: leverages temporal distance-aware representations for skill discovery.
PEG [2]: plans obtain goals with maximum exploration rewards.
LEXA [10]: uses world model to train an Achiever and Explorer policy.
APT [27]: maximizes the entropy reward estimated from the k-nearest neighbors in minibatch.
RND [34]: uses the distillation loss of a network to random target network as rewards.
Disagreement [35]: utilizes the disagreement among ensemble of world models as rewards.
4.2 Quantitative Results
In Figure 4, we compare the state coverage during training (i.e. the number of x 1 sized (z, v)- bins occupied by any of the training trajectories). TLDR outperforms all prior works, except in HalfCheetah compared to METRA. METRA learns low-dimensional skills and focuses on extending the temporal distance along a few directions specified by the skills, providing strong inductive bias for simple locomotion tasks like HalfCheetah. On the other hand, TLDR achieves much larger state coverage in complex environments than METRA, including AntMaze-Large, AntMaze- Ultra, and
TLDR (Ours)
METRA
PEG
60
Distanee 50
Reathed Goals (1)
Reached Goals 01)
Goal Distance (1)
Goal Distarce (1)
20
15
Time (hrs)
Time (hrs)
Time (hrs)
Time (hrs)
Time (hrs)
(a) Ant
(b) HalfCheetah
(c) Humanoid--Rum
AntMaze-Large AntMaze-Ultee
Figure 5: Goal-reaching metrics of a goal-conditioned policy. For (a) Ant, (b) HalfChectah, and (c) Humanoid-Run we report the average distance between goals and the last states of trajectories (lower is better). TLDR achieves comparable average goal distance to METRA. For AntMaze environments, we report the number of pre-defined goals reached by geal-reaching policy (7 for (d) LntMaze-Large and 21 for (e) AntMaze- Ultra), and TLDR significantly outperforms prior works.

---

TLDR (Ours)
METRA
LEXA
Coverga (1)
State 900+
Cool Dosare
Time
(a) Quadruped (Pixel)
(b) Kitchen (Pixel)
Figure 6: Results in pixel-based environments. We compare TLDR with prior works in pixel-based Quadruped and Kitchen environments. In Quadruped (Pixel), TLDR demonstrates a slow learning speed compared to METRA and LEXA. For Kitchen (Pixel), TLDR could interact with all six objects during training but shows low success rates for evaluation.
Quadruped-Eseeae Escape, where all other methods struggle and only explore limited regions. This shows the strength of our method in the exploration of complex environments.
We then compare the goal-reaching performance of TLDR with PEG and METRA in Figure by measuring the average distance between goals and the last states of trajectories. The results in Figures 5a to 5c show that TLDR navigate towards the given goals closer than at least par with METRA. Figures 5d and 5e show that TLDR is the only method that can navigate towards various set of goals in both mazes, demonstrating its superior exploration and goal-conditioned policy learning with temporal distance.
In Appendix B, we further show the comparisons environment steps, in hours. PEG shows better sample efficiency in relatively low-d low-dimensional or easy-exploration tasks, such Ant and HalfCheetah. However, the state coverages of PEG quickly converge narrower regions, especially in AntMazes, than those of TLDR. METRA generally shows worse sample efficiency than TLDR.
Figure shows the results in pixel-based environments. In Quadruped (Pixel), TLDR explores diverse regions but learns slower than LEXA and METRA. For Kitchen (Pixel), TLDR interacts with all six objects during training, but struggles at learning the goal-conditioned policy. Further analysis in Appendix F suggests that the performance bottleneck is related to conditiond policy learning with pixel observations. We leave more detailed analyses for future works.
4.3 Qualitative Results
Figure 7 visualizes the learned goal-reaching behaviors the AntMaze-Ultre environment. TLDR can successfully reach both near and far- away goals in diverse regions. On the other hand, METRA and PEG fail to navigate to di- verse goals. METRA could reach some goals distant from the initial position, whereas PEG fails reach temporally faraway goals. This clearly shows the benefit of using temporal dis- tance in unsupervised GCRL. More qualitative results can be found in Appendix D.
(a) TLDR (ours)
(b) METRA
(C) PEG
Figure TLDR can cover more goals compared to METRA and PEG in AntMaze-Ulta.
4.4 Ablation Studies
To investigate the importance of temporal distance-aware representations in algorithm, conduct ablation studies on GCRL reward designs and exploration strategies.
GCRL reward design. We compare with three different goal-conditioned policy learning methods: (1) QRL [14], which learns a quasimetric value function and latent dynamics model, (2) sparse HER [8], which uses the sparse goal-reaching reward - (s * g), and (3) DDL [19], which uses
7

---

TLDR QRL
HER
DDL
TLDR
Disagreement
State Covenggh
20000
Art
State Coverge
300
ArtMaae Large
State Govergg
30000
Art
Coverge
400
ArtMaze Large
23000
500-
20000
600
20000
300
200
Time (17s)
Time (trs)
Time (hrs)
Time (tes)
(a) TLDR with different GCRL rewards
(b) TLDR with different exploration methods
Figure 8: We evaluate our method with different design choices for (a) GCRL rewards and (b) ex- ploration methods on Ant and AntMaze- TLDR shows better state coverages than its ablated
versions in both ablation studies, indicating the importance of using temporal distance-and aware repre- sentations for both exploration and GCRL.
expected temporal distances as rewards. Figure 8a and Figure 21 show the superior performance of our temporal distance-based GCRL reward over HER and DDL, suggesting the importance of using optimal temporal distance as a dense reward signal. Furthermore, although QRL learns a value function that preserves optimal temporal distances, it struggles to learn an effective goal-reaching policy. Unlike QRL, which directly uses the learned value function along with an additional latent dynamics model, TLDR leverages temporal distance-aware representations to compute dense rewards for the goal-conditioned policy and shows better performance. In Appendix G, show that this trend also holds with fixed dataset, which ignores the effect of exploration and only compares goal-reaching reward designs for goal-reaching performances.
Exploration strategy. For goal selection and exploration rewards, we replace TLDR reward in Equation (2), with other exploration bonuses: APT (with ICM [36] representations), RND, and Disagreement. Note that goal-conditioned policies are still trained with the same temporal distance- based rewards as TLDR, thereby comparing only exploration strategies. As shown in Figure 8b, using TLDR reward for goal selection and exploration rewards achieves significantly higher performance than other exploration bonuses. This result implies that our temporal distance-based rewards are effective for unsupervised exploration.
Conclusion
In this paper, we introduce TLDR, an unsupervised GCRL algorithm that incorporates temporal distance-aware representations. TLDR leverages temporal distance for exploration and learning the goal-reaching policy. By pursuing states with larger temporal distances, TLDR can continuously explore challenging regions, achieving better state coverage. The experimental results demonstrate that TLDR cover significantly larger state spaces across diverse environments than existing unsupervised RL algorithms.
Limitations. While TLDR achieves remarkable state coverages, it still has several limitations:
TLDR shows a slow learning speed compared to METRA in pixel-based environments. Our analysis in Appendix F demonstrates the need for further research in representation learning and GCRL for pixel observations.
Our temporal distance-aware representations do not capture asymmetric temporal distance between states, which can make policy learning challenging for highly asymmetric environments.
Applying unsupervised RI. to real robots has many challenges, including safety. While not tested on real robots, our preliminary results in Appendix E indicate that combining TLDR with safety-aware techniques [37, 38] is a promising future direction for real robotic systems.
TLDR achieves high efficiency in terms of wall clock time, but not in terms of sample efficiency, shown in Appendix B. We believe that increasing the update- to-te--t--ata ratio using model- based RL could enhance the sample efficiency of our method.

---

Acknowledgments
This work was supported in part by the Institute of Information & Communications Technology Planning & Evaluation (ITTP) grant (RS-2020-11201366 Artificial Intelligence Graduate School
Program (Yonsci University)), the National Research Foundation of Korea (NRF) grant (RS-2024- 00333634), and the Electronics and Telecommunications Research Institute (ETRI) grant (24ZR1 100) funded by the Korean Government (MSIT).
References
[1] S. Park, O. Rybkin, and S. Levine. Metra: Scalable unsupervised rl with metric-aware abstrac- tion. In International Conference Learning Representations, 2024.
[2] E. S. Hu, R. Chang, O. Rybkin, and D. Jayaraman. Planning goals for exploration. In International Conference on Learning Representations, 2022.
[3] L. Kaelbling. Learning to achieve goals. In International Joint Conference on Artificial Intelligence, volume 2, pages 1094-8, 1993.
[4] K. Deguchi and I. Takahashi. Image- based simultancous control of robot and target object motions by Intelligent Robots and Systems, pages 375-380, 1999.
method. In JEEE/RSJ International Conference on
[5] Schaul, D. Horgan, Gregor, and D. Silver. Universal value function approximators. In International Conference on Machine Learning, pages 1312-1320, 2015.
[6] M. Watter, J. Springenberg. J. Boedecker, and M. Riedmiller. Embed control: A locally linear latent dynamics model for control from raw images. In Advances in Neural Information Processing Systems, pages 2746-2754, 2015.
[7] C. Finn, X. Y. Tan, Y. Duan, T. Darrell, S. Levine, and P. Abbeel. Deep spatial autoencoders for visuomotor learning. IEEE International Conference on Robotics and Automation, pages 512-519, IEEE, 2016.
[8] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, O. Pieter Abbeel, and W. Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems, volume 30, 2017.
[9] Y. Zhu, R. Mottaghi, E. Kolve, J. Lim, A. Gupta, L. Fei-Fei, and A. Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In IEEE International Conference Robotics and Automation, pages 3357-3364, 2017.
[10] R. Mendonca, O. Rybkin, K. Daniilidis, D. Hafner, and D. Pathak. Discovering and achieving goals via world models. In Neural Information Processing Systems, 2021.
[11] S. Pitis, H. Chan, S. Zhao, B. Stadie, and J. Ba. Maximum entropy gain exploration for long horizon multi-goal reinforcement learning. In International Conference on Machine Learning, pages 7750-7761. PMLR, 2020.
[12] D. Hafner. K.-H. Lee, I. Fischer, and P. Abbeel. Deep hierarchical planning from pixels. In Neural Information Processing Systems, volume 35, pages 26091-26104 2022.
[13] S. Park, Kreiman, and S. Levine. Foundation policies with hilbert representations. In International Conference on Machine Learning, 2024.
[14] Wang, A. Torralba, P. Isola, and A. Zhang. Optimal goal-reaching reinforcement learning via quasimetric learning. In International Conference Machine Learning, pages (6411-364300 PMLR, 2023,

---

[15] V. H. Pong. M. Dalal, S. Lin, A. Nair, S. Bahl, and S. Levine. Skew-Fit: State-covering self-supervised reinforcement learning. International Conference on Machine Learning, 2020.
[16] S. Pitis, H. Chan, S. Zhao, B. C. Stadie, and J. Ba. Maximum entropy gain exploration for long horizon multi-goal reinforcement learning. In International Conference on Machine Learning. 2020.
[17] Y. Lee, S.-H. Sun, S. Somasundaram, E. S. Hu, and J. Lim. Composing complex skills by learning transition policies. In International Conference Learning Representations, 2019. URL https: hollhttss://opoe http::::///peeeieee/ee www. http://wneee//eeteeeeeeeeee weeee//oorriiirrggBeeeeeee
[18] Y. Lee, Szot, S.-H. Sun, and Lim. Generalizable imitation learning from observation via inferring goal proximity. Neural Information Processing Systems, 2021.
[19] K. Hartikainen, X. Geng, T. Haarnoja, and S. Levine. Dynamical distance learning for semi- supervised and unsupervised skill discovery. In International Conference on Learning Repre- sentations, 2020.
[20] A. Ecoffet, J. Huizinga, J. Lehman, K. O. Stanley, and J. Clune. First return, then explore. Nature, 59077847))))888-588 2021.
[21] V. H. Pong, M. Dalal, S. Lin, A. Nair, S. Bahl, and S. Levine Skew-fit: State-covering self-supervised reinforcement learning. In International Conference Machine Learning, 2020.
[22] R. Sekar, O. Rybkin, K. Daniilidis, P. Abbeel, Hafner, and Pathak. Planning explore via self-supervised world models. In International Conference on Machine Learning, 2020.
[23] K. Gregor, D. Rezende, and D. Wierstra. Variational intrinsic control. ArXiv, abs/ abs//611.07507,, 2016.
[24] J. Achiam, H. Edwards, D. Amodei, and P. Abbeel. Variational option discovery algorithms. ArXiv, abs/1807. 10299, 2018.
[25] B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine. Diversity is all you need:
Learning skills
without reward function. In International Conference Learning Representations (ICLR), 2019.
[26] A. Sharma, S. Gu, Levine, V. Kumar, and K. Hausman. Dynamics-aware unsupervised discovery of skills. In International Conference on Learning Representations (ICLR), 2020.
[27] H. Liu and P. Abbeel. Behavior from the void: Unsupervised active pre-training. In Neural Information Processing Systems, 2021.
[28] S. Park, J. Choi, J. Kim, H. Lee, and G. Kim. Lipschite-constrriied unsupervised skill discovery. In International Conference on Learning Representations, 2022.
[29] Park, K. Lee, Y. Lee, and P. Abbeel. Controllability unsupervised skill discovery. In International Conference Machine Learning, pages 27225-27245 PMLR, 2023.
[30] G. Brockman, V. Cheung, L.. Pettersson, J. Schneider, J. Schulman, Tang, and W. Zaremba. OpenAl Gym. ArXiv, abs/1606.01540) 2016.
[31] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. de Las Casas, D. Budden, A. Abdolmaleki,
J. Merel, A. Lefrancq. T. P. Lillicrap, and M. A. Riedmiller. Deepmind control suite. arXiv preprint arXiv: 1801.00690 2018.
[32] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning, arXiv preprint 20X1:22004.07219, 2020.
10

---

[33] Z. Jiang, Т. Zhang, M. Janner, Y. Li, T. Rocktäschel, E. Grefenstette, and Y. Tian. Efficient plan- ning in a compact latent action space. In International Conference on Learning Representations, 2023.
[34] Y. Burda, H. Edwards, A. J. Storkey, and O. Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2019.
[35] D. Pathak, D. Gandhi, and K. Gupta. Self-supervised exploration via disagreement. In International Conference on Machine Learning, 2019.
[36] D. Pathak, P. Agrawal, A. Efros, and T. Darrell. Curiosity-driven exploration by self- supervised prediction. International conference on machine learning, pages 2778-2787.
PMLR, 2017.
[37] K. Srinivasan, B. Eysenbach, S. Ha, J. Tan, and C. Finn. Learning to be safe: Deep rl with safety critic. arXiv preprint arXiv:2010. 14000 2020.
[38] S. Kim, J. Kwon, T. Lee, Y. Park, and J. Perez. Safety-aware unsupervised skill discovery, In IEEE International Conference on Robotics and Automation, pages 894-900. IEEE, 2023.
[39] Haarnoja, Zhou, Abbeel, and Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference Machine Learning, 2018.
[40] M. Laskin, D. Yarats, H. Liu, K. Lee, A. Zhan, K. Lu, C. Cang, L. Pinto, and P. Abbeel. Urlb: Unsupervised reinforcement learning benchmark. Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2021.
[41] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations, 2015.
[42] J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv: 2016.
[43] A. Gupta, Kumar, C. Lynch, Levine, and K. Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. In Conference Robot Learning, 2019.
[44] K. Zakka, Y. Tassa, and MuloCo Menagerie Contributors. MuloCo Menagerie: collection of high-quality simulation models for MuloCo, Sept. 2022.
[45] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, et al. Bootstrap your own latent-a new ap- proach to self-supervised learning. Neural Information Processing Systems, 33:21271-21284, 2020.
[46] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, pages 1597-1607. PMLR, 2020.
[47] D. Hafner, T. P. Lillicrap, J. Ba, and M. Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2020.

---

A Training Details
A.1 Computing Resources and Experiments
All experiments are done on a single RTX 4090 GPU and 4 CPU cores. Each state-based experiment takes 12 hours for all methods, following METRA [1], which trains each method for 10-12 hours. We report the number of environment steps used for the methods in our experiments in Table 1. We use 5 random seeds for all experiments and report the mean and standard deviation of the results.
Table 1: The number of environment steps for experiments.
Environment
TLDR
METRA
PEG
LEXA
APT
RND
Disagreement
Ant HalfCheetah
56.5M
83.2M
0.7M
2.4M
4.1M
4.8M
51.4M
103.5M
0.7M
2.5M
4.2M
5.0M
AntMaze-Large
42.6M
62.5M
0.7M
2.4M
6.4M
5.0M
AntMaze-Ultra
31.2M
44.5M
0.6M
2.4M
4.5M
3.4M
Quadruped-Eseee Escape
28.0M
34.8M
0.6M
2.2M
4.5M
4.4M
Humanoid Run
40.8M
59.9M
0.6M
3.5M
4.7M
4.7M
Quadruped (Pixel)
3.9M
4.1M
2.1M
Kitchen (Pixel)
1.1M
1.7M
1.0M
A.2 Implementation Details
Our method, TLDR, is implemented on top of the official implementation of METRA. Similar to METRA, we use SAC [39] for learning the goal-reaching policy and exploration policy. We train our temporal distance-aware representation o(s) by maximizing the following objective:
Em-peg--a [f([]((s) <(g))) + X min (e, 1 ([ (s)))))),,
(5)
where f is an trine-transformed softplus function:
f(x) softplus(500 x, = 0.01),
(6)
which prevents the distances |] ((g) from diverging, following QRL [14].
For training the exploration policy, normalize the TLDR reward used in Equation (3) to keep the
rewards on a consistent scale. We simply divide the TLDR reward by running estimate of its mean value, following APT [27].
For METRA, PEG, and LEXA, we use their official implementation. For random exploration approaches (APT, RND, Disagreement), we use the implementation from URLB [40].
A.3 Hyperparameters
The hyperparameters used in our experiments are summarized Table
2-D continuous skills for Ant, 16-D discrete skills for HalfCheetah, 24-D discrete skills for Kitchen (Pixel), and 4-D continuous skills for other environments. We use the batch size of 1024 for state-based environments and 256 for pixel-based environments. We set the number of gradient steps per epoch for each experiment to be the same as ours. We use the default values for the remaining hyperparameters. To perform goal-reaching tasks with METRA, we set the skill z as o(g)-c(s) for continuous skills or arg maxdim (ø(g) - d(s)) for discrete skills. [o(x)=((s]
For METRA,
In PEG, we use the same hyperparameters used in their AntMaze experiments. Since PEG uses the normalized goal space, we measure the range of the observations and normalize the goal states according to the minimum and maximum range.
In LEXA, we follow their hyperparameters and opt for the temporal distance reward for training the Achiever policy.
12

---

For APT (with ICM encoder), RND, and Disagreement, we use the same hyperparameters as in URLB [40].
For the ablation with QRL, we use the learning rate of 0.0003 for the critic. We use an (input dim)-1024-1024-122 network for the encoder, 256-1024-2048 for the projector, IQE-maxmean head of 64 components of size 32, and 10--102--102-111 for the latent dynamics model. The transition loss is weighted by 1. For HER, we use the discount factor = 0.99.
Table 2: List of hyperparameters.
Hyperparameter
Value
Learning rate
0.0001
Learning rate for ø
0.0005
Batch size
1024 (State), 256 (Pixel)
Replay buffer
10t (State), 10 (Quadruped (Pixel)), 10² (Kitchen)
Frame stack (Pixel)
Optimizer
Adam [41]
Relaxation constant in Eq. (5)
10-3
dim ((s)
(Kitchen), (Others)
in Eq. (2)
12
Initial À
3 10a
SAC entropy coefficient
0.01 (Kitchen), target entropy as (- A) (others)
Discount factor 7
0.97 (Goal-reaching policy), 0.99 (Exploration policy)
Normalization
LayerNorm [42) for the critics, None for á and actors
Encoder for image observations
CNN
MLP dimensions
1024
MLP depths
Goal relabelling
0.8 (sampled from future observations), 0.2 (no relabelling)
50 (Ant, HalfCheetah, Humanoid-Run, Quadruped-Fscape),
of gradient steps per epoch
75 (AntMaze-Large). 100 (Kitchen (Pixel)), 150 (AntMaze-Ulna))
200 (Quadruped (Pixel))
of episode rollouts per epoch
for updating the target network
0.995
A.4 Environment Details
Ant. We use the MuJoCo Ant environment in OpenAl gym [30]. The observation space is 29-D and the action space is 8-D. Following METRA, normalize the observations for Ant with fixed mean and standard deviation of observations computed from randomly generated trajectories. The episode length is 200.
HalfCheetah We use the MuJoCo HalfCheetah environment OpenAl gym [30]. The observation space 18-D and the action space 6-D. Following METRA, we normalize the observations for HalfCheetah with fixed mean and standard deviation of observations from randomly generated trajectories. The episode length is 200.
We use the Humanoid-Run task from DeepMind Control Suite [31]. The global J, coordinates of the agent are added to the observation. Humanoid has 55-D observation space with 21-D action space. The episode length is 200.
Humanoid-Run.
Quadruped-Eseaa Quadruped-Escapp is included in DeepMind Control Suite [31]. The quadruped robot is initialized a basin surrounded by complex terrains. Due to the complex terrains, moving further away from the initial position is challenging- Similar the AntMaze environments, we fix the terrain shape. Also, we add the global x, y, coordinates of the agent the observation. Quadruped-Escapp Escape has 104-D observation space with 12-D action space. The episode length is 200.
13

---

AntMaze-Large. We use -ataaze-aarge-paayy in D4RL [32]. The observation and action spaces are the same as the Ant environment. The episode length 300. To make exploration more challenging, we fix the initial location of the agent to be the bottom right corner of the maze, as shown in Figure 3 (AntMaze-Large).
AntMaze-Ultra. We use antmaze-ultra-plae and action spaces the same the Ant environment. The episode length is 600. Similar to AntMaze- Large, we fix the initial location of the agent to be the bottom right corner of the maze, shown in Figure (AntMaze-Ulrra.
proposed by Jiang al. [33]. The observation
Quadruped (Pixel). We use the pixel-based version of the Quadruped environment [31] used in METRA [1]. Specifically, we use the image size of 64 64 3 with episode length of 200.
Kitchen (Pixel). We use the pixel-based version of the Kitchen environment [43] used in ME- TRA [1] and LEXA [10]. Specifically, we use the image size of 64 x 64 3 with the episode length of 50. The action space has 9 dimensions.
A.5 Evaluation Protocol
For Ant, Humanoid, and Quadruped (Pixel), we sample goals with (Ir, y)-oorddinnssss from [-50, 50]2, [-40, 40]3, and [-15, 15]2, respectively. For the rest of the goal state (e.g. joint poses), we use the initial robot configuration following Park al. [1].
For HalfCheetah, we sample goals with z-coordinates from [-100, 100].
For AntMaze-Large and AntMaze-Ultre we use the pre-defined goals as shown in Figure A goal deemed be reached when an ant gets closer than 0.5 the goal.
For Kitchen (Pixel), we use the same 6 single-task goal images used in LEXA [10], which consist of interactions with Kettle, Microwave, Light switch, Hinge cabinet, Slide cabinet, and Bottom burner. We report the total number of achieved tasks during evaluation.
For all environments, we use full state as a goal. Specifically, for state-based observations, we use the observation upon reset the base observation and switch the æ, y coordinates (or for HalfCheetah) to the corresponding dimensions. For Quadruped (Pixel), render the image of the state where the agent is at the goal position and use it as the goal.
For each environment, state coverage is calculated by the number of 1-sized (æ, yiins ((r)-bins for HalfCheetah) occupied by any of the training trajectories. For Kitchen (Pixel), the state coverage calculated the number of tasks achieved at least once during the last 100000 environment steps.
B Sample Efficiency Comparison
We compare the sample efficiency of TLDR, METRA and PEG under the same setting in Sec- tion 4.2. Since PEG requires longer training time for the same amount of environment steps compared to TLDR. PEG is trained for more than 72 hours, while TLDR and METRA are each trained for 12 hours.
Figures 9 and 10 illustrate the state coverage and goal-reaching metries with respect the number of environment steps used for training. TLDDR exhibits superior state coverages and goal-reaching performance in hard-exploration tasks like AntMazes. In contrast, PEG tends to be more sample- efficient in environments with relatively low-dimensional state and action spaces, such as Ant and HalfCheetah, but quickly converges to narrower regions in other environments that require hard- exploration (AntMazes) or have higher-dimensional state and action spaces Seaddrped-Escappp Humanoid-Run Run). METRA shows worse sample efficiency overall compared to TLDR.
PEG's s exploration via latent disagreement may over-prioritize less critical dimensions of the state spaces (e.g.. joint angles) and may not scale well to high-dimensional observation spaces. Additionally,
14

---

its reliance on expected temporal distances can be less effective for training a goal-reaching policy than TLDR's s optimal temporal distances, as shown Section 4.4. Moreover, METRA'S skill learning objective can incentivize revisiting known distant states rather than exploring new ones, leading to suboptimal convergence.
Efficient exploration in eighddimessionll spaces remains major challenge in learning complex real-world tasks. Unlike other methods that quickly converge to suboptimal solutions in these settings, TLDR effectively handles this challenge and continues to improve steadily. We believe that increasing the update-to-de to-data ratio or incorporating model-based reinforcement learning approaches could further enhance TLDR's s sample efficiency.
TLDR (Ours)
METRA
PEG
State Coverage
State Coverage
State Coverage
20000
300
6000
100000
2000
100
15
Steps (M)
Steps (M)
Steps (M)
9000
450
6000
300
400
01
0
Steps (M)
Steps (M)
Steps (M)
(a) Ant
(b) HalfCheetah
(c) Humanoid-RR -unnnnndddnunn
State Coverage
State Coverage
Sate Coverage
300
600
1000
130
500
15
Steps (M)
Sceps (M)
Steps (M)
200
300
800
150
0-
Steps (M)
Steps (M)
Steps (M)
(d) Quadruped-Escape
(c) AntMaze-Large
(f) AntMaze-Ultra
Figure State coverage in state-based environments (sample efficiency). We plot the state coverage in terms of the environment steps used for training. PEG is trained for >72 hours for comparison. PEG, as a model-based GCRL algorithm, is more sample efficient for relatively low- dimensional tasks like Ant or HalfCheetah but struggles to learn in more challenging environments such AntMaze. METRA generally less sample efficient compared to TLDR.
15

---

TLDR (Ours)
METRA
PEG
Goal Distance (L)
Distance (L)
Goal Distance (L)
80
60
30
Steps (M)
Steps (M)
Steps (M)
45
40
30
0-
Steps (M)
Steps (M)
Steps (M)
(a) Ant
(b) HalfCheetah
(c) Humanoid-Ra Run
# Reached Goals (1)
Reached Goals (t)
14
Steps (M)
Steps (M)
14-
2-
WW
Steps (M)
Steps (M)
(d) AntMazz-Large
(c) IntMaze-Ultra
Figure 10: Geal-ngg metries of a goal-conditioned policy (sample efficiency). Similar the results in Figure 9, while PEG can be trained efficiently in relatively low-dimensional tasks, TL.DR has better sample complexity in more challenging tasks.
16

---

C More Ablation Studies
We conduct the ablation studies on the number of nearest neighbors & (Figure 11) and dim ((s) (Figure 12) used Equation (2). Figure 11 shows that Ant environment, 12 provides best results, with exploration slightly degrading at k = or 20; in the AntMaze-lLarge environment, the performance is rarely affected by the changes in k. Regarding dim (s)), the performance is nearly the same across different settings. Our main experimental results in Section 4.2 use k = 12 and dim ((s) 4, which demonstrates robust performance across diverse environments.
30000
900
State Coverage
State Coverage
20000
600
10000
0-
3
12
12
Time (hrs)
Time (hrs)
(a) Ant
(b) AntMaze-Large
Figure 11: State coverage on state-based environments with different k. We measure the state coverage of method with l E {5, 12, 20) used for calculating the TLDR reward in Equation (2).
For Ant, & = 12 works the best. For AntMaze-Large does not affect the final state coverage.
d(s) =
dim dimo(s)
dim o(8)
dim (Ks)
30000
900
State Coverage
State Coverage
20000
10000
300
0
3
6
12
12
Time (hrs)
Time (hrs)
(a) Ant
(b) AntMaze-Large
Figure 12: State coverage on state-based environments with different dim ((s). We measure the state coverage of our method with dim o(s) {2, 4, 8, 16), where dim ((s) is the dimension of the temporal distance-aware representations. The results show that dim ((s) does not have critical impact the performance in these environments.
17

---

D More Qualitative Results
We include more qualitative results in Figures 13 to 16. For the qualitative results in Quadruped- Escape (Figure 14), we evenly select 48 states satisfying x² + y 102, where &, y represents the agent position. The coordinate is selected as the minimum possible height that the agent does not collide with the terrain. For all environments, TLDR achieves the best goal-reaching behaviors compared to the other unsupervised GCRL methods, covering the goals in more diverse regions.
(a) TLDR
(b) METRA
(c) PEG
Figure 13: Geal-reaching ability in Humanoid We evaluate each method with the goals sampled according to Appendix A.5. TLDR moves further towards the goal in diverse directions compared to other methods.
(a) TLDR
(b) METRA
(c) PEG
Figure 14: Goal-reaching ability in Quadruped-Escapp We evaluate each method with the goals that are evenly selected at the same distance from the origin. TLDR can not only cover more regions also have better goal-reaching capability than other methods.
18

---

(a) TLDR
(b) METRA
(c) PEG
Figure 15: Goal-reaching ability in AntMaze-Large. TLDR can reach most of the goals in AntMaze-Large, while other GCRL methods struggle to reach distant goals.
(a) TLDR
(b) METRA
(c) PEG
Figure 16: Goal-reaching ability in AntMaze-Ultra, Similar to Figure 15, TLDR can cover the most number of goals in AntMaze-Ultee outperforming other methods.
19

---

E Unitree A1 Simulation Results
While most unsupervised goal-conditioned RL and skill dis- covery research currently focuses on simulated environments, unsupervised RI. holds great potential for learning emergent and efficient skills for real-world robots.
To investigate whether TLDR can explore in environments with real-world robotic counterparts, we train TLDR on the Unitree A1 robot in simulation [44] (Figure 17), consider- ing sim-to-real transfer approaches. TLDR and METRA are trained for hours, with the same hyperparameter settings used in Quadruped-Escape and the episode length of 200. For goal-conditione evaluation, sample goals with (z, y)- coordinates from 15, 15)².
Figure 17: Unitree A1 Simulation Environment. To demonstrate the potential applicability for real-world robots, we choose the environment that simulates a Unitree Al robot with DoFs.
As shown in Figure 18a and Figure 18b, TLDR achieves sub- stantially better state coverage and goal-reaching performance compared to METRA, suggesting the potential of TLDR for au- tonomous exploration and effective goal-reaching gaits learn- ing in real robotics systems.
However, the learned behaviors with TLDR might be unsafe to transfer to reality since does not impose any constraint on the learned behaviors beyond the goal-reaching objective. To address this, we test incorporating a safety reward for learning the exploration and goal-conditioned policies. The safety reward is defined T'aee = [0, 0, 1] Viorso. where Viceso is the orientation of robot torso, which equals to [0, 0, 1] when the robot is upright.
The results in Figure 18 show that TLDR with this safety reward can match the performance of TLDR without regularization in terms of state coverage and goal-reaching metrics while also maximizing the safety reward. These findings indicate that TLDR is compatible with additional reward signals, and applying advanced safety-aware techinques [37, 38] could facilitate the learning of safer behaviors. Videos learned behaviors can be found at https: hamp:://hheee htt///eaatt.../ gi thub. io/tldr.
TLDR
TLDR w/ safety reward
METRA
State Coverage (1)
1500
Goal Detance (U)
45
1.0
Safety Reward
1000
10-
0.0
300
-0.5
-1.
0.0
1.5
3.0
4.5
6.0
0.0
1.5
3.0
4.5
6.0
0.0
1.5
3.0
4.5
6.0
Time (hrs)
Time (hrs)
Time (hrs)
(a) State Coverage
(b) Goal Distance
(c) Safety Reward
Figure 18: Learning curves of Unitree A1 Simulation Results. TLDR achieves better state
coverage (a) and goal-reaching performance (b) compared to METRA. Since the learned behavior can be unsafe, we consider the setting that the safety reward (c) given by raate 00, Vtorso, where Vtorso is the orientation of robot torso which equals to [0, 0, 1] with upright direction. Even with adding the reward, TLDR can (a) still explore the state space and (b) learn effective goal-reaching behaviors (c) while maximizing the safety reward.
20

---

Analysis on Pixel-based Environments
While achieving remarkable exploration and goal-reaching performance in state-based environments, exploration slows down in pixel-based environments, as observed in Figure 6. To identify the perfor- mance bottleneck for learning in pixel-based settings, we compare the performance when replacing pixel observations with state observations for the inputs of goal-conditioned policy, exploration policy, and TLDR encoder, respectively.
Figure 19 shows that the performance of TLDR becomes comparable to METRA when state observa- tions are used instead of pixel observations for the goal-reaching policy, while other modifications do not improve upon the original TLDR. This suggests that the main bottleneck for exploration likely be the representations for the coal-conditioned policy, Based on this result, promising future direction for improving TLDR in pixel-based environments could be the integration of advanced representation learning techniques [45, 46, 47] into our learning pipeline.
TLDR
TLDR GC policy
TLDR exploration
TLDR TLDR encoder
METRA
State Coverage (t)
1200
Goal Distance (J)
900
10-
5-
12
16
0
12
16
Time (hrs)
Time (hrs)
(a) State Coverage
(b) Goal Distance
Figure 19: Result of component-wise analysis in Quadruped (Pixel). To identify the bottleneck of exploration with pixel observations, we swap pixel observations to state observations for the input of goal-conditioned policy, exploration policy, and TLDR encoder, respectively. Exploration of TLDR significantly improves when we input state observations to the goal-conditioned policy, which suggests the main bottleneck for exploration is the representations for the goal-conditioned policy.
Additionally, while METRA quickly learns achieve skills in Kitchen (Pixel) environment compared to TLDR, we observe that its performance degrades with continuous skills, shown in Figure 20. This implies the difficulty of learning policies with continuous goals in TLDR's condiconni policy learning, possibly because learning to reach arbitrary states is more challenging than mastering a specific set of discrete behaviors. Investigating ways to restrict the size of the goal space in TLDR could be an interesting direction for future research.
TLDR (Ours)
METRA: Discrete skills
METRA: Continuous 4.D skills
METRA: Continuous 8.D skills
# Achieved Tasks (t)
12
Time (hrs)
Figure 20: Performance of METRA in Kitchen (Pixel) with different skill settings. When METRA uses continuous skill vectors in Kitchen (Pixel), METRA's performance substantially degrades with continuous skills in Kitchen (Pixel) environment, which is similar to our setting of learning to reach any goals.
21

---

G Analysis on Goal-reaching Reward Design
We compare the goal-reaching performance of TLDR with different coal-conditioned policy learning methods using the same experimental setup as in Figure 8a. As presented in Figure 21, other goal-reaching policy learning methods cannot reach the same level performance as TL.D.. This highlights the importance of our GCRL reward design on goal-reaching performance, consistent with the results in Figure 8a, where TLDR achieves the best state coverage while other methods struggle.
TLDR
QRL
HER
DDL
60
Reached Goals (f)
Goal Distance (4)
40
20
*
3
6
12
3
12
Time (hrs)
Time (hrs)
(a) Ant
(b) AntMaze-Large
Figure 21: Goal-reaching metrics with GCRL reward design ablations. TLDR shows better goal-reaching performance compared to other choices of coal-conditioned policy learning methods, showing the effectiveness of our design of the GCRL reward.
To further isolate the impact of the exploration strategy and focus solely on goal-reaching policy learning, we evaluate the goal-reaching performance an offline learning setting. In this setup, policies are trained fixed dataset of M samples collected from rollouts of trained TLDR policy.
Although goal-reaching performances are degraded in this setting due to off-policy training, our choice of the goal-reaching reward still demonstrates superior results compared to other methods, as shown in Figure 22. This suggests that our design of the goal-reaching reward reward-minimizing the L2 distance to the goal in the temporal distance- representation space perovdds effective signals for goal-reaching.
TLDR
QRL
HER
DDL
Reached Goals (t)
1.5
1.0
0.5
* 0.0
0.0
2.5
7.5
Gradient Steps (100K)
Figure 22: Goal-conditioned policy learning ablation with fixed dataset. With million samples of rollouts from trained TLDR policy, learn the goal-conditioned policy without adding the data to the replay buffer, differing only the goal-conditioned policy learning methods. TLDR shows the best performance in this setting where the impact of exploration isolated.
22


Reviews:
[{"review_id": "oLQ1TgIz3Y", "reviewer": "Reviewer_uLZM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces a novel approach to unsupervised goal-conditioned reinforcement learning (GCRL). The proposed method addresses the challenges of limited exploration and sparse or noisy rewards by leveraging TemporaL Distance-aware Representations (TLDR). TLDR enhances both exploration and goal-reaching capabilities by selecting distant goals and using temporal distances as rewards. This strategy allows the agent to explore a wider range of states and achieve better goal-reaching performance. Experimental results in various simulated robotic environments demonstrate that TLDR significantly outperforms some previous methods in state coverage and efficiency.", "summary_of_recommendation": "Would like the authors to address some of the comments questions in order to be more confident about acceptance.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 2, "quality_of_limitations": 3, "questions_for_rebuttal": "Reference to some prior work is missing - Dynamical Distance Learning for Semi-Supervised and Unsupervised Skill Discovery (Kristian Hartikainen) et al. Discussion about how is this different thatn LEXA[10] would be helpful\nIs there a reason you did not compare with these methods?\nThe explanation of the ablation experiments is can be expanded. Especially, the reasoning behind plot 8b.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rAC7C2U9WI", "reviewer": "Reviewer_7NjC", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The work presents TLDR, which learns temporal distance aware representations to improve exploration and goal-conditioned behaviors.", "summary_of_recommendation": "It's a valid work, with a novel method and interesting results, but it's clearly not a good fit for a robotic conference, given the absence of focus on robotics problems and/or real-world experiments", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 2, "quality_of_limitations": 2, "questions_for_rebuttal": "* Why do the results in the Figures presented use \"time\" as the x-axis? It's more common (and a more sensible choice) to adopt the number of steps in the environment\n* To get better insights into the performance of using TLDR as a representation for goal reaching, it would be useful to show results like the ones presented in Figure 5, but to be obtained by training on the same dataset for all the goal-reaching approaches (removing the dependence of the exploration strategy)\n* It would be useful to have some more insights or ideas on why the approach is failing in pixel-based settings\n* In order to show intent, I believe the work should have addressed problems or techniques relevant for robotics. While GCRL could potentially be applied for robotics, works that have a robotic focus tend to show how to apply such systems in real world in a stable way, e.g. [1,2]. Another way the authors could have shown intent, it's by benchmarking on real robotic systems in simulation, e.g. using the assets from [3]. The environments shown (apart, maybe, for the Franka one) have no real counterparts and are used to test out general approaches for control, not because you would expect the algorithms to transfer to real-world systems. There are other suites that are more appropriate for that purpose, e.g. [4,5].  \n* Exploration in robotics is generally considered unsafe and the work does not address this issue (it doesn't even discuss it as a limitation). Given that a significant part of the contribution comes from better exploration performance, the problem of (un)safe exploration should be addressed or at least discussed\n\n[1] Stabilizing Contrastive RL: Techniques for Robotic Goal Reaching from Offline Data, Zheng et al\n\n[2] From Play to Policy: Conditional Behavior Generation from Uncurated Robot Data, Cui et al\n\n[3] https://github.com/google-deepmind/mujoco_menagerie\n\n[4] https://github.com/google-research/realworldrl_suite\n\n[5] HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulation, Sferrazza et al", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zWtLVv0jNw", "reviewer": "Reviewer_nYH6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Prior unsupervised goal-conditioned reinforcement learning (GCRL) methods struggle to achieve good exploration in complex environments. The authors propose TLDR, an unsupervised GCRL method that uses temporal distance-aware representations to learn exploration and goal-reaching policies. The exploration policy of TLDR tries to explore states far away from states that have already been visited, whereas the goal-reaching policy learns to minimize the distance to the goal. The experiments in simulated robotic environments demonstrate that TLDR achieves excellent state coverage in the benchmark environments.", "summary_of_recommendation": "Getting pure exploration methods to reliably scale to complex environments with high intrinsic dimensionality has been a challenge, and TLDR suffers from the same issue. Given that the method in its current form lacks clear theoretical inspiration, struggles in image-based environments, is not evaluated on real-world robotics, and the goal-reaching performance in state-based environments is good but not great, I cannot recommend accepting this paper in its current form, despite the excellent state coverage achieved in state-based environments. Edit: updated to weak accept after author response.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 2, "quality_of_limitations": 3, "questions_for_rebuttal": "- Eq 4: Are the signs of the terms incorrect? For example, assuming that s' == g, the first component of the reward will be zero, and the reward will be equal to the negative distance between s and g. Note that the code (L1646 in metra.py) has the signs the other way around.\n- More insight into the downstream (goal-reaching) performance would benefit the paper. Ultimately, that is why we care about exploration. For instance, why does the superior state coverage in Humanoid-Run not convert into better downstream performance (Fig. 5)?\n- Fig 5b vs Fig 10b discrepancy. In Fig 5b, TLDR outperforms METRA and PEG on HalfCheetah, whereas in Fig 10b, the methods are approximately equal. Which is correct, or are the plots measuring something different?\n- Did you analyze or have any intuitions as to why the performance in pixel-based environments is lacking?\n- Will you open-source the code if the paper is accepted?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces novel ideas that are shown to be effective on some tasks. There are several concerns: Most importantly for CoRL, its applicability to robotics is unclear. While the methods are potentially relevant to robotics, it is evaluated entirely in simulation. The most robotics-related task is the Kitchen environment but this plays a less-central role in the paper, and TLDR fails to show superior performance. Sim2Real transfer is not discussed. Thus, arguably this work is out of scope for CoRL. Secondly, TLDR performs poorly in pixel-based environments; this would merit further discussion. Thirdly, the methods should be more explicitly contrasted with LEXA and METRA.

During the rebuttal, the authors clarified how TLDR relates to LEXA and METRA. They provided additional experimental results on visual environments. Without solving the issue of lower performance, they do shed some light on the issue.

This paper falls into the *Gray* area of CoRL. It clearly does not address the *Outcome* criterion, and the *Intent* arguably very weakly. However, it does hold promise for eventual application in robotics; thus, in my judgment, it should not be rejected on the grounds of being out of scope.
Decision: Accept