Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shaping
Authors: Lina Mezghani
Sainbayar Sukhbaatar
Piotr Bojanowski
Alessandro Lazaric
Karteek Alahari
Venue: CoRL 2022 Poster (2022)
Keywords: Offline Reinforcement Learning
Self-Supervised Learning
Goal-Conditioned RL

Abstract: Developing agents that can execute multiple skills by learning from pre-collected datasets is an important problem in robotics, where online interaction with the environment is extremely time-consuming. Moreover, manually designing reward functions for every single desired skill is prohibitive. Prior works targeted these challenges by learning goal-conditioned policies from offline datasets without manually specified rewards, through hindsight relabeling. These methods suffer from the issue of sparsity of rewards, and fail at long-horizon tasks. In this work, we propose a novel self-supervised learning phase on the pre-collected dataset to understand the structure and the dynamics of the model, and shape a dense reward function for learning policies offline. We evaluate our method on three continuous control tasks, and show that our model significantly outperforms existing approaches, especially on tasks that involve long-term planning.

Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shaping
Lina Mezghani Meta AI, Inria"
Sainbayar Sukhbaatar
Piotr Bojanowski Meta Al
Meta AI
1 linamezghani@fb.c com
Alessandro Lazaric
Karteek Alahari
Meta Al
Inria
Abstract:
Developing agents that can execute multiple skills by learning from pre-collected datasets is an important problem robotics, where online interaction with the en- vironment is extremely time-consuming. Moreover, manually designing reward functions for every single desired skill is prohibitive. Prior works [1, 2] targeted these challenges by learning goal-conditioned policies from offline datasets with- out manually specified rewards, through hindsight relabeling. These methods suffer from the issue of sparsity of rewards, and fail at long-horizon tasks. In this work, we propose a novel self-supervised learning phase on the pre-collected dataset understand the structure and the dynamics of the model, and shape a dense reward function for learning policies offline. We evaluate our method on three continuous control tasks, and show that model significantly outperforms existing approaches [1, 2]. especially on tasks that involve long-term planning.
Keywords:
Offline RL, Self-Supervised Learning, Goal-Conditioned RL
Introduction
While the goal of realizing general autonomous agents requires mastery of a large and diverse set of skills, achieving this by focusing on each skill individually with standard reinforcement learning (RL) frameworks is prohibitive. This is primarily due to the need for manually designed reward func- tions and environment interactions for each skill. Unsupervised RL has opened way for learning agents that can execute diverse abilities without supervision (i.e., hand-crafted rewards), and then be further adapted to downstream tasks through few-shot or zero-shot generalization [3, 4, 5, 6]. How- ever, learning policies with such methods is impractical with real robots as they require millions of interactions when trained online.
Recently, a line of study has emerged that uses pre-collected datasets of trajectories and trains poli- cies offline (i.e., without additional interactions with the environment) [7, 8]. More precisely, given a dataset of reward- -wward-free trajectories and reward function designed to solve a specific task, the agent learns offline by relabeling the transitions in the dataset with the reward function. This setting is par- ticularly relevant in robotics, where data collection is extremely time-consuming: disentangling data collection and policy learning in this context allows for faster policy iteration. However, it would require designing one specific reward function and learning one policy for each individual task.
An important question to scale offline robot learning is therefore to find ways of learning multi-task policies from already collected datasets. Recent works [1, 9, 10], have targeted this problem from coaliconditioned perspective: given a dataset of previously collected trajectories, the objective is to learn coal-oriented agent that can reach any state the dataset. The advantages of this formulation are two-fold: first, it makes it easy to interpret skills, and second it does not require any adaptation at
Unii,. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK, 38000 Grenoble, France Project page: https: httpe://Ainanezghan.: github. wo/gooo//geeaae
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

test time. Making this framework unsupervised requires to break free from hand-crafted rewards, as proposed by Chebotar al. [1]. where they learn goal-conditioeed policies offline through hindsight relabeling [2]. However, their approach is subject to the pitfall of learning from sparse rewards, and can be inefficient
ong-horizoon tasks.
In this work, we present a self-supervised reward shaping method that enables building an offline dataset with dense rewards. To this end, we develop self-supervised learning phase that aims at learning the structure and dynamics of the environment before training the policy. During this phase, we: (i) train a reachability network [11] to estimate the local distance in the state space S, then (ii) extract a set of representative states that covers S, and finally (iii) build a graph on this set to approximate the global distance in S. When training the goal-conditioned policy, we use the graph in two ways: to compute rewards through shortest path distance, and to create transitions of intermediate difficulty on the path to the goal.
We evaluate our method on complex continuous control tasks, and compare it to previous state- of-the-art offline [1, 2] approaches. We show that our graph-based reward method learns good goal-conditioned policies by leveraging transitions from a dataset of past experience with neither any additional interactions with the environment nor manually-designed rewards. Moreover, we show that, contrary to prior work that uses datasets collected with policy trained with supervised rewards [1], our method allows for learning condiiondi policies even from datasets of poor quality, e.g. containing trajectories sampled with random policy. Our work is thus the first to learn conlicond policies from offline datasets without any supervision, as it does not require any hand-crafted reward function at any stage: data collection, policy training and evaluation.
2 Related Work
Goal-conditioned RL. In its original formulation, goal-conditioned reinforcement learning was tackled by several methods [12, 13, 2. 14]. The policy learning process is supervised in these works: the set of evaluation goals available at train time as well as a reward function that guides the agent to the goal. Several works propose solutions for generating goals automatically when training goal- conditioned policies, including self-play [15, 16, 17], and adversarial student-teacher policies [18]. A recent line of research [19, 20, 21, 22, 23, 24, 26] focuses on learning goal-conditionss policies in an unsupervised fashion. The objective is to train general agents that can reach any goal state in the environment without any supervision (reward, goal-reaching function) at train time. In particular, Mendonca et al. [25] trains a model-based agent that learns to discover novel goals with an explorer model, and reach them with an achiever policy via imagined rollouts.
Offiine RL. The data collection technique an important aspect when studying the training of policies from pre-collected datasets. In this context, the first works assumed access to policies trained with task-specific rewards [27, 28). More recently, methods proposed to leverage unsuper- vised exploration to collect datasets for offline RL [7, 8). In particular, Yarats et al. [7] creates dataset of pre-collected trajectories, ExoRL, on the DeepMind control suite [29] generated without any hand-crafted rewards. Similar to URLB [30], ExoRL benchmarks number of exploration al- gorithms [3, 6, 31, 5), and evaluates the performance of policy trained on the corresponding offline datasets relabeled with task-specific rewards.
Multi-task Offline RL. Recent works proposed to learn multiple tasks from pre-collected datasets, starting with methods [32] that generate goals to improve the offline data collection process in a self- supervised way. This connection has also been studied the supervised setting [9, 33) and when learning hierarchical policies [10]. In a setting closely related to our work, Actionable Models [1] considers the problem of learning goal-conditioned policies from offline datasets without interacting with the environment, and with no task-specifi rewards. They employ with hindsight relabeling [2]. As opposed to their work that relies on learning from sparse rewards, propose to leverage supe-suue training stage to densely shape rewards.
coldccondiioneed Q-learning
3 Preliminaries
Let 8 = (S, A, P. Po. 7, define a reward-free Markov decision process (MDP), where S and A are state and action spaces respectively, P x Ax S - IR_ is state-transition probability

---

add odge
NN..
NM
Figure 1: Overview of the graph building algo- rithm. Given transition (s;, S141) E D, we add 8i as node if it is distant enough from existing nodes in the graph. Moreover, the graph between the incoming nearest neighbor of s; and the outgoing nearest neighbor of $1+1-
D
$io
add edge in
add node
function, Po S IR+ is an initial state distribution, y is the discount factor, and T is the task horizon. In the goal-conditioned setting, the objective is to learn a policy т : S G - A that maximizes the expectation of the cumulative return over the goal distribution, where G denotes the goal space. Here, we make the common assumption that states and goals are defined the same form, i.e., 9 S.
We assume that we have access to dataset D of pre-collected episodes generated by using any data collection algorithm in E. Each episode is stored in D as series of (s, a, 8') tuples, where 8, s E S and a E A. In the general offline formulation introduced by Yarats et al. [7], the dataset D can be relabeled by evaluating any reward function r S x A - IR at each tuple in DD and adding the resulting tuple (s, a, r(s, a), s') in the relabeled dataset D,. We can extend this protocol to the goal- oriented setting by considering goal distribution Pg in the goal space, and any goal-conditioned reward function r S x A G + IR. Given tuple (s, a, s') in D, we relabel it by sampling a goal 9 Pg, computing r(s, 0, 9) and adding the resulting tuple (s, a, 9. r(s, a, g, 8') in the relabeled dataset po.
Once the relabeled dataset Dr.pg is generated, we can learn goal-conditioned policy by executing any offline RL algorithm. The algorithm runs completely offline, by sampling tuples from D Dr.pg and without any interaction with the environment. The goal-conditioned policy is then evaluated online & on set of fixed evaluation goals that not known during training.
Self-supervised Reward Shaping
We now describe our self-supervised reward shaping method. It comprises three stages that we will detail below. In the first stage, we train a Reachability Network (RNet) [11] on the trajectories in D to predict whether two states are reachable from one another. The second stage consists in building directed graph M whose nodes are a subset of states in D. and edges connect reachable states. We employ the RNet as a criterion to avoid adding similar states to M so that its nodes cover the states D uniformly. The final stage consists in training the goal-conditioned policy on transitions and goals sampled from D. It is trained with dense rewards computed as the sum of global (based on the graph distance in M) and local (based on the RNet) distance terms. The important aspect of our method is that the whole training only uses trajectories from the pre-collected dataset 'D without running a single action in the environment. We now describe each component more detail.
4.1 Reachability network
In order to learn a good local distance between states in D, we adopt an asymmetric version of the Reachability Network (RNet) [11]. The general idea of RNet is to approximate the distance between states in the environment by the average number steps it takes for a random policy to go from state to another. We adapted the original formulation with two modifications: first, we use exploration trajectories from 'D instead of random trajectories and second, we leverage the temporal direction because state can be reachable from another without the converse being true. Let (sf,..., x denote a trajectory in D, where a is a trajectory index. We define reachability label aa for each pair of observations (sf, ( by
if a b and S - is Treach:
yi
O otherwise,
for S i.j ST
where the reachability threshold Trench is a hyperparameter. The reachability label equal to 1 iff the states are in the same trajectory and the number of steps from sa to si is below Treach as shown
3

---

R(si, 8))
0.8
0.5
sbortest path
8,
Sj
global distance d
local distance de
(b) Visualisation of the reward computation
(a) Training lahels for RNet
Figure 2: Visualization of dense reward shaping method. (a) shows how training labels generated for training the RNet: given a state 8i, positive pairs are sampled in the same trajectory within a threshold Trench, and the rest of the trajectory forms negative pairs. (b) presents how rewards are implemented as a combination of a global distance term (green), computed with the shortest path in the graph between the outgoing nearest neighbor (NN of the state 8c+1 and the incoming nearest neighbor of the goal (NNi)), and a local distance term (red) computed using the RNet value between NN and g.
in Figure 2a. Note that Vig * Dje- We train a siamese neural network R, the RNet, to predict the reachability label yo from pair of observations (54,5) in D. The RNet consists of an embedding network g. and a fully-connected network f to compare the embeddings, i.e.
R( b))
(2)
where a is a sigmoid function. A higher R value indicates two states reachable easily with random walk, so they can be considered close in the environment. More precisely, R takes values in (0, 1) and s is reachable from if R(ss s') 2 0.5. RNet is learned in a self-supervised fashion, as the ground-truth labels needed to train the network are generated automatically.
4.2 Directed graph
In the next phase, dynamics of the environment, as illustrated in Figure 1. We want the nodes of M to evenly represent the states in 'D. This is achieved by filtering the states in D: state is added M only if it is distant enough from all the other nodes in M. More precisely, a state E DD is added to M if and only if
trajectories in D to build directed graph M that captures high-level
R(s, n) < 0.5 and R(n, s) 0.5, for all n E M.
(3)
Note that we require both the directions to be novel. This filtering avoids redundancy by preventing similar states be added to the memory. also has a balancing effect because it limits the number of states that can be added from a certain area even if it is visited by the agent many times in D.
Once the nodes are selected, we connect pairs that are reachable from one to another. To this end, we employ trajectories in 'D because they contain actual feasible transitions. Given tramsition s; + 8j in we add edge n; + n2 if s; can be reached from node n, and node ny can be reached from sy. This way, we have a chain n. - 8; 8j + n, and can assume ny is reachable from Concretely, we select node n, to be the incoming nearest neighbor (NNj)) to ss, and n, to be the outgoing nearest neighbor (NN..) from 8j. i.e.
74 = NN (s.) argmax R(n, st), n2J NNours,) = argmmax R(s,, n). neM
(4)
nEM
By performing this action over all the transitions in D. turn M into directed graph where edges represent reachability from one node to another.
4.3 Distance function for policy training
We then use the obtained directed graph to compute a global distance in the state space. Indeed, RNet predicts reachability between 8; and 8, so we can directly use it as a distance metric
de(8,, 8,)
R(s,, 83),
VS,, 8, S.
(5)

---

However, this reachability metric is confined to a certain threshold, so there is no guarantee that the RNet predictions will have good global properties.
In contrast, the directed graph M captures high-level global dynamics of the environment. We can easily derive a distance function dM(n,, between any pair of nodes in M by computing the length of the shortest path in this graph, provided the graph is connected. In practice, we can use a trick connect the graph if necessary, by adding an edge between the pair of nodes from different connected components with the maximum RNet value. Moreover, we can extend this distance d.M to a global distance function dg in the state space S by finding, for any pair s; and $j in S their nearest neighbors in the corresponding direction. More precisely,
dg (5t, 0y) = d (NN (*),, (N)((,))), V8,,8, 8, E.S
(6)
The distance d between two states in the state space becomes the length of the shortest path between their respective closest nodes in the graph. This process, summarized in Figure 2b, propagates the good local properties of RNet to get well-shaped distance function for states that are further away. Since dg captures global distances while di captures local fine-grained distance, we use their combination as final distance function: Vs,, 83 E S, 35) = dg(5;, 8j) + di(s;, 8y).
4.4 Policy training
The last phase of our method is training the goal-conditioned policy offline. Here, we create an of- fline replay buffer B that is filled with relabeled data. We randomly sample transition (st, ag st+1) from D as well as a goal g and relabel the transition with reward r = 11++ g). We then push the relabeled transition (st, Oz, 9. Tr, St+1) 1 to B. In order to create curriculum that artificially guides the agent towards the goal, we experimented with two different transition augmentation techniques:
Sub-goal augmentation. Let (st, at, Tt: 8141) denote a relabeled transition and (no, ... 71p-1) the shortest path in the graph M between no (s.) and np-1 NN) The augmen- tation technique consists in adding to the replay buffer every transition (s,, at, n4, ri, st41) for all 1). where rt = d((s+1, n4). other words, given a transition (st, at St+1) )) and goal from D, we push to the replay buffer set of relabeled transitions with all goals on the shortest path from 8 to g (and their corresponding rewards).
i E {0,
Edge augmentation. Similar to the subgoal augmentation technique, we consider relabeled tran- sition (st, ae, g. Ft. St41) and the associated shortest path .p.......--111 This time, we keep the same goal g for every augmented transition, but for every edge (n:-1. E {1, - 1}. we 14-1, (N n, and rf -d(s;, g). Note that the existence of such transition in D guaranteed by construction: an edge is added to the graph from one node to another iff there exist transition in DD whose corresponding nearest neighbors are these two nodes (in the same order).
add the relabeled transition (s/, af. g. g.r, 11+1 to B where (s di, s1+1) E D, oNuu (s))
Once the replay buffer B is filled, the goal-conditioned policy can be trained using any off-policy algorithm. In implementation, hyper-parameter tuning, and is widely used in the literature.
chose Soft Actor-Critic [34], it is known to require few
5 Experiments
5.1 Environments & data collection
We perform experiments on three continuous control tasks with state-based inputs.
UMaze [35]. The first environment, shown in Figure 3a. is two-dimensional U-shaped maze with continuous action space and a fixed initial position. We generate the training data for this environment by deploying a random policy with randomized start position in the maze. We collect 10k trajectories of length 1k. We evaluate the goal-conditioned agent by giving the agent a goal sampled at random in the environment and computing the final euclidean distance to the goal.
Robo Yoga Walker [25]. Introduced by Mendonca et al. [25], the challenging Robo RoboYoga bench- mark is based on the Walker domain of the DeepMind control suite [29], and consists of 12 goals
5

---

Stecoose Ratie
12.35
0.0
Avg Room Koren&konnnneoon
Graph Reward
RNet Reward
(a) UMaze
(b) RNet distance
(c) Graph distance
(d) RNet vs. Graph Rewards
Figure 3: (a) UMaze environment, Heatmap of rewards computed with RNet (b) and graph (c) distances, and (d) Performance of the goal-conditioned policy trained with RNet and graph-based rewards on UMaze. In (b) and (c); high rewards are shown in yellow, and low rewards in black.
that correspond to body poses inspired from yoga (e.g. lying down, raising one leg or balancing). We consider the state-based version of the task, and use the task-agnostic dataset from Yarats et al. [7] generated with unsupervised exploration policy. It contains 10k trajectories of length 1k obtained by deploying the "proto" [5] algorithm in the Walker domain. The success metric of the evaluation policy is assessed by the pose of the humanoid the end of the episode.
Pusher [20]. We also apply our method on Pusher, a realistic robotic environment shown in Fig- ure (left), where a robot arm (red) needs to push a puck (blue) specified location on table. To build the offline dataset, we generated 10k random trajectories of length 200. Similar to prior works [20, 22, 26], we generated 500 goals random in the state space, and we measured the performance as the final Euclidean distance between the puck and its target location.
5.2 Ablation & design choices
We first show that the graph structure necessary for long-term planning. Then, we explain the importance of the directness of the graph on tasks with asymmetrie behaviours. Finally, show the impact of transition augmentation techniques when labeling data for the coal-conditioned policy.
Necessity of graph-based rewards. An important component of method is the construction of the graph M that enables computing a distance with good global properties. To empirically validate this hypothesis, we performed comparison between the goal-conditioned policy trained with RNet rewards (i.e., by using the distance d, from equation (5)) and the one trained with both distance terms as reward. We run this experiment on the UMaze environment, and show results in Figure 3d. We note that the model trained with graph rewards outperforms the one trained with RNet rewards overall, particularly for distant goals (ie. rooms 3 and 4). We also notice that the model trained with RNet rewards is slightly better for goals that are close to the initial position. This highlights the fact that RNet good at estimating local distances. The qualitative visualization in Figure 3b & 3c confirms this observation, as it shows low values between states in the first and fourth rooms.
Importance of graph directness. We then investigate the importance of the asymmetry of the RNet and the directness of the graph. To this end, implement an undirected version of our method where the RNet is symmetric and the graph is undirected. All other components our method are unchanged. First, compare the performance of both variants in the UMaze task in Figure 4a, and note that asymmetric RNet and directed graph in our approach significantly improve the coal-conditioned policy performance (+11% initial location, i.e., goals in rooms and 2. We then analyze qualitative visualizations of the shortest path in the undirected and directed graphs in the Robo Yoga task, shown in Figure 4b. In the undirected case, the humanoid defies the laws of gravity and is encouraged to stand its head by flipping backwards, which might be extremely difficult, or even infeasible. In the directed case, the shortest path fosters the agent to first get back on its legs, and then Iean forward. In this exemple, the gravity makes the dynamics of the environment non-symmetric and non-fully reversible, which justifies the directed formulation described in our method.
success rate), especially goals close to the
Transition sampling strategy.
a final ablation study, we study the utility of the transition aug- mentation techniques described in subsection 4.4. We evaluate four possible variants of our method:

---

0.5
Sacces Rate
0.25
Avg
Rosml
Boan2
Roomd
Roané
Directed
Undirected
(b) Shortest Path visualization for undirected (top) and directed (bottom) graphs
(a) Comparison on the UMaze task
Figure 4: Importance of graph directness (a) the UMaze task and (b) the RoboYoga Walker task.
(i) without any augmentation, (ii) with edge augmentation only, (iii) with subgoal augmentation only, and (iv) with both augmentations. We execute this experiment on the Robo Yoga task, and show results in Figure 6b. We observe that both of the augmentation techniques improve the per- formance of the goal-conditioned agent, with subgoal augmentation showing greater improvement. Moreover, note that combining both augmentations improves the performance further. For the reminder of the experiments, we use both these augmentation techniques.
Average
Room
Room
Room 3
Room 4
0.6
Success Rate
0.4
500 epoch
1000
500 epoch
1000
500 opech
1000
500 epoch
1000
500 epoch
1000
HER
HER
rundom negative action
Actionable Models
Figure 5: Performance on the UMaze task. We show the success rate for goals sampled at random in each of the four rooms, well the average over all rooms.
5.3 Comparison to prior work
Baselines. We compare our method to prior work on unsupervised goal-conditioned policy learning- We perform an apples-to-apple comparison by implementing the baselines using the same learn- ing framework as our method, and changing the reward relabeling process. We compare with the following baselines:
Hindsight Experience Replay [HER] [2] This is re-implementation of the standard un- supervised RL technique, adapted to the offline setting. More precisely, we relabel sub- trajectories from D with sparse reward, which is equal only for the final transition of the sub-trajectory, and everywhere else. Following Chebotar et al. [1], we also label sub-trajectories with goals sampled random in D and zero reward.
HER [2] with random negative action is variant of HER where, for transition in D we sample an action uniformly at random the action space and label with zero reward. This helps overcoming the problem of over-estimation of the Q-values for unseen actions mentioned Chebotar al. [1].
Actionable Models [1] This approach is based on condicoond Q-learing with hind- sight relabeling. We re-implemented the goal relabeling procedure that uses the Q-value at the final state of sub-trajectories in D enable goal chaining, well the negative action sampling trick.
Comparison on UMaze. We compare our method to the baselines on the UMaze task, and show results in Figure We observe that our model outperforms baselines overall, and shows greater improvements challenging goals that are far from the initial position. Interestingly. we note that Actionable Models reaches goals in the first room only. This confirms the intuition that sparse rewards make it difficult for the policy to learn long-horizon tasks.
7

---

0.4
0.4
Ours
Success Rate
Subgoal + Edge
Success
0.2
HER
Subgoal Only
HER + random neg action
0.2
Actionable Models
Edge Only
0.0
No Augmentation
500 epoch
1000
0.0
(a) Comparison to baselines
(b) Impact of Transition Augmentation
Figure 6: Performance on the Robo Yoga Walker task
Comparison on Robo Yoga Walker. In second experiment, we compare our method to baselines on the RoboYoga task, as shown in Figure 6a. Here again, our method outperforms prior work, and Actionable Models does not make any significant improvement over HER. The results broken down by goal are shown in the supplementary material. Overall these results suggest that our dense reward shaping method allows for faster and more robust offline goal-conditioned policy training.
Average Distance
Hand Distance
Puck Distance
K
0.12
0.08
0.04
0.00
500
1000
1000
1000
epoch
spoch
epoch
Ours
HER
HER random neg action
Actionable Models
Figure 7: Performance the Pusher task (lower is better). We report the final average, hand, and puck distance to the goal for our model and all baselines.
Comparison on Pusher. As a final experiment, we compared our method to prior work on a realistic robotic environment, shown in Figure 7. Our policy trained offline is evaluated by sampling a goal at random in the state space, and measuring three different metrics: (i) the hand distance, which corresponds to the final distance between the end of the robot arm and the target, (ii) the puck distance, which measures the distance between the final puck location and the target, and (iii) the average distance, the average of the first metrics. Our method outperforms the baselines this task, and our coal-conditioned agent is able to sequentially place the puck at the goal location, and then place the hand at its target location. On the contrary, HER [2] places the puck at the target location with a performance similar to our method, but lacks precision on the hand location.
6 Conclusion: Summary and Limitations
We proposed method for learning multi-task policies from pre-generated datasets in an offline and unsupervised fashion, i.e., without requiring any additional interaction with the environment, nor manually designed rewards. Our method leverages self-supervised stage that aims learn- ing the dynamics of the environment from the offline dataset, and that allows for shaping dense reward function. It shows significant improvement over prior works based on hindsight relabeling, especially on long- long-horizon tasks, where dense rewards are crucial for learning good policy.
The main limitation of our method is that it relies on the availability of pre-collected dataset of trajectories, with sufficiently large coverage of the state space for proper policy learning. Although such data can be already available, for the Robo Yoga Walker task, that offline dataset collection could be done with random policies, as we did on the UMaze and Pusher tasks, this step can be challenging for other environments. Another limitation is that we evaluated our method exclusively on simulated environments, and we did not perform any experiments on real robots, for which pre- collected dataset with expert demonstrations can be available [1].
8

---

Acknowledgments
Karteek Alahari is supported in part by the ANR grant AVENUE (ANR-18-CE23-0011)
References
[1] Y. Chebotar, K. Hausman, Y. Lu, T. Xiao, D. Kalashnikov, J. Varley, A. Irpan, B. Eysenbach, R. C. Julian, C. Finn, et al. Actionable models: Unsupervised offline reinforcement learning of robotic skills. In International Conference on Machine Learning, pages 1518-1528. PMLR, 2021.
[2] M. Andrychowicz, F. Wolski, A. Ray, Schneider, R. Fong, P. Welinder, B. McGrew, J. To- bin, O. Pieter Abbeel, and W. Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
[3] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell. Curiosity-driven exploration by self- supervised prediction. In International conference on machine learning, pages 2778-2787. PMLR, 2017.
[4] Y. Burda, H. Edwards, A. Storkey, and O. Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2018.
[5] D. Yarats, R. Fergus, A. Lazaric, and L. Pinto. Reinforcement learning with prototypical rep- resentations. In International Conference on Machine Learning, pages 11920-1 1931. PMLR, 2021.
[6] B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine. Diversity is all you need: Learning skills without reward function. In International Conference on Learning Representations, 2018.
[7] D. Yarats, D. Brandfonbrener, H. Liu, M. Laskin, P. Abbeel, A. Lazaric, and L. Pinto. Don't t change the algorithm, change the data: Exploratory data for offline reinforcement learning. In ICLR 2022 Workshop on Generalizable Policy Learning in Physical World, 2022.
[8] N. Lambert, M. Wulfmeier, W. Whitney, A. Byravan, M. Bloesch, V. Dasagi, T. Hertweck, and M. Riedmiller. The challenges of exploration for offline reinforcement learning. arXiv preprint arXiv:2201.1 //61, 2022.
[9] R. Yang. Y. Lu, W. Li, H. Sun, M. Fang, Y. Du, X. Li, L. Han, and C. Zhang. Rethink- ing goal-conditioned supervised learning and its connection to offline rl. arXiv preprint arXiv:2202.00 04478, 2022.
[10] J. Li, C. Tang, M. Tomizuka, and W. Zhan. Hierarchical planning through goal-conditioned offline reinforcement learning. arXiv preprint arXiv:2205.1/770 2022.
[11] N. Savinov, A. Raichuk, D. Vincent, R. Marinier, M. Pollefeys, T. Lillicrap, and S. Gelly. Episodic curiosity through reachability, In International Conference on Learning Representa- fions, 2018.
[12] L. P. Kaelbling. Learning to achieve goals. In IJCAI, volume 2, pages 1094-8. Citeseer, 1993.
[13] T. Schaul, D. Horgan, K. Gregor, and D. Silver. Universal value function approximators. In International conference on machine learning, pages 1312-1320. PMLR, 2015.
[14] S. Nasiriany, V. Pong, S. Lin, and S. Levine. Planning with goal-conditioned policies. Ad- vances in Neural Information Processing Systems, 32, 2019.
[15] S. Sukhbaatar, Z. Lin, I. Kostrikov, G. Synnaeve, A. Szlam, and R. Fergus. Intrinsic motivation and automatic curricula via asymmetric self-play. In International Conference on Learning Representations, 2018.
[16] Sukhbaatar, Denton, Szlam, and R. Fergus. Learning goal embeddings via self-play for hierarchical reinforcement learning. arXiv preprint arXiv: 11///110000 1,09083, 2018.

---

[17] O. OpenAL, M. Plappert. R. Sampedro, T. Xu, I. Akkaya, V. Kosaraju, P. Welinder, R. D'Sa, A. Petron, H. P. d. O. Pinto, al. Asymmetric self-play for automatic goal discovery in robotic manipulation. arXiv preprint arXiv:2101.08880 2021.
[18] A. Campero, R. Raileanu, H. Kuttler, J. B. Tenenbaum, T. Rocktäschel, and E. Grefenstette. Learning with amigo: Adversarially motivated intrinsic goals. In International Conference on Learning Representations, 2020.
[19] D. Fardee-Farley, T. Van de Wiele, Kulkarni, C. Ionescu, S. Hansen, and V. Mnih. Unsuper- vised control through non parametric discriminative rewards. In International Conference Learning Representations, 2018.
[20] A. V. Nair, V. Pong, M. Dalal, S. Bahl, S. Lin, and S. Levine. Visual reinforcement learning with imagined goals. Advances in Neural Information Processing Systems, 31:9191-9200, 2018.
[21] A. Ecoffet, Huizinga, J. Lehman, K. O. Stanley, and J. Clune. Go-explore: new approach for hard-exploration problems. arXiv preprint arXiv: 19X.:v99011.0995.. 2019.
[22] V. Pong, M. Dalal, S. Lin, A. Nair, S. Bahl, and S. Levine. Skew-fit: State-covering self- supervised reinforcement learning. In International Conference on Machine Learning, pages 7783-7792. PMLR, 2020.
[23] S. Yenkattaramanujam, E. Crawford, T. Doan, and D. Precup. Self-supervised learning of dis- tance functions for goal-condin condilood reinforcement learning. arXiv preprint 2019.
02998,
[24] K. Hartikainen, X. Geng, T. Haarnoja, and S. Levine. Dynamical distance learning for semi- supervised and unsupervised skill discovery. In International Conference on Learning Repre- sentations, 2019.
[25] R. Mendonca, O. Rybkin, K. Daniilidis, D. Hafner, and D. Pathak. Discovering and achieving goals via world models. Advances in Neural Information Processing Systems, 34, 2021.
[26] L. Mezghani, P. Bojanowski, K. Alahari, and S. Sukhbaatar. Walk the random walk: Learning to discover and reach goals without supervision. In ICLR Workshop on Agent Learning in Open-Endedness, 2022.
[27] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072111 2020.
[28] C. Gulcehre, Z. Wang, A. Novikov, T. Paine, S. Gómez, K. Zolna, R. Agarwal, S. Merel, D. J. Mankowitz, C. Paduraru, et al. RI unplugged: A suite of benchmarks for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:7248-72535 2020.
[29] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. d. L. Casas, D. Budden, A. Abdolmaleki,
J. Merel, A. Lefrancq, et al. Deepmind control suite. arXiv preprint 1rXiv:/801.00660 2018.
[30] M. Laskin, D. Yarats, Liu, Lee, A. Zhan, K. Lu, C. Cang, L., Pinto, and Abbeel. Urlb: Unsupervised reinforcement learning benchmark. arXiv preprint arXiv:2/ 15/9/, 2021.
[31] D. Pathak, D. Gandhi, and A. Gupta. Self-supervised exploration via disagreement. In Inter- national conference on machine learning, pages 5062-5071, PMLR, 2019.
[32] S. Endrawis, G. Leibovich, G. Jacob, G. Novik, and A. Tamar. Efficient self-supervised data collection for offline robot learning. 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 4650-4656. IEEE, 2021.
[33] Y. J. Ma. J. Yan, D. Jayaraman, and O. Bastani. How far i'll go: Offline goal-conditioned reinforcement learning via aaa regression. arXiv preprint arXiv:2206.030023 2022.
[34] T. Haarooaa, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[35] Y. Kanagawa. mujoco-maze. https: /gtgggthuuuu.... htwptthiw.coo//ggggg///moooomoaa 2020.
10


Reviews:
[{"review_id": "rymEA5jyks7", "reviewer": "Reviewer_iP9J", "summary": "The paper addresses the problem of learning long-horizon tasks from offline (pre-collected) datasets of behaviors. The paper proposes to learn the structure of the task by learning a global distance in the state space of the MDP by building a graph prior to the offline training. Similar to prior works, using hindsight experience replay, the method constructs a dense reward signal to be used for goal reaching tasks with sparse rewards. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \n* The paper proposes a novel method that can solve long-horizon RL tasks provided offline datasets of state-action transitions. \n\nWeaknesses:\n* The paper does not compare to most recent self-supervised offline RL algorithms such as \"RETHINKING GOAL-CONDITIONED SUPERVISED LEARNING AND ITS CONNECTION TO OFFLINE RL\" (reference [3]). \n* The experimental results are limited to only two tasks. \n* The method does not seem to scale to vision-based policy training tasks since the graph cannot be constructed over image space. \n* Please also see the issues. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper introduces a novel method to solve challenging long-horizon tasks in offline RL. The experimental results and the ablations show benefits of the method on two long-horizon offline RL tasks. However, one important baseline is missing and, since the experiments are done in simulation, one can expect results on more than two tasks. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Line 164-165: \"we can use a trick to connect the graph if necessary by adding ...\": This is very unclear to understand what the problem is and what the solution is. \n\nLine 186 (Edge augmentation): It is  unclear where the action a_t^i comes from. A new transition (augmented one) is found by finding s being an incoming state to the node n_{i-1}, and s' being an outgoing state for the node n_i. Therefore, to connect s to s' we need more than one action, most likely a sequence of actions. \n\nLine 229 (Importance of graph directness): It is obvious that we need asymmetric graph for the given task. A symmetric graph would work for tasks like 2D maze in which actions are reversible. Considering this, I don't understand the purpose of the ablation other than answering a very obvious question.  "}}, {"review_id": "eIqWik2awmWs", "reviewer": "Reviewer_nCHc", "summary": "This paper is concerned with unsupervised goal-conditioned policy learning. Manually designing reward functions for every single desired skill is very expensive. The authors propose a novel self-supervised learning phase on the pre-collected dataset to understand the structure and the dynamics of the model, and shape a dense reward function for learning policies offline. The proposed algorithm is benchmarked on a gridworld environment UMaze and RoboYoga walker environment from DMControl. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n\n* This paper is well-written and easy to follow. The motivation of goal-conditioned offline policy learning is well explained. \n* To my knowledge, the proposed method of combining Reachability Network (RNet) and a graph-based distance (reward) metric is novel. The method and details are also well-explained. \n\n**Weaknesses**:\n\nThe proposed algorithm is benchmarked on 2 environments, UMaze (a toy gridworld) and RoboYoga (DMControl, another artificially constructed environment). While the experimental results are informative, neither environments have practical value or concrete applications beyond a simple demonstration. For example, the Walker bot in RoboYoga does not have a real counterpart and cannot be realized on available hardware. In prior works such as LEXA [1], empirical results are shown on simulated robotics task like RoboBins and RoboKitchen that involve realistic robot grippers. Since this is a robot learning conference, it is appropriate to show more meaningful and useful robotic tasks. \n\nFurthermore, limitations are not well discussed in Sec. 6. The current statement is insufficient. \n\n[1] Discovering and Achieving Goals via World Models. Mendonca et al. 2021. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I am willing to consider raising score if the proposed method can achieve convincing results on simulated robotic tasks that are more meaningful and practically useful. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "My main issue is with the toy experiment settings. Please see \"Weakness\" section. \n\nIn addition, the limitation section is not really there in Sec. 6, requires a lot more expansion. ", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "QdFtOdYjFcL", "reviewer": "Reviewer_4myo", "summary": "The paper proposes a novel method to learn from offline datasets. The paper proposes to first learn an RNet, and then distill the RNet predictions onto a DAG to obtain self supervised rewards from the datasets in a self-supervised manner, which does not require reward specification on any sort, at any stage of the pipeline. The paper evaluated the methods on two continous control tasks and improves upon prior self-supervised reward learning methods like HER. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The employment of both Rnet and the graph distance seems like a novel and exciting approach. \n- The subgoal and edge augmentation are very interesting and practical solutions during policy training, since the\n- Results on YogaWalker are good.\n- The ablation is thorough and discusses all different variants of their proposed algorithm.\n- Fig 1 is extremely well done. \n\nWeaknesses:\n- The way the paper is presented, it seems to be mostly for discrete state spaces to construct the graph. But the results shown on YogaWalker is obviously a cts state space. I am confused how the paper constructed the graph for a cts space.\n- I would ideally like to see more experiments with cts state space tasks.\n- I would like to see the paper explore real robotics tasks, if possible. \n- RNet predicted distance in fig 3 a seem very wrong. Rnet is seemingly overfitting to the xy position distance not the actual map of how \"reachable\" the far ends of the maps are. I am not convinced the RNet is learning anything meaningfully correct, and the graph is doing the heavy lifting. To me it seems like R is not learning anything particularly meaningful, and the distillation to the DAG is somewhat lucky due to the hill-climbing way of learning.\n- Using DAGs are a sensible choice, but the paper doesn't empirically show it in a convincing manner. \n- \"we leverage the temporal direction because a state can be reachable from another without the converse being true\", i would be interested in a short analysis of if the fwd-bwd models are consistent, and some intuition about if it makes sense for them to be consistent. For most robotic tasks, I can imagine the A->B and B->A is equally reachable, but I am not sure if that holds for some of the very specific tasks considered. \n- Intuitively, the method would perform well on similar tasks as the original HER paper. I would like to see some results on manipulation tasks.\n- NIT: typically ablation should follow the main experiments / comparisons with the SOTA\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper presents a novel, and very interesting approach to learn from offline datasets, using a self-supervised approach, which employs learning a graph to measure the reachability between two states. Overall, the method is extremely interesting, but the experimental evaluation is not sufficiently thorough. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "Please see weaknesses above", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}, {"review_id": "0FnXqWXU91t", "reviewer": "Reviewer_4D4F", "summary": "This paper proposes a method for learning goal-conditioned policies from reward-free offline datasets. To this end, the method first learns a reachability network, constructs a graph over a state space, and re-label and augment the dataset using the reward defined as a sum of distance measured using the graph and the reachability network. Then the off-policy goal-conditioned RL agent is trained using the dataset. The method is evaluated on UMaze and RoboYoga environments.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Method is a bit complicated but makes sense and shown to be effective on various tasks\n- Experiments are well conducted to show the benefit of the proposed techniques with ablations\n\nWeaknesses\n- Experiments are conducted on relatively simple tasks -- Performance on more complex tasks, especially manipulation tasks, could further support the claim of the proposed method for learning distance in state space. As a baseline (i.e., Actionable Models) shows its effectiveness on various robotic experiments, I would expect this could also be the case for the proposed method, but explcitly showing this would be required for making the paper strong and more relevant to robotics.\n- In terms of presentation, Figure 1 is helpful for understanding high-level concept but more figures for detailed methods would help understanding in detail, especially for augmentations, because notations are a bit complex. For notation, introducing superscripts as a notation for trajectory index and also introducing subscript differently for augmentations makes it a bit difficult to understand what's going on.\n- I would not say that this is necessary; but it would be interesting to see how the proposed method performs with high-dimensional observations (pixels) by training a reachability network on pixel observations. At least including the discussion on this extension in the paper could be interesting and helpful for future researches.\n\nMinor\n- It's awkward to say to call a robot used in DMC Walker as 'humanoid'\n- Figure 3: information about what does each color mean is missing", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I would recommend this paper to be accepted because the method is intuitive and results are strong. But it's difficult to argue strong acceptance because there is a room for improvement, e.g., including additional robotic experiments and improving presentation.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations section not present", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "- Additional robotic experiments, especially manipulation experiments that would enable direct comparison with the baseline\n- Improved presentation\n- Experiments with high-dimensional observations", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)