Dynamical Distance Learning for Semi-Supervised and Unsupervised Skill Discovery
Authors: Kristian Hartikainen
Xinyang Geng
Tuomas Haarnoja
Sergey Levine
Venue:  (2020)
Keywords: reinforcement learning
semi-supervised learning
unsupervised learning
robotics
deep learning

Abstract: Reinforcement learning requires manual specification of a reward function to learn a task. While in principle this reward function only needs to specify the task goal, in practice reinforcement learning can be very time-consuming or even infeasible unless the reward function is shaped so as to provide a smooth gradient towards a successful outcome. This shaping is difficult to specify by hand, particularly when the task is learned from raw observations, such as images. In this paper, we study how we can automatically learn dynamical distances: a measure of the expected number of time steps to reach a given goal state from any other state. These dynamical distances can be used to provide well-shaped reward functions for reaching new goals, making it possible to learn complex tasks efficiently. We show that dynamical distances can be used in a semi-supervised regime, where unsupervised interaction with the environment is used to learn the dynamical distances, while a small amount of preference supervision is used to determine the task goal, without any manually engineered reward function or goal examples. We evaluate our method both on a real-world robot and in simulation. We show that our method can learn to turn a valve with a real-world 9-DoF hand, using raw image observations and just ten preference labels, without any other supervision. Videos of the learned skills can be found on the project website: https://sites.google.com/view/dynamical-distance-learning

Published as a conference paper at ICLR 2020
DYNAMICAL DISTANCE LEARNING FOR SEMI-SUPERVISED AND UNSUPERVISED SKILL DISCOVERY
Kristian Hartikainen"
Xinyang Geng University of California, Berkeley
University of California, Berkeley University of Oxford
Tuomas Haarnoja
Sergey Levine
University of California, Berkeley Google DeepMind
University of California, Berkeley
ABSTRACT
Reinforcement learning requires manual specification a reward function learn a task. While in principle this reward function only needs to specify the task goal, in practice reinforcement learning can be very time-consuming or even in- feasible unless the reward function is shaped so towards a successful outcome. This shaping is difficult to specify by hand, par- ticularly when the task is learned from raw observations, such as images. In this paper, we study how we can automatically learn dynamical distances: a mea- sure of the expected number of time steps to reach given goal state from any other state. These dynamical distances can be used to provide well-shaped re- ward functions for reaching new goals, making it possible to learn complex tasks efficiently. We show that dynamical distances can be used in a semi-supervised regime, where unsupervised interaction with the environment used learn the dynamical distances, while small amount of preference supervision is used to determine the task goal, without any manually engineered reward function or goal examples. We evaluate our method both real-world robot and in simulation. We show that our method can learn to turn a valve with a real-world 9-DoF hand, using image observations and just ten preference labels, without any other supervision. Videos of the learned skills can be found on the project website: https://steesooogge..co//vvvyyy/ hatpr/aammmmmiiisttteeeeeeaiiiia -isttacc--ccaminng
provide smooth gradient
INTRODUCTION
The manual design of reward functions represents major barrier to the adoption of reinforce- ment learning (RL). particularly robotics, where bissed-based policies can be learned end-to- end (Levine et al. 2016 Haarnoja et al.] 2018c), but still require reward functions that themselves might need visual detectors to designed by hand (Singh et al.) 2019). While in principle the reward only needs to specify the goal of the task, in practice RL can be exceptionally time-consuming or even infeasible unless the reward function is shaped so as to provide smooth gradient towards a suc- cessful outcome. Prior work tackles such situations with dedicated exploration methods (Houthooft ctt al.] 2016)) Osband et al. Andrychwwwi et al. 2017). or by using large amounts of random exploration (Mnih et al. 2015), which is feasible in simulation but infeasible for real-world robotic learning. It also common to employ heuristic shaping, such as the Cartesian distance to a goal for object relocation task (Mahmood et al. 2018 Haarnoja al.) 2018a). However, this kind of shaping is brittle and requires manual insight, and is often impossible when ground truth state observations unavailable, such when learning from image observations.
"Correspondence to Kristian.hartikaainn
1 Equal advising.

---

Published as a conference paper at ICLR 2020
We present dynamical distance learning (DDL) method that can learn 9-DoF real-world dexterous manipulation task directly from raw image observations. DDL does not assume access to the true reward function and solves the 180 degree valve-rotation task in 8 hours by relying only on 10 human-provided preference labels.
Figure 1:
In this paper, we aim to address these challenges by introducing dynamical distance learning (DDL), general method for learning distance functions that can provide effective shaping for goal-reaching tasks without manual engineering. Instead of imposing heuristic metrics that have no relationship to the system dynamics, we quantify the distance between two states in terms of the number of time steps needed to transition between them. This is a natural choice for dynamical systems, and prior works have explored learning such distances in simple and low-dimensional domains (Kaelbling) 1993). While such distances can be learned using standard model-free reinforcement learning al- gorithms, such as Q-learning, we show that such methods generally struggle to acquire meaningful distances for more complex systems, particularly with high-dimensional observations such as im- ages. We present simple method that employs supervised regression to fit dynamical distances, and then uses these distances to provide reward shaping, guide exploration, and discover distinct skills.
The most direct use of DDL is to provide reward shaping for a standard deep RL algorithm, to optimize policy to reach a given goal state. We can also formulate a semi-supervised skill learn- ing method, where a user expresses preferences over goals, and the agent autonomously collects experience to learn dynamical distances in self-supervised way. Finally, we can use DDL in a fully unsupervised method, where the most distant states are selected for exploration, resulting in an unsupervised reinforcement learning procedure that discovers difficult skills that reach dynamically distant states from given start state. All of these applications avoid the need for manually designed reward functions, demonstrations, or user- user-provided examples, and involve minimal modification to existing deep RL algorithms.
DDL is a simple and scalable approach to learning dynamical distances that can readily accommo- date raw image inputs and, as shown in experiments, substantially outperforms prior methods that learn goal-conditioned policies or distances using approximate dynamic programming tech- niques, such as Q-learning. We show that using dynamical distances reward function in stan- dard reiemortement learning methods results in policies that take the shortest path to a given goal, despite the additional shaping. Empirically, to prior techniques for learning from preferences. We also compare our method to prior methods for unsupervised skill discovery on tasks ranging from 2D navigation to quadrupedal locomotion. Our experimental evaluation demonstrates that DDL can learn complex locomotion skills without any supervision at all, and that the preferences-baste version of DDL can learn turn valve with a real- real-world 9-DoF hand, using raw image observations and 10 human-provided preference labels, without any other supervision.
compare the semi-supervised variant of our method
2 RELATED WORK
Dynamical distance learning is most closely related to methods that learn goal-conditioned policies or value functions (Schaul et al. 2013 Sutton et al. (01T). Many of these works learn goal-reaching directly via model-free RL, often by using temporal difference updates to learn the distance function as a value function (Kaelbling et al. 1996. Schaul et al.] 2015 Andrychowicz et al. 2017] Pong cet al. 20188 Nair et al., 2018 Florensa et al. 2019). For example, Kaelbling (1993) learns a goal conditioned Q-function to represent the shortest path between any two states, and Andrychowicz et al. (2017) learns a value function that resembles a distance to goals, under a user-specified low- dimensional goal representation. Unlike these methods, DDL leams collcy-conditioned distances with an explicit supervised learning procedure, and then employs these distances to recover a reward function for RI.. We experimentally compare to RL--aased distance learning methods, and show that
2

---

Published as a conference paper at ICLR 2020
DDL attains substantially better results, especially with complex observations. Another line of prior work uses a learned distance to build a search graph over a set of visited states (Savinov et al. 2018 Eysenbach al., 2019), which can then be used to plan to reach new states via the shortest path. Our method also learns a distance function separately from the policy, but instead of using it to build graph, use it to obtain reward function for separate model-free RI. algorithm.
The semi-supervised variant of DDL is guided by a small number of preference queries. Prior work has explored several ways to elicit goals from users, such using outcome examples and a small number of label queries (Singh et al. 2019), or using a large number of relatively cheap preferences (Christiano al. 2017). The preference queries that our semi superviie method uses are easy to obtain and, in contrast to prior work (Christiano et al. (0177). ) we only need small number of these queries to learn policy that reliably achieves the user's desired goal. Our method is also well suited for fully unsupervised learning, in which case DDL uses the distance function to propose goals for unsupervised skill discovery. Prior work on unsupervised reinforcement learning has proposed choosing goals based on variety of unsupervised criteria, typically with the aim of attaining broad state coverage (Nair et al. 2018 Florensa et al.) 2018: Eysenbach et al. 2018 Warde-Farley al. 2018; Pong al.] 2019). Our method instead repeatedly chooses the most distant state as the goal, which produces rapid exploration and quickly discovers relatively complex skills. We provide comparative evaluation in our experiments.
PRELIMINARIES
In this work, we study control of systems defined by fully observed Markovian dynamics p(s"|s. as.a:: x S A - R2o, where S and A are continuous state and action spaces. We aim to learn stochastic policy rraas)) A x S Rzo. to reach a goal state g E S. We will denote trajectory with 7 _ (So, ag .... ST) ST) and So is sampled from an initial state distribution p(So). The policy can be optimized using any reinforcement learning algorithm by maximizing
Px: where Prr is the trajectory distribution induced by the policy n,
L(T) Eroep
Σ (T) 1=0
where rgg: S x A 1-Rming Rmax] is a bounded reward function and y [0, 1) is discount factor" However, we do not assume that we have access to shaped reward function. In principle, we could set the reward to rg (s, = 0 if s = and Tgg (s, a) = -1 otherwise to learn a policy to reach the goal in as few time steps as possible. Unfortunately, such sparse reward signal is extremely hard to optimize, as it does not provide any gradient towards the optimal solution until the goal is actually reached. Instead, in ISSection 4| will show that can efficiently learn reach goals by making use of a learned dynamical distance function.
4 DYNAMICAL DISTANCE LEARNING
The aim of our method to learn policies that reach goal states. These goal states can selected either in unsupervised fashion, to discover complex skills, selected manually by the user. The learning process alternates between two steps: in the distance evaluation step, we learn a policy- specific dynamical distance, which is defined in the following subsection. In the policy improvement step, the policy is optimized to reach the desired goal by using the distance function as the negative reward. This process will lead sequence of policies and dynamical distance functions that converge to an effective goal-reaching policy. Under certain assumptions, we can prove that this process converges to a policy that minimizes the distance from any state to any goal, as discussed in Appendix B In this section, we define dynamical distances and describe our dynamical distance learning (DDL) procedure. In Section [ we will describe the different ways that the goals can be chosen to instantiate our method as semi-supervised or unsupervised skill learning procedure.
In practice, we use soft actor-critic to lear the policy, which uses related maximum entropy objec- tive (Haarnoja al. 2018c).
3

---

Published as a conference paper at ICLR 2020
4.1 DYNAMICAL DISTANCE FUNCTIONS
The dynamical distance associated with a policy 9, which we write as (si, 8y), is defined as the expected number of time steps it took for π to reach a state sj from state S;, given that the two were visited in the same episode Mathematically, the distance is defined as:
d= (s, s') A Eywals 8,8, f21
Σ
cess,, S4+1)
(2)
t=
where 7 is sampled from the conditional distribution of trajectories that passes through first s and then s', and where is some local cost of moving from 8; to 8,+1- For example, in typical case in the absence of supervision, we can set c(s,, St+1) = 1 analogously to the binary reward function in Equation I] in which case the sum reduces to jo- i, and we recover the expected number of time steps to reach s'. principle, we could also trivially incorporate more complex local costs c, for example to include action costs. This modification would be straightforward, though we focus on the simple S(+1) = 1 in our derivation and experiments. We include the discount factor to extend the definition to infinitely long trajectories, but in practice we set y 1.
4.2 DISTANCE EVALUATION
In the distance evaluation step, we learn a distance function di (s,s', parameterized by th, to esti- mate the dynamical distance between pairs of states visited by a given policy Tg. parameterized by e We first roll out the policy multiple times to sample trajectories тą of length T. The empirical distance between states s;, Sj E Tke where 0 S S j I T, is given by j --i. Because the trajectories have a finite length, we are effectively ignoring the cases where reaching s, from s; would take more than T' - i steps, biasing this estimate toward zero, but since the bias becomes smaller for shorter distances, we did not find this to be a major limitation in practice. We can now learn the distance function via supervised regression by minimizing
1 E r-pp,
Ca(v)
(3)
2
in 00.7
[i.T]
As we will show in our experimental evaluation, this supervised regression approach makes fea- sible to learn dynamical distances for complex tasks with raw image observations, something that has proven exceptionally challenging for methods that learn distances via goal-conditioned policies or value functions and rely temporal difference-style methods. In direct comparisons, find that such methods generally struggle to learn on the more complex tasks with image observations. On the other hand, disadvantage of supervised regression is that requires on-policy experience, potentially leading to poor sample efficiency. However, because we use the distance as an interme- diate representation that guides off-policy policy learning. as we will discuss in /Section 4.3) we did not find the on-policy updates for the distance to slow down learning. Indeed, our experiments in Section 6. show that we can learn a manipulation task on real robot with roughly the same amount of experience as is necessary when using well-shaped and hand-tuned reward function.
4.3 POLICY IMPROVEMENT
In the policy improvement step, we use d to optimize a policy Te. parameterized by d. to reach a goal g. In principle, we could optimize the policy by choosing actions that greedily minimize the distance to the goal, which essentially treats negative distances as the values of value function, and would be equivalent to the policy improvement step in standard policy iteration. However, acting greedily with respect to the dynamical distance defined in Equation 2) would result in policy that is optimistic with respect to the dynamics.
This is because the dynamical distance is defined the expected number of time steps conditioned on the policy successfully reaching the second state from the first state, and therefore does not account for the where the sec- ond state is not reached successfully. In some cases, this results in pathologi- cally bad value functions. For example, consider the MDP shown on the right,
g
Dynamical distances are true distance metrics, since they do not in general satisfy triangle inequalities.
4

---

Published as a conference paper at ICLR 2020
where the agent reach the goal g using one of two paths. The first path has one intermediate state that leads to the target state with probability p. and an absorbing terminal state ST with probability 1- p. The other path has two intermediate states, but allows the agent to reach the target every time. The optimal dynamical distance will be 2, regardless of the value of p. causing the policy to always choose the risky path and potentially miss the target completely.
The definition of dynamical distances in Equation follows directly from how we learn the distance function, by choosing both S; and Sj from the same trajectory. Conditioning on both S; and sy is needed when the state space is continuous or large, since visiting two states by chance has zero or near-zero probability. We instead propose to use the distance as a negative reward, and apply reinforcement learning to minimize the cumulative distance on the path to the goal:
Do (st.
(Ø) P.p.
(5. g)
(4)
This amounts to minimizing the cumulative distance over visited states, and thus taking a risky action becomes unfavourable if it takes the agent to state that is far from the target at a later time. We further show that, under certain assumption, the policy that optimizes [Equation 4 will indeed acquire the correct behavior, as discussed in Appendix [A] and will converge to policy that takes the shortest path to the goal, as we show in Appendix B
We note that our simulated experiments below are run in deterministic environments and do not fully understand why cumulative distances work better than greedily minimizing the distances even in those cases. A comparison between these two cases is shown in Section 6.2
4.4 ALGORITHM SUMMARY
The dynamical distance learning (DDL) algorithm is described in Figure I] Our implementation uses
Algorithm 1 Dynamical Distance Learning
soft actor-critic (SAC) (Haarnoja et al. 2018c) as the policy optimizer, but one could also use any other off-the-shelf algorithm. In each iteration, DDL first samples a trajectory using the current policy, and saves it in a replay pool DD. In the sec- ond step, DDL updates the distance function by minimizing the loss in Equation 3) The distance
1: Input: d, V
> Initiaal policy and distance parameters > Emply replay pool
2:
Input: D
3:
repeat
4:
T Px. D DU
> Sample a new trajectory
5:
for = t N₂ do
V LLVlia a)
Minimize distance loss
7:
end for
8:
choose choose.goal())) g
D Chouse goal state
9:
for t N.
10:
- Φb - d v c. (d: d, g) Minimize policy loss end
function optimized for a fixed number of Nu stochastic gradient steps. Note that this method requires that we use recent experience from D. so
11:
12:
until converged
as to learn the distance corresponding to the current policy. In the third step, DDL. chooses a goal state from the recent experience buffer. We will describe two methods choose these goal states in Section 5] In the fourth step, DDL updates the policy by taking N₂ gradient steps to minimize the loss in Eqquaion 4| The implementation of this step depends on the RL algorithm of choice. These steps are then repeated until convergence.
5 GOAL PROPOSALS
In the previous section, we discussed how we can utilize learned distance function to efficiently optimize goal-reaching policy. However, a learned distance function only meaningful if evalu- ated at states from the distribution it has been trained on, suggesting that the goal states should be chosen from the replay pool. Choosing a goal that the policy can already reach might at first appear strange, but it turns out to yield efficient directed exploration, explained next.
Simple random exploration, such as e-greedy exploration or other strategies that add noise to the actions, can effectively cover states that are close to the starting state, in terms of dynamical dis- tance. However, when high-reward states or goal states are far away from the start state, such naive strategies unlikely to reach them. From this observation, exploration strategy that leverages the learned dynamical distances: we first use the policy to reach a known goal quickly as possible and then explore the vicinity of that goal. This way more time is
devise simple and effective
5

---

Published as a conference paper at ICLR 2020
DCLaw (hardware)
(b) DClaw (simulation)
(c) Inverted DoublcPendulum
(d) Hopper
HalfCheetah
(f) Ant
Figure 2: We evaluate method both in simulation and real-world robot. We show that method can learn to turn a valve with a real-world 9-DoF hand (a). and run ablations in the simulated version of the same task (b). We also demonstrate that method can learn pole balancing (c) and locomotion d e i skills in simulation.
left to randomly explore states far from the initial state and this way likely discovering useful states. We propose two different strategies for choosing the goals below.
5.1 SEMI-SUPERVISED LEARNING FROM PREFERENCES
DDL can be used to learn to reach specific goals elicited from user. The simplest way to do this is for a user to provide the goal state directly, either by specifying the full state, or selecting the state manually from the replay pool. However, desired state with preference queries. In this setting, the user repeatedly presented with a small slate of candidate states from the replay pool, and asked to select the that they prefer most. In practice, we present the user with a visualization the final state in several of the most recent episodes, and the user selects the one that they consider closest their desired goal.
also provide more convenient way to elicit the
For example, if the user wishes to train a legged robot to walk forward, they might pick the state where the robot has progressed the largest distance in the desired direction. The required user effort in selecting these states minimal, and most of the agent's 's experience still unsupervised, simply show that this semi-supervised
using the latest user-chosen state the goal. In our experiments, learning procedure, which we call dynamical distance learning from preferences (DDLfP) can learn rotate valve with real-world hand from just ten queries, and can learn simulated locomotion tasks using 100 simulated queries.
5.2 UNSUPERVISED EXPLORATION AND SKILL ACQUISITION
We can also use DDL to efficiently acquire complex behaviors, such as locomotion skills, in completely unsupervised fashion. From the observation that many high-reward states are far away from the start state, we can devise simple and effective exploration strategy that leverages our learned dynamical distances: we can simply select goals that are far from the initial state according to their estimated dynamical distance. We call this variant of our method "dynamical distance learning unsupervised" (DDLUS).
Intuitively, this method causes the agent to explore the "frontier" of hard-to-reach states, either discovering shorter paths for reaching them and thus making them no longer be on frontier, or else finding states further the fringe through additive random exploration. In practice, find that this allows the agent to quickly explore distant states in a directed fashion. In [Section 6 show that, by setting choose gDal arg maxged dg (So: g), where So is the initial state, can acquire effective running gaits and pole balancing skills in variety of simulated settings, While this approach is not guaranteed to discover interesting and useful skills in general, we find that, on variety of commonly used benchmark tasks, this approach to unsupervised goal selection actually discovers behaviors that perform better with respect to the (unknown) task reward than previously proposed unsupervised reinforcement learning objectives.
EXPERIMENTS
Our experimental evaluation aims to study the following empirical questions: (1) Does supervised regression provide good estimator of the true dynamical distance? (2) Is DDL applicable real-

---

Published as a conference paper at ICLR 2020
world, basied robotic control tasks? (3) Does DDL provide efficient method of learning skills a) from user-provided preferences, and b) completely unsupervised?
We evaluate our method both in the real world and in simulation on set of state- and vision- based continuous control tasks. We consider a 9-DoF real-world dexterous manipulation task and standard OpenAI Gym tasks (Hopper-v3, HalfCheetah-v3, Ant-v3, and InvertedDoubleeendune v2). For all of the tasks, we parameterize our distance function as a neural network, and use soft actor-critic (SAC) (Haarnoja al. 2018b) with the default hyperparameters For state-based tasks, we use ferd-forward neural networks and for the vision-based tasks we add a convolutional preprocessing network before these fully connected layers. The image observation for all the bssonn tasks are 3072 dimensional (32x32 RGB images). Further details are presented in Appendix E
learn the policy.
We study question (1) using a simple didactic example involving navigation through a two- dimensional S-shaped maze, which we present Appendix C are studied in the following sections.
The other two research questions
6.1 AISION-BASED WEALD MANIPULATION FROM HUMAN PREFERENCES
To study the question (2), we apply DDLfP to a real-world vision-based robotic manipulation
state
SAC W roward)
Vislan
(rns distrce to goo)
IP [CAPs prrkenooe)
VIDE
task. The domain consists of 9-DoF "DClaw"
hand introduced by Ahn et al. (2019), and the manipulation task requires the hand to rotate
valve 180 degrees, as shown in Figure I The human operator is queried for a preference ev- ery 10K environment steps. Both the vision- and state-based experiments with the real robot use 10 queries during the first 4 hours of an 8- hour training period. Note that, for this and all the subsequent experiments, DDLíP does not have access to the true reward, and must Iearn entirely from preference queries, which in this case are provided by human operator.
BDX 103K
150
100K 200K
300K
number samplea
numtber sam plus
Figure 3: (Left) learning curves for the valve tation task learned from state. (Right) Same task from vision. The curves correspond to the final distance (measured in radians) of the valve from the target angle during a rollout. Our method (DDLIP, orange) solves the task in 8 hours. Its performance is comparable to that of SAC with true rewards, and VICE with example outcome images. DDLiP only requires 10 preference queries, and learns without true rewards or out- come images. We compare our method in the sim- ulated version of this task in Figure 5
Figure 3
presents the performance over the course of training. DDLfP uses 10 preference queries learn the task and its performance is comparable to that of SAC trained with ground truth shaped reward function. We also show a comparison to variational inverse control with events (VICE) (Singh et al. 2019). a recent Massifier-based reward specification framework. Instead of preference queries, VICE requires the user to provide examples of the desired goal state at the beginning of training (20 images in this case). For vision-based tasks, VICE involves directly showing images of the desired outcome to the user, which requires physically arranging scene and taking picture of it. Preferences, on the other hand, require a user to simply select one state out of a small set, which can be done with a button press and done e.g. remotely, thus often making it substantially less labor-intensive than VICE. As we can see in the experiments, DDLIP achieves similar performance with substantially less operator effort, using only a small number of preference queries. The series of goal preferences queried from the human operator are shown in Appendix D
6.2 ABLATIONS, COMPARISONS, AND ANALYSIS
Next, we analyze design decisions in our method and compare it to prior methods in simulation. First, we replace the cumulative objective in Equation 4 with objective that greedily minimizes the distance function trained with supervised loss. This objective is unable to learn the task from either state or vision observations. Next, we replace the supervised loss in Equation of our DDL method with a temporal difference (TD) Q-learning style update rule that learns dynamical distances with approximate dynamic programming. The results in Figure 5) show that, all else being equal, the TDDbaaeed method fails to learn successfully from both low-dimensional state and vision observa-
7

---

Published as a conference paper at ICLR 2020
Hopperud
Aptu JCLP brand
Criittac at Ghece W m1000)
-006
000
9.59
8:18
0.15
9.20
2
(náloo 1803
Bon deape
málton SKOS
10 depe.
Figure 4: Learning curves for MuloCo tasks with DDLfP. The y-axis presents the true return of the task. We compare DDLff SAC trained directly from the true reward function, which provides an oracle upper bound baseline, and the prior method proposed by Christiano et al. (2017). The prior method uses an on-policy RL algorithm which typically requires more samples than off-policy algorithms, and thus we also plot its final performance after 20M training steps with red star. At the time of the submission, the Ant-v3 run is still in progress and the complete learning curve will be included in the final.
tions. Figure 5 further shows a comparison between using the dynamical distance as the reward in comparison to reward of 1 for each step until the goal reached, which corresponds to hindsight experience replay (HER) with goal sampling replaced with preference goals (Andrychowicz et al., 2017]. We see that dynamical distances allow the policy to reach the goal when learning both from state and from images, while HER is only successful when learning from now-dimensional states.
These results are corroborated by prior results in the literature that have found that tempo- ral difference learning struggles to capture the true value accurately (Lillicrap
abrll
DDL tours)
vision
< Sin doan t3 gool
DDL groady ot, jablatoni
DDK wi TE uporee (usstion)
HER (Aootrychiiezzz wfal 2012)
al., 2015) Fujimoto et al., 2018). Note that prior work work does not use the full state as the goal, but rather manually selects a low-dimensional subspace, such as the location of an object, forcing the distance to focus on rele-aele objects (Andrychowicz et al.) 2017). Our method learns distances between full image states (3072- 2072-dimensionna)) while HER uses 3- dimensional goals, a difference of two orders of magnitude in dimensionality. This difficulty of learning complex image- image-based based goals is further corroborated in prior work (Pong et al. 2018 Warde-
1004
201K
300K
103K
200K
3004
number of
number of timesteps
Figure 5: We compare DDL against alternative methods for learning distances on the simulated valve turning task, when learning from the under- lying low-dimensional state (left) and from im- ages (right). Dynamical distances used greed- ily (orange) or learned with TD (green) generally perform poorly. HER (red) can learn from low- dimensional states, but fails to learn from images. Our method, DDL.fP (blue) successfully learns the task from either states or images.
Nair et al. 2018 Pong et al. 2019
Farley et al.
2018
Figure 4 presents results for learning from preferences via DDLIP (in green) set of continuous control tasks to further study the question (3,a). The plots show the true reward for each method on each task. DDLIP receives only sparse preferences as task-specific supervision, and the preferences in this case are provided synthetically, choosing the state that has progressed the largest distance from the initial state in the desired direction, i.e. the state with largest x-coordinate value. However, this still provides substantially less supervision signal than access to the true reward for all samples. We compare to (Christiano et al. 2017). which also uses preferences for learning skills, but without the use of dynamical distances. The prior method is provided with 750 preference queries over the course of training, while our method uses 100 for all locomotion tasks, and only a single query for the worrreddDDDubleeednuuuuuee
as the initial state and the goal states coincides Note that Christiano et al. (2077) utilizes an on-policy RL algorithms, which is less efficient than SAC. However, DDLIP outperforms this prior method in terms of both final performance and learning speed all tasks, except for the H3pper-v3 task.
Locomotion tasks like the ones considered here do not fit into DDL. framework directly. In this particular case of locomotion tasks, we can fix the issue by considering a case where the ultimate task to reach a specific goal, i.c. the operator would always choose the goal to be the state closest
In our case, one preference query amounts choosing one of five states, whereas in Christiano et al. query consists always of two state sequences.
2017
8

---

Published as a conference paper at ICLR 2020
Hopper-wp
Ha Chestah-v
500
202
DOLUB DVAYN
6K
100
0.20
0.03
5.10
0.15
0.0
2 iillion stupo
sappa
min staps
DD.05 Jouns)
DANN
sslanoa
disance
dissares
Figure 6: (Top) Learning curves for DDLUS. The y-axis plots the environment return (not accessible during the training) for for H3pper-v3, HalfCheetah-väs and Ant-v3. (Bottom) Frequency histograms of skills learned with DDLUS (blue) and DIAYN (orange) (Eysenbach et al.) (018) across different training runs, evalu- ated according to the travelled L2-distance from the origin.
and the L2-distance travelled from the origin
to the "ultimate task goal". In that case, we can see the locomotion task to be the limit case where the ultimate goal is as far as possibly reachable within the maximum episode length.
6.3 ACQUTRING UNSUPERVISED SKILI.S
Finally, we study question (3,b) in order to understand how well DDLUS can acquire skills without any supervision. We structure these experiments analogously to the unsupervised skill learning experiments proposed by EEsenbach et al.] (2018), and compare to the DIAYN algorithm, another unsupervised skill discovery method, proposed in their prior work. While our method maximizes the complexity of the learned skills by attempting to reach the furthest possible goal, DIAYN maximizes the diversity of learned skills. This of course produces different biases in the skills produced by the two methods. FFigure 6 shows both learning curves and histograms of the skills learned in the locomotion tasks with the two methods, evaluated according to how far the simulated robot in each domain travels from the initial state. Our DDLUS method learns skills that travel further than DIAYN, while still providing a variety of different behaviors (e.g., travel in different directions). This experiment aims to provide direct comparison to the DIAYN algorithm (Eysenbach et al.) 2018), though a reasonable criticism is that maximizing dynamical distance is particularly well- suited for the criteria proposed by Eysenbach et al.] (2018). We also evaluated DDLUS on the InvertedDoubleeenduuum
domain, where the task is to balance a pole on a cart. As can be seen from Figure 6. DDLUS can efficiently solve the task without the true reward, as reaching dynamically far states amounts to avoiding failure as far as possible.
7 CONCLUSION
We presented dynamical distance learning (DDL.,, an algorithm for learning dynamical distances that can be used to specify reward functions for goal reaching policies, and support both unsuper- vised and semi-supervised exploration and skill discovery. Our algorithm uses a simple and stable supervised learning procedure to learn dynamical distances, which are then used to provide a reward function for standard reinforcement learning method. This makes DDL straightforward to apply even with complex and high-dimensional observations, such as images. By removing the need for manual reward function design and manual reward shaping, our method makes it substantially more practical to employ deep reinforcement learning to acquire skills even with real-world robotic sys- tems. We demonstrate this by learning valve-turning task with a real-world robotic hand, using 10 preference queries from human, without any manual reward design or other examples or su- pervision. One of the main limitations of our current approach that, although it can be used with an off-policy reinforcement learning algorithm, it requires on-policy data collection for learning the dynamical distances. While the resulting method is still efficient enough to learn directly in the real world, the efficiency of our approach can likely be improved in future work by lifting this limitation.
This would not only make learning faster but would also make it possible to pre-traie pre-train dynamical distances using previously collected experience, potentially making it feasible to scale our method to a multi-task learning setting, where the same dynamical distance function can be used to learn multiple distinct skills.

---

Published as a conference paper at ICLR 2020
ACKNOWLEDGMENTS
We thank Vikash Kumar for the DClaw robot design, Nicolas Heess for helpful discussion, and Henry Zhu and Justin Yu for their help on setting and running the hardware experiments. This research was supported by the Office of Naval Research, the National Science Foundation through IIS-165 181313513331388 and IIS- 170-17,, and Berkeley DeepDrive.
REFERENCES
Michael Ahn, Henry Zhu, Kristian Hartikainen, Hugo Ponte, Abhishek Gupta, Sergey Levine, and Vikash Kumar. Robel: Robotics benchmarks for learning with low-cost robots. Conference on Robot Learning (CoRL), 2019.
Marcin Andrychowicz, Filip Wolski, Alex Ray. Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems (NeurIPS), pp. 1048-5055 2017.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems (NeurIPS), pp. 4299-4307, 2017.
B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine. Diversity is all you need: Learning skills without reward function. arXiv preprint 1802.v/800.000 2018.
Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. Search on the replay buffer: Bridg- ing planning and reinforcement learning. arXiv preprini 19X6vv1906.05222 2019.
Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. International Conference on Machine Learning (ICML), 2018.
Carlos Florensa, Jonas Degrave, Nicolas Heess, Jost Tobias Springenberg, and Martin Ried- miller. Self-supervised learning of image embedding for continuous control. arXiv preprint arXiv:1901.00943 2019.
Fujimoto, H. van Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning (ICMI), 2018.
T. Haarnoja, V. Pong, A. Zhou, M. Dalal, P. Abbeel, and S. Levine. Composable deep reinforce- ment learning for robotic manipulation. International Conference on Robotics and Automation (ICRA). IEEE, 2018a.
T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning (ICML), 2018b.
T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, and S. Levine. Soft actor-critic algorithms and applications. arXiv preprint 11X1v:1812.05505 2018c.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. In Advances in Neural Information Processing Systems (NeurIPS). pp. 1109-1117, 2016.
L. P. Kaelbling, M. L. Littman, and A. W. Moore. Reinforcement learning: A survey. Journal of artificial intelligence research, 4:237-285, 1996.
Leslie Pack Kaelbling. Learning to achieve goals. In International Joint Conference on Artificial Intelligence (IJCAI), pp. 1094-1099 Citeseer, 1993.
S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17(39):1-40 2016.
P. Lillicrap, J. Hunt, A. Pritzel, N. Heess, Erez, Y. Tassa, D. Silver, and D. Wierstra. Contin- control with deep reinforcement learning. arXiv preprint arXiv: 15090097,, 2015.
10

---

Published as a conference paper at ICLR 2020
A Rupam Mahmood, Dmytro Korenkevych, Brent J Komer, and James Bergstra. Setting up reinforcement learning task with a weal-world robot. In International Conference on Intelligent Robots and Systems (IROS), pp. 4635-4640. IEEE, 2018.
V. Mnih, K. Kavukcuoglu, D. Silver, A. A Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Ried- miller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-555 533, 2015.
Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Vi- sual reinforcement learning with imagined goals. In Advances in Neural Information Processing Systems, pp. 9191-9200, 2018.
lan Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In Advances in neural information processing systems (NeurIPS), pp. 4026- 4034, 2016.
Vitchyr Pong, Shixiang Gu, Murtaza Dalal, and Sergey Levine. Temporal difference models: Model- free deep RL for model-based control. International Conference on Learning Representations (ICLR), 2018.
Vitchyr Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew-fit: State-covering self-supervised reinforcement learning. arXiv preprint arXiv:9003.03698, 2019.
Nikolay Savinov, Alexey Dosovitskiy, and Vladlen Koltun. Semi-parametric topological memory for navigation. In International Conference of Learning Representations (ICIRR). 2018.
Schaul, D. Horgan, K. Gregor, and Silver. Universal value function approximators. In Inter- national Conference on Machine Learning (ICML), pp. 1312-1320, 2015.
Avi Singh, Larry Yang, Kristian Hartikainen, Chelsea Finn, and Sergey Levine. End-to-end robotic reinforcement learning without reward engineering. arXiv preprint UrXiv:1904.07833 2019.
Richard S Sutton, Joseph Modayil, Michael Delp. Thomas Degris, Patrick M Pilarski, Adam White, and Doina Precup. Horde: A scalable real-time architecture for learning knowledge from unsuper- vised sensorimotor interaction. In The 10th International Conference on Autonomous Agents and Multiagent Systems-Volume 2, pp. 761-768. International Foundation for Autonomous Agents and Multiagent Systems, 2011.
David Warde-Farles Tom Van de Wiele, Tejas Kulkarni, Catalin Ionescu, Steven Hansen, and Volodymyr Mnih. Unsupervised control through non-parametric discriminative rewards. arXiv preprint arXiv:1811.11335 2018.

---

Published as a conference paper at ICLR 2020
Appendices
A CORRECT BEHAVIOR IN THE PATHOLOGICAL MDP
In this appendix we show that the policy that maximizes the objective in in(Equaiion with the reward rg (s, = -d (s. g), where d' is given by Eqquation 2 prefers safe actions over risky actions.
Assume that c(st, S(41) = 1g [se] is an indicator function that is st is a goal state or terminal state and I for all the other states. We can now write the definition of d" infinite sum and substitute rg((s, a) + -d" g) in Equation
20
((x) LE ETT
Σ
1 [Sk] 56
St a/j
a.
(5)
The first term (k 0) the inner sum depends only on s, which is given, and the term can thus be moved outside the inner expectation:
00
o Σ
((x)
E
Σ
(1 + Σ
E
11 [s]] S/ St, ab a,
(6)
Next, note that the statistics of the inner expectation over (si, a1) are the same as the outer expec- tation over ($1, a1), they are both conditioned on the same az). Thus, we can condition the second expectation directly on (s',, = (se-1: az+1):
0
]]
((T) B
E'' [Sc] + Σ ET'Er
B
Σ
1 (s4) s S1,1, a++1
We can now apply the argument before and move 1g [s1] outside the inner expectation. Repeating these steps multiple times yields
0c Σ
[(x) Eyon
Σ
, [st]
+ Σ ++1 1 Ag
[st+1]
2222 ig [Sr+2] t=0
8
=-E
( 1) [c
Assuming that the agent always reaches the goal relatively quickly compared to the discount factor, such that 7² As 1, the trajectories that take longer dominate the loss due to the (t + 1) factor. There- fore, an optimal agent prefers actions that reduce the risk of long, highly suboptimal trajectories, avoiding the pathological behavior discussed in Section
B POLICY IMPROVEMENT WHEN USING DISTANCE AS REWARD
In this appendix we show that, when we use the negative dynamical distance -d" as the reward function in RL, we can learn an optimal policy with respect the true dynamical distance, leading to policies that optimize the actual number of time steps needed to reach the goal. This result is non- trivial, since the reward function does not first glance directly optimize for shortest paths. Our proof relies the assumption that the MDP has deterministic dynamics. However, this assumption holds all our experiments, since the MuJoCo benchmark tasks are governed by deterministic dynamics. Under this assumption, DDL will learn policies that take the shortest path to the goal at convergence, despite using the negative dynamical distance as the reward.
Let d' (s, min, d (s. g) be the optimal distance from state to goal state g. Let n be the optimal policy for the reinforcement learning problem with reward rg (s, a) = -d" (s.g), DDL can be viewed alternating between fitting dº the current policy n, and learning new policy n' that
12

---

Published as a conference paper at ICLR 2020
is optimal with respect to the reward function given by -d We can now state main theorem as follows:
Theorem 1. Under deterministic dynamics, for any state s and g, we have:
1. d' (s,g) S dd
2. If d (s.g) = d" (s. then d=' (s,g) = =d (s. (s.g).
This implies that, when the policy converges. such that nO n, the policy nt achieves the optimal distance to any goal, and therefore is the optimal policy for the shortest path reward function (e.g., the reward function that assigns a reward of -1 for any step that does not reach the goal).
Proof.
Without loss of generality. we assume that our policy is deterministic, since the set of optimal policies in an MDP always includes at least one deterministic policy. We also assume that is terminal state and thus d(g, g) 0. Let denote the action of policy æ state as (s)). We start by showing that d=' (s, I d= ds. We fix a particular goal g. Let S& = {s; d" (s.g) k} be the set of states that takes & steps under π to reach the goal. We show that d= (s, g) S d g) k for all E Sş for each & by contradiction.
Part 1
For k = 0. Sg {g} is just the single goal state and d (g.g) (g. 0 by definition. For = 1, for all E S1, there is an action a that reaches the goal state as the direct next state. Therefore, the optimized policy n' would still take the same action a on these states and d' (s. = 1.
Now assume that the opposite is true, that d' (s, > d" (8. g) for some states. Then, there must be a smallest number K' > and state So E Sx such that d" (So, g) = T > d= (su. g) = K. Now let us denote the trajectory of states taken by x starting from Sy as {sa. $1, $1...., SK g). and the trajectory taken by x as {%) = 8o, $1 .... St =g} Let (.) denote the accumulated discounted sum of distance as defined in Equation + By our assumption T > K, and since is optimal with
respect to the reward rg (s, a) d g), we have
K-1
K-1 7(K - 1
Cx = Σ
S = S
dd = Σ
(9)
1=0
1=0
1=0
Then there must be time i < K such that d" (5 g) < d (s g) = K - 1- E. Therefore ss E Sk for some < K f. However, starting from s/c, we have d= (s)) T dd (s/ g). Therefore, we reached a contradiction with our assumption that d=' (s,g) S d d= (s, for all k K such that E Sy. Therefore, d'' (s. d= (s,g) holds for all states.
T - 1- K -
Now we show the second part: if (s, g) d=' (s,g), then (s g) d (s. g). We prove this with similar argument, grouping states by distance. Let S = ds g) = k- be the set of states that takes steps under the optimal policy to reach the goal. Note that, for any arbitrary policy we have d (s. g) 2 d' (s,, g) by definition, since d' is the optimal distance.
Part 2
Suppose that d= g) > (s, g) for some state s. Then there must be a smallest integer K 2 such that there exists state So E Si where d (su. g) > d' (So. g). For all k K, have d' ds g) = d' for all s E St- Now starting from that state Su, let the trajectory of states taken by be (So. S1..... ST = g}. Note that since d (So. g) d" (S g) K, T K. Let ň be the policy such that agrees with x= on So and agrees with и everywhere else. At the first step, â lands state s/. Since So is K steps away from g under d", ii must be K 1 steps away under d and ii E S Therefore, since and TT" agrees on all states that are less than K steps away from goal
*Of course, the actual DDL algorithm interleaves policy updates and distance updates. In this appendix, we analyze the "policy iteration" variant that optimizes the policy to convergence, but the result can likely he extended to interleaved updates in the same way that policy iteration can be extended into interleaved actor-critic method.
13

---

Published as a conference paper at ICLR 2020
g, ft would take the same action as n and hence take another K steps to goal g. Now let us denote the trajectory taken by स as {s/ = So: sp...., = g]. We compare the discounted sum of rewards of and â under the reward function "'g(s, a) -d (s, g).
T-1
T-1
(T) Σ
r'd (8i, () = d (So: g) +
Σ
d
(81,
1=
1m1
dso g)
7 7T - 2 d (S0. g)
Σ
7'(K -
i=1
K-1
K-1
d= d g)
Σ
odd (s/
Σ
n' g) = (x)
(10)
10
Therefore, we can see that स is better policy than и. Then the optimal policy x' under this reward must be different from æ on at least one state. Hence d' (s, g) + d=d (s, g).
We' 'e now reached the conclusion that if d= (s.g) * d' then d" (s.g) * d d'' g). Hence, by contraposition, if d= (s, g) = d= (s. g), then it must be that d= (s,g) = d" (s. g). Our proof is thus complete.
C DIDACTIC EXAMPLE
Our didactic example involves simple 2D point robot navigating an S-shaped maze. The state space is two-dimensional, and the action is two-dimensional velocity vector. This experiment is visualized in Figure The black rectangles correspond blue star. The learned distance from all points in the maze to the goal is illustrated with a heat map, in which lighter colors correspond to closer states and darker colors to distant states. During the training, the initial state is chosen uniformly at random, and the policy is trained to reach the goal state. From the visualization, is apparent that DDL learns an accurate estimate of the true dynam- ical distances in this domain. Note that, in contrast to naïve metrics, such Euclidean distance, the dynamical distances conform to the walls and provide an accurate estimate of reachability, making them ideally suited for reward shaping.
walls, and the goal is depicted with
14

---

Published as a conference paper at ICLR 2020
0
0
2
-4
-2
0
2
-4
-2
2
2
4
8
10
12
14
0
2
4
8
10
12
14
(a) Our method
(b) Ground truth
Figure Evaluation of the learned distance in a 2D point environment, The state is the xy- coordinates of the point, and action corresponds to 2D velocities. The black bars denote walls, blue star is goal state, and the heat map denotes the estimated distance to the goal. (a) Our method learns an accurate estimate of the shape of the distance function. (b) Ground-truth distance.
15

---

a Published as a conference paper at ICLR 2020
 SUURRITE FOR
FOR REAL- WORRD D OCLAN REAL
D
presented in Sec A Figure& 8: Human preference ceeference quue queries for the -asion-based vision-based DClaw experiment pepeermeettpreeeee p tion 6. Each image row presents the set of images
i a shown to the human oumanoopa operator on a
quebt sing single
round. On each now,th corresponds to the last goal. For each query. most
the first 10 images correspond to the last states of the most ecent wweaaanopppltttti pich
reenttoon rollouts co q the human operator picks a h
and new
right-
the fettt-most-immme image S inputting its index (between 0-10) into a text- bee.baaeeeeeee interface. The goals selected by h soeggoas
goal by
are highlighted with white borders.
16

---

Published as a conference paper at ICLR 2020
E TECHNICAL DETAILS
All our experiments use Soft Actor-Critic as the policy optimizer, trained the default parameters by provided by the authors in (Haarnoja et al. 2018c).
For all of the tasks, we parameterize our distance function as a neural network. For state-based tasks, use feed-forward neural networks with two 256-unit hidden layers. For the vision-based tasks add a convolutional preprocessing network before these fully-connected layers, consisting of four convolutional layers, each with 64 3x3 filters. Both cases use Adam optimizer with learning rate 3e- 4 and TensorFlow's default momentum parameters. The image observation for all the vision-based tasks are 3072 dimensional (32x32 RGB images).
Most important hyperparameters that we swept over in the final experiments, namely the size of the on-policy pool for training the distance function and the number of gradient steps per environment samples, are presented in Table below:
Environment
gradient steps per environment steps
on-policy pool size
Inverte/DaablePeeddlua222 Hopper-v3
1/64
100k:
1 1/64
16k
HalfCheetah-v3
1/16
16k
Ant-v3
1/64
10k
DClaw (both state and vision)
1/16
100k
Table 1: Distance estimator hyperparameters.
For the DDLUS goal proposals, we consider all the samples in the distance on-policy pool as the goal candidates. For DDLfP, we present the operator the last states (ST-1) of the last N episodes, 10 for the hardware DClaw.
where = 5 for all the simulated experiments, and
As discussed in Seeciion 5 for both DDLUS and DDLfP, the agent needs explore in the vicinity of the goal state. In practice, we implement this by switching to random uniform policy after 0.9T timesteps of each episode, where T is the maximum episode length (1000 for all the mujoco tasks and 200 for the DClaw task).
17


Reviews:
[{"review_id": "SJghMgk0YH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\nThe authors propose learning the expected # of time steps to reach a given goal state from any other state which they call: dynamical distances. The motivation suggests that such an ability would help the agent in reaching new goal states and therefore in learning complex tasks. Task goal is given with a small amount of preference supervision. The claim is that dynamical distances can be used in a semi-supervised regime, where unsupervised interaction with the env is used to learn the dynamical distances. Evaluation is done on a real-world robot and in simulation, the method can learn to turn a valve with a real-world 9-DoF hand, using raw image observations and just ten preference labels, without any other supervision. The ideas presented in this work are interesting, but I have some major concerns (please see detailed comments below). The paper is well written and is mostly easy to understand. \n\t\t\t\t\nDetailed comments:\nThe learning process follows two steps: \n-distance evaluation step: learn a policy specific dynamical distance parameterized by \\psi: by computing the expected # of time steps it took for \\pi to reach the goal state.\n-policy improvement step: use the learned distance fn to optimize a policy to reach the goal\n\nOne weakness of the proposed approach is that the supervised regression step requires an on-policy experience which can be very expensive as pointed out by the authors as well. The authors suggest that because they use this as an intermediate representation this does not impact learning. Wouldn’t it be possible to *simultaneously* learn the dynamical distance during the off-policy policy-learning step?\n\nThe second weakness of the proposed approach is that in the policy improvement step it is assumed that a goal state is given. It is also weird to use goals that are already reachable. Please provide clarifications. \n\nAlgorithm 1 learns the distance corresponding to the current policy. Wouldn’t it be a more generalized approach to learn the dynamical distance irrespective of the policy i.e., not policy-specific? Or even better, would it be possible to instead learn the dd applicable to a class of policies? Assuming behaviourally similar policies can have the same dynamical distance? \n\nOn an intuitive level, it would be useful for the reader to make concrete how is this approach different from learning goal conditioned policies? The dynamical distances reflect the distance to the goals: goals are either 1) sampled from an experience replay buffer, 2) given by user preference, 3) chosen more smartly by the dynamical distance learned.\n\nIt is not clear if in sec 4.3 “the optimal dynamical distance will be 2”, but since the Eq.2. use the expected distance, does the above statement hold?\n\nEmpirical analysis: The authors aim to address three questions through the experiments: Regarding Q2: Is DDL applicable to real-world, vision-based robotic control tasks? Fig 1 shows interesting tasks and a useful contribution. \n\nIn terms of learning from human preferences; the experiments do not seem sufficient and need more comparative analysis. In particular, the authors should compare to other well-known techniques where learning from human preferences is performed such as [1], [2]. \n\nThe same is the case with unsupervised skill acquisition, the authors only compared to Diayn. I would suggest comparing to other relevant baselines such as [3], [4], [5].  Fig 6 primarily suggests the obvious that Diayn maximizes for diversity while DDL maximizes for distance. Moreover, even this comparison seems incomplete as I am not sure why the top row in Fig 6 only shows the proposed approach and not the baseline (diayn). Unsupervised skill acquisition makes it even more important to understand what is the nature of skills learned. I find that analysis is missing here as well, and would add completeness to these results. \n\nConsidering the paper’s claims are heavily based on the ability to learn skills, there is no qualitative analysis of what kind of skills are learned. For e.g. it is mentioned that DDL method that can learn a 9-DoF real-world dexterous manipulation task, but what are the exact skills learned here per task? It would be useful to do a more rigorous analysis of each skill learnt. “Videos of the learned skills can be found on the project website: https://sites.google.com/view/skills-via-distance-learning.” However, the link throws a 404 error.\n\nOverall:\t\t\nThe paper is well written, real-world experiments are a ++, interesting mix of ideas along with applicability to multiple problems. The main experiment is shown with preferences where comparisons do not seem enough and the paper could be made stronger by thorough comparisons. \n\n[1] Christiano, Paul F., et al. \"Deep reinforcement learning from human preferences.\" Advances in Neural Information Processing Systems. 2017.\n[2] Ashesh Jain, Brian Wojcik, Thorsten Joachims, and shutosh Saxena. Learning trajectory preferences for manipulators via iterative improvement. In NIPS, 2013.\n[3] Gupta, Abhishek, et al. \"Unsupervised meta-learning for reinforcement learning.\" arXiv preprint arXiv:1806.04640 (2018).\n[4] Achiam, Joshua, et al. \"Variational option discovery algorithms.\" arXiv preprint arXiv:1807.10299 (2018).\n[5] Karol Gregor, Danilo Rezende, and Daan Wierstra. Variational Intrinsic Control. pages 1–15, 2016.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "S1l-_6UGtB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an approach to do reinforcement learning without reward engineering. Instead, steps to reach goals are used to update policy.\n\nHere are some questions:\n(1) The reason to use cumulative distance instead of greedily chooses the shortest distance is to avoid risky state. But it is also mentioned in the paper that the experiments in the paper all have deterministic dynamics (and one of the assumptions in Appendix B). Can the authors elaborate on this? For example, how is the performance of a policy that just directly minimizes the distance function? Instead of using the made-up toy example.\n\n(2) For the locomotion task examples, I am confused as to why choosing the state with the highest immediate reward will be a good preference state. For example, a humanoid that falls forward can have high forward velocity while has no chance for recovery. While the performance seems reasonable, the systems presented are mostly stable systems (with the exception of hopper). It will be nice to show similar performance in more unstable systems, e.g, walker2d, humanoid.\n\n(3) Some assumption in the proof in appendix B is not true. d(g, g) is not 0 for all dynamical systems. For example, a hopper with a state that has none zero forward velocity will take more than 0 steps to return to the same state.\n\n(4) “Because the trajectories have a finite length, we are effectively ignoring the cases where reaching sj from si would take more than T − i steps, biasing this estimate toward zero, but since the bias becomes smaller for shorter distances.” I am confused about this sentence, what does it mean? For example, if the dynamical system is the hopper, (s_i, s_{i+1}) will make d(s_i, s_{I+1}) be 1, and for all we know d(s_{i+1}, s_i) can be huge. How does this sentence apply in this situation?\n\n(5) An appendix with hyperparamters will be helpful. For example, how much data are needed for updating the distance function?\n\n(6) “We also evaluated DDLUS on the InvertedDoublePendulumv2 domain, where the task is to balance a pole on a cart. As can be seen from Figure 6, DDLUS can efficiently solve the task without the true reward, as reaching dynamically far states amounts to avoiding failure as far as possible.” Why reaching dynamically far states is equivalent to avoiding failure? Shouldn’t the policy learn to stay as close to the initial state as possible?\n\nDespite the questions raised above, the techniques are interesting, simple, and effective, as demonstrated on some image-based tasks on real robots. While I understand that this method will be very effective for relatively static tasks like the valve turning, I fail to see how it can apply in unstable locomotion tasks, as mentioned in some of the questions above. \n\nNormally I will give rating 5 to this paper, but the system doesn't give me this option, so I have to lower it to 3 instead. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HJgmdWcJqr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I'm afraid I found this paper somewhat confusing and hard to see the big picture but I also acknowledge that I am not an expert in deep, model-free RL and my RL experience is mostly in model-based RL and I am happy for this to be taken into consideration when evaluating my review - apologies if I miss something that is well known in that community.\n\n\nAfter a few readings my understanding of the paper is that there is a desire to develop a proxy (or basis) that is first determined in a reward free setting, agnostic to a goal. Subsequently, this proxy can then be combined with a goal and improve efficiency in developing a suitable policy for that goal rather than starting from scratch with something like Q-learning. Is this the case or have I misunderstood something? \n\nIf this is the case, I can envisage an argument for such a setting although it surprises me that something similar has not been performed previously - the approach seems to be to be quite similar to a model-based setting where this distance regressor is in many ways similar to a model in that it is trained from unsupervised trajectories and effectively encodes the transitions between states? Given that the distance in (3) is essentially trained from trajectories with a fixed reward structure in (2) I'm very surprised that a Q-Learning approach doesn't offer similar performance when the task is specified - can the authors please provide more details as to why this approach is expected to be so superior to Q-learning?\n\nThe distances between states from the regressor is policy dependent and then this distance is used to inform the policy update - how can we be certain that this alternating optimization will not fall into a local minimum depending on the initial policy? None of the experiments seem to check for this?\n\nThe experiments are quite specific, both in terms to the particular experiment and very specific previous work, which makes it hard to judge empirically the merits of the approach. Again, I would defer to others with more experience in this field to know whether or not this is standard practice?\n\n\n- Something that I think needs to be changed is the continual use of the term distance when it is not a valid mathematical distance (this point is noted by the authors but then the term is used continuously and the notation used would be standard notation for an actual distance). Would it not be more appropriate to call it a dissimilarity?\n\nOther notes:\n\n- In section 4.4 I think it is really bad practice to suggest that a fixed number of stochastic gradient descent steps avoid over-fitting - I know of nothing that guarantees this statement.\n\n- I think it is not a given that it is easier to look through trajectories and express preferences over certain video frames as opposed to taking images of the desired goal at the start - are the experiments comparing like with like? I think both methods could be presented with the same supervision to make the comparisons fair.\n\n- I find the term skill discovery strange - would it not be more helpful to include model-free RL in the title?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors present a method to learn the expected number of time steps to reach any given state from any other state in a reinforcement learning setting.  They show that these so-called dynamical distances can be used to increase learning efficiency by helping to shape reward.  After some initial discussion, the reviewers had concerns about the applicability of this method to continuing problems without a clear goal state, learning issues due to the dependence of distance estimates on policy (and vice versa), experimental thoroughness, and a variety of smaller technical issues.  While some of these were resolved, the largest outstanding issue is whether the proper comparisons were made to existing work other than DIAYN.  The authors appear to agree that additional baselines would benefit the paper, but are uncertain whether this can occur in time.  Nonetheless, after discussion the reviewers all appeared to agree on the merit of the core idea, though I strongly encourage the authors to address as many technical and baseline issues as possible before the camera ready deadline.  In summary, I recommend this paper for acceptance.
Decision: Accept (Poster)