Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: I'm afraid I found this paper somewhat confusing and hard to see the big picture but I also acknowledge that I am not an expert in deep, model-free RL and my RL experience is mostly in model-based RL and I am happy for this to be taken into consideration when evaluating my review - apologies if I miss something that is well known in that community.


After a few readings my understanding of the paper is that there is a desire to develop a proxy (or basis) that is first determined in a reward free setting, agnostic to a goal. Subsequently, this proxy can then be combined with a goal and improve efficiency in developing a suitable policy for that goal rather than starting from scratch with something like Q-learning. Is this the case or have I misunderstood something? 

If this is the case, I can envisage an argument for such a setting although it surprises me that something similar has not been performed previously - the approach seems to be to be quite similar to a model-based setting where this distance regressor is in many ways similar to a model in that it is trained from unsupervised trajectories and effectively encodes the transitions between states? Given that the distance in (3) is essentially trained from trajectories with a fixed reward structure in (2) I'm very surprised that a Q-Learning approach doesn't offer similar performance when the task is specified - can the authors please provide more details as to why this approach is expected to be so superior to Q-learning?

The distances between states from the regressor is policy dependent and then this distance is used to inform the policy update - how can we be certain that this alternating optimization will not fall into a local minimum depending on the initial policy? None of the experiments seem to check for this?

The experiments are quite specific, both in terms to the particular experiment and very specific previous work, which makes it hard to judge empirically the merits of the approach. Again, I would defer to others with more experience in this field to know whether or not this is standard practice?


- Something that I think needs to be changed is the continual use of the term distance when it is not a valid mathematical distance (this point is noted by the authors but then the term is used continuously and the notation used would be standard notation for an actual distance). Would it not be more appropriate to call it a dissimilarity?

Other notes:

- In section 4.4 I think it is really bad practice to suggest that a fixed number of stochastic gradient descent steps avoid over-fitting - I know of nothing that guarantees this statement.

- I think it is not a given that it is easier to look through trajectories and express preferences over certain video frames as opposed to taking images of the desired goal at the start - are the experiments comparing like with like? I think both methods could be presented with the same supervision to make the comparisons fair.

- I find the term skill discovery strange - would it not be more helpful to include model-free RL in the title?
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: Summary
The authors propose learning the expected # of time steps to reach a given goal state from any other state which they call: dynamical distances. The motivation suggests that such an ability would help the agent in reaching new goal states and therefore in learning complex tasks. Task goal is given with a small amount of preference supervision. The claim is that dynamical distances can be used in a semi-supervised regime, where unsupervised interaction with the env is used to learn the dynamical distances. Evaluation is done on a real-world robot and in simulation, the method can learn to turn a valve with a real-world 9-DoF hand, using raw image observations and just ten preference labels, without any other supervision. The ideas presented in this work are interesting, but I have some major concerns (please see detailed comments below). The paper is well written and is mostly easy to understand. 
				
Detailed comments:
The learning process follows two steps: 
-distance evaluation step: learn a policy specific dynamical distance parameterized by \psi: by computing the expected # of time steps it took for \pi to reach the goal state.
-policy improvement step: use the learned distance fn to optimize a policy to reach the goal

One weakness of the proposed approach is that the supervised regression step requires an on-policy experience which can be very expensive as pointed out by the authors as well. The authors suggest that because they use this as an intermediate representation this does not impact learning. Wouldn’t it be possible to *simultaneously* learn the dynamical distance during the off-policy policy-learning step?

The second weakness of the proposed approach is that in the policy improvement step it is assumed that a goal state is given. It is also weird to use goals that are already reachable. Please provide clarifications. 

Algorithm 1 learns the distance corresponding to the current policy. Wouldn’t it be a more generalized approach to learn the dynamical distance irrespective of the policy i.e., not policy-specific? Or even better, would it be possible to instead learn the dd applicable to a class of policies? Assuming behaviourally similar policies can have the same dynamical distance? 

On an intuitive level, it would be useful for the reader to make concrete how is this approach different from learning goal conditioned policies? The dynamical distances reflect the distance to the goals: goals are either 1) sampled from an experience replay buffer, 2) given by user preference, 3) chosen more smartly by the dynamical distance learned.

It is not clear if in sec 4.3 “the optimal dynamical distance will be 2”, but since the Eq.2. use the expected distance, does the above statement hold?

Empirical analysis: The authors aim to address three questions through the experiments: Regarding Q2: Is DDL applicable to real-world, vision-based robotic control tasks? Fig 1 shows interesting tasks and a useful contribution. 

In terms of learning from human preferences; the experiments do not seem sufficient and need more comparative analysis. In particular, the authors should compare to other well-known techniques where learning from human preferences is performed such as [1], [2]. 

The same is the case with unsupervised skill acquisition, the authors only compared to Diayn. I would suggest comparing to other relevant baselines such as [3], [4], [5].  Fig 6 primarily suggests the obvious that Diayn maximizes for diversity while DDL maximizes for distance. Moreover, even this comparison seems incomplete as I am not sure why the top row in Fig 6 only shows the proposed approach and not the baseline (diayn). Unsupervised skill acquisition makes it even more important to understand what is the nature of skills learned. I find that analysis is missing here as well, and would add completeness to these results. 

Considering the paper’s claims are heavily based on the ability to learn skills, there is no qualitative analysis of what kind of skills are learned. For e.g. it is mentioned that DDL method that can learn a 9-DoF real-world dexterous manipulation task, but what are the exact skills learned here per task? It would be useful to do a more rigorous analysis of each skill learnt. “Videos of the learned skills can be found on the project website: https://sites.google.com/view/skills-via-distance-learning.” However, the link throws a 404 error.

Overall:		
The paper is well written, real-world experiments are a ++, interesting mix of ideas along with applicability to multiple problems. The main experiment is shown with preferences where comparisons do not seem enough and the paper could be made stronger by thorough comparisons. 

[1] Christiano, Paul F., et al. "Deep reinforcement learning from human preferences." Advances in Neural Information Processing Systems. 2017.
[2] Ashesh Jain, Brian Wojcik, Thorsten Joachims, and shutosh Saxena. Learning trajectory preferences for manipulators via iterative improvement. In NIPS, 2013.
[3] Gupta, Abhishek, et al. "Unsupervised meta-learning for reinforcement learning." arXiv preprint arXiv:1806.04640 (2018).
[4] Achiam, Joshua, et al. "Variational option discovery algorithms." arXiv preprint arXiv:1807.10299 (2018).
[5] Karol Gregor, Danilo Rezende, and Daan Wierstra. Variational Intrinsic Control. pages 1–15, 2016.

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper presents an approach to do reinforcement learning without reward engineering. Instead, steps to reach goals are used to update policy.

Here are some questions:
(1) The reason to use cumulative distance instead of greedily chooses the shortest distance is to avoid risky state. But it is also mentioned in the paper that the experiments in the paper all have deterministic dynamics (and one of the assumptions in Appendix B). Can the authors elaborate on this? For example, how is the performance of a policy that just directly minimizes the distance function? Instead of using the made-up toy example.

(2) For the locomotion task examples, I am confused as to why choosing the state with the highest immediate reward will be a good preference state. For example, a humanoid that falls forward can have high forward velocity while has no chance for recovery. While the performance seems reasonable, the systems presented are mostly stable systems (with the exception of hopper). It will be nice to show similar performance in more unstable systems, e.g, walker2d, humanoid.

(3) Some assumption in the proof in appendix B is not true. d(g, g) is not 0 for all dynamical systems. For example, a hopper with a state that has none zero forward velocity will take more than 0 steps to return to the same state.

(4) “Because the trajectories have a finite length, we are effectively ignoring the cases where reaching sj from si would take more than T − i steps, biasing this estimate toward zero, but since the bias becomes smaller for shorter distances.” I am confused about this sentence, what does it mean? For example, if the dynamical system is the hopper, (s_i, s_{i+1}) will make d(s_i, s_{I+1}) be 1, and for all we know d(s_{i+1}, s_i) can be huge. How does this sentence apply in this situation?

(5) An appendix with hyperparamters will be helpful. For example, how much data are needed for updating the distance function?

(6) “We also evaluated DDLUS on the InvertedDoublePendulumv2 domain, where the task is to balance a pole on a cart. As can be seen from Figure 6, DDLUS can efficiently solve the task without the true reward, as reaching dynamically far states amounts to avoiding failure as far as possible.” Why reaching dynamically far states is equivalent to avoiding failure? Shouldn’t the policy learn to stay as close to the initial state as possible?

Despite the questions raised above, the techniques are interesting, simple, and effective, as demonstrated on some image-based tasks on real robots. While I understand that this method will be very effective for relatively static tasks like the valve turning, I fail to see how it can apply in unstable locomotion tasks, as mentioned in some of the questions above. 

Normally I will give rating 5 to this paper, but the system doesn't give me this option, so I have to lower it to 3 instead. 
review_title: Official Blind Review #1

Meta-review: The authors present a method to learn the expected number of time steps to reach any given state from any other state in a reinforcement learning setting.  They show that these so-called dynamical distances can be used to increase learning efficiency by helping to shape reward.  After some initial discussion, the reviewers had concerns about the applicability of this method to continuing problems without a clear goal state, learning issues due to the dependence of distance estimates on policy (and vice versa), experimental thoroughness, and a variety of smaller technical issues.  While some of these were resolved, the largest outstanding issue is whether the proper comparisons were made to existing work other than DIAYN.  The authors appear to agree that additional baselines would benefit the paper, but are uncertain whether this can occur in time.  Nonetheless, after discussion the reviewers all appeared to agree on the merit of the core idea, though I strongly encourage the authors to address as many technical and baseline issues as possible before the camera ready deadline.  In summary, I recommend this paper for acceptance.
Decision: Accept (Poster)