Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: The paper aims at learning a latent representation of images in the setting of sequential decision making. The representation are learned such that there exists a metric that correctly reflects the difference in reachability between points in the neighborhood of the current observation and the goal.

I'm struggeling to understand the core approach of the paper. More specifically, while learning the local metric (Alg. 1) seems clear, I can not understand the details of Alg.2 (which btw. is never referenced in the text). The surrounding paragraph is not detailed enough. Why is \Phi(x, x') denoted a global embedding? \Phi has two inputs, shouldn't that be some sort of a metric? How is "find n" done? There is a lot of talk about embeddings, but they are actually not occuring in section 3. What is a 'plannable representation'?

Some of the experiments compare to VAE and its learned embedding space. Shouldn't the comparision be to models that think about the Riemannian geometry of a VAE, e.g. "Latent space oddity: on the curva-ture of deep generative models". There are a several citations missing in that direction, going back to at least "Metrics  for  probabilistic geometries" by Tossi et al. in 2014. As it was also pointed out in a public comment, relevant citations related to "Learning for planning" seem to be missing, too. Finally, a wider set of experiments needs to demonstrate the method (again, considering the scope of the as-of-now-not-cited papers).
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: ## Paper Summary

While cast slightly differently in the intro, it seems to me that this paper learns a goal-conditioned value function that is used at test time to construct visual plans by selecting an appropriate sequence of data points from the training data. Similar to prior work they learn a local distance metric without supervision using a temporal proximity objective and then construct a graph of the training data points using this metric. The main novelty that this paper introduces seems to be the idea to distill the results of planning algorithms run at training time into a global, goal-conditioned value function, which allows to reduce the required planning time at test time. The authors perform experiments on constructing visual plans for a simulated toy navigation task, a robotic rope manipulation task and the StreetLearn navigation task. The paper reports favorable results under a time-constrained test setting but does not include strong baselines that were designed for this setting. 

## Strengths

- bootstrapping a learned local distance metric to a global distance metric to reduce test-time planning cost is an interesting problem
- the paper has nice visualizations / analysis on the toy dataset
- the learning procedure for the local distance metric is clearly described
- the paper uses a large variety of different visualizations to make concepts and results clearer 

## Weaknesses

(1) missing links to related work: the author's treatment of related work does not address the connections to some relevant papers (e.g. [1-3]) or is only done in the appendix (especially for [4]). It is not clearly delineated between techniques and ideas that are introduced in other papers (see (2) below) and the novel parts of this work. This makes it hard to understand the actual contributions of this paper. 

(2) only minor contribution: the core parts of this paper build heavily on prior work: time-contrastive objectives for distance learning have been introduced in [5] and also been used in a very similar setup as here in [4], further [4, 3] also use semi-parametric, graph-like representations for planning with a learned local distance metric. The major contribution seems to be to distill the plans derived with either (a) n-step greedy rollout or (b) Dijkstra graph-search into a value-function so that planning does not need to be performed at test time. This somewhat small contribution is in contrast to the claims from the introduction that this paper "pose[s] the problem of unsupervised learning a plannable representation as learning a cognitive map of the domain".

(3) comparison to weak baselines: the main comparison in the experimental section is to a version of [4] where the authors constrain the planning horizon to a single step, which means effectively greedily using the local metric from Sec. 3.1. To be clear: this is in no way the method of [4]: they use Dijkstra-based planning at test time and it is clear that a "version" of [4] that does not use planning is not able to work. To me this seems rather like an ablation of the proposed method than a real baseline. The baseline that plans greedily with embeddings based on visual similarity has even less hope of working. The paper lacks thorough comparison to (a) baselines with the same semi-parametric structure that perform planning at test time (like the real method of [4]) and (b) methods that generate reactive policies without constructing a semi-parametric memory (e.g. off-policy RL). Only then a thorough comparison of pros and cons of planning at training/test time is possible (see detailed suggestions below).

(4) lack of qualitative samples for generated plans: for both the rope and the StreetLearn domain the authors do not provide thorough evaluation. For the rope domain only a single qualitative rollout is shown, for the StreetLearn domain no qualitative samples are provided for either the proposed method or the comparisons. (see suggestions for further evaluation below)

(5) explanation of core algorithmic part unclear: the explanation of how the local metric is used to learn the global value function is somewhat unclear and the used notation is confusing. Key problems seem to be the double-introduction of symbols for the local metric in Alg. 2 and the confusing usage of the terms "global embedding" and "value function" (see detailed questions below) 

(6) terms used / writing structure makes paper hard to follow: the connection between used concepts like "global embedding", "plannable representation" and "goal-conditioned value function" are not clear in the writing of the paper. The authors talk about concepts without introducing them clearly before (e.g. problems of RL are listed in the intro without any prior reference to RL).

(7) lacks detail for reproducing results: the paper does not provide sufficient detail for reproducing the results. Neither in the main paper nor in the appendix do the authors provide details regarding architecture and used hyperparameters. It is unclear what policy was used to collect the training data, it is unclear how the baselines are working in detail (e.g. how the 1-step planning works) and how produced plans are checked for their validity.


## Questions

(A) What policy is used to collect the training data on each environment?
(B) What is the relation between the "global embedding" \Phi and the "goal-conditioned value function" V_\Phi(x, x_prime) in Algorithm 2? 
(C) What is the difference between the local metric function \phi and the reward function in Algorithm 2? Are they the same?
(D) If they are the same, how can the local metric accurately estimate rewards for states x and x_g that are far apart from one another as would naturally be the case when training the value function?
(E) What does the notation N(1, \eps) in line 5 of Algorithm 2 mean?
(F) What is the expectation over the length of trajectories between start and goal on the StreetLearn environment (to estimate what percentage of that the success horizon of 50 steps is)?


## Suggestions to improve the paper

(for 1) please add a more thorough treatment of the closest related works on semi-parametric memory + learned visual planning + learned distance functions (some mentioned below [1-5]) to the main part of the paper, clearly stating differences and carving out which parts are similar and where actual novelty lies.

(for 2) please explain clearly the added value of distilling the training-plans into a value function for O(1) test-time planning and point out that this is the main difference e.g. to [4] and therefore the main contribution of the paper. 

(for 3) in order to better understand the trade-offs between doing planning at test time (like [3,4]) or learning an O(1) planner contrast runtime and performance of both options (i.e. compare to the proper method of [4]). This will help readers understand how much speed they gain from the proposed method vs how much performance they loose. It might also make sense to include an off-policy RL algorithm (e.g. SAC) that uses the local metric as reward function (without constructing the graph) to investigate how much planning via graph-search can help at training time. Another interesting direction can be to investigate the generalization performance to a new environment (e.g. new street maze, new rope setup) after training on a variety of environment configurations. [3] showed that explicit test-time planning performs better than "pure" RL, it would be interesting how the proposed "hybrid" approach performs.

(for 4) please add randomly sampled qualitative results for both environments and all methods to the appendix. It can additionally be helpful to add GIFs of executions to a website. It might also be interesting to add a quantitative evaluation for the plans from the rope environment as was performed in Kurutach et al. 2018.

(for 5) please incorporate answers to questions (B-E) into the text in Sec 3.2 explaining Algorithm 2. It might also help to structure the text in such a way as to follow the flow of the algorithm.

(for 6) restructure and shorten the introduction, clarify terms like "inductive prior within image generation" or "non-local concepts of distances and direction" or "conceptual reward" or "planning network", clarify how the authors connect the proposed representation learning objective and RL. Avoid sentences that are a highly compressed summary of the paper but for which the reader lacks background, like in the intro: "training a planning agent to master an imagined “reaching game” on a graph".

(for 7) add details for architecture and hyperparameters to the appendix, add details for how baselines are constructed to the appendix. add details about data collection and evaluation for all datasets to the appendix (e.g. how is checked that a plan is coherent in StreetLearn). It might also help to add an algorithm box for the test time procedure for the proposed method.


## Minor Edit Suggestions
- Fig 2 seems to define the blue square as the target, the text next to it describes the blue square as the agent, please make coherent
- for Fig 7: the numbers contained in the figure are not explained in the caption, especially the numbers below the images are cryptic, please explain or omit


[Novelty]: minor
[technical novelty]: minor
[Experimental Design]: Okay
[potential impact]: minor

################
[overall recommendation]: weakReject - The exposition of the problem and treatment of related work are not sufficient, the actual novelty of the proposed paper is low and the lack of comparison to strong baselines push this paper below the bar for acceptance.
[Confidence]: High


[1] Cognitive Planning and Mapping, Gupta et al., 2017
[2] Universal Planning Networks, Srinivas et al., 2018
[3] Search on the Replay Buffer: Bridging Planning and Reinforcement Learning, Eysenbach et al., 2019
[4] Semi-Parametric Topological Memory for Navigation, Savinov et al., 2018
[5] Time-Contrastive Networks, Sermanet et al., 2017


### Post-rebuttal reply ###
I appreciate the author's reply, the experiments that were added during the rebuttal are definitely a good step forward. The authors added comparison to a model-free RL baseline as well as proper comparison to a multi-step planning version of SPTM. However, these comparisons were only performed on the most simple environment: the open room environment without any obstacle. These evaluations are not sufficient to prove the merit of the proposed method, especially given that it is sold as an alternative to planning methods. The method needs to be tested against fair baselines on more complicated environments; the current submission only contains baselines that *cannot* work on the more complicated tasks. I therefore don't see grounds to improve my rating.
review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: The paper proposes to learn a representation space that is useful for planning. This is done by a 2-step process: (1) learn a metric to reflect locality, (2) utilize that metric to learn a global metric by utilizing UVFAs from Reinforcement Learning literature. While the idea may be an interesting direction to improve sample efficiency of planning, it is not the first work that proposes to combine planning with representation learning, and I do not think the work is clearly presented/motivated/validated sufficiently to deem it acceptable.

As presented, there are many details that are unclear or imprecise. I list them below.

Feedback:
(1) The contrast between learning playable representations vs. planning via representation learning (as proposed) is  poorly motivated. The presentation ignores well recognized works in the latter in terms of UPNs [1], VINs [2].
(2) The RL background section is poorly presented -- as written Equation (4) seems incorrect. Further, is a dynamics model learned to be able to utilize the correct form of Equation (4)? It is never specified. How exactly is a plan extracted for any of the experiments?
(3) Possibly wrong citation for Equation (2) -- maybe cite the Oord et. al. paper? Further, c vs C -- the notation in general in the paper is inconsistent and poor.
(4) Algorithm 1, line 4 -- typos for x. Further for Contrastive learning methods to be effective, the negative sampling needs to be much higher. What ratio of samples is used for local metric learning?
(5) If an actual forward dynamics model is not used (Section 3 end), then as given in Algorithm 2 from pseudocode I'm completely unsure how a plan is extracted -- How is the UVFA used to extract a plan? What is N(1,\epsilon)? P-norm?
(6) The Dijkstra psuedocode is rather incomplete.
(7) How is sampling done with the local metric function? How is a plan generated? These important details are missing.
(8) Figure 4 is poorly presented/annotated. In the description "Plan2Vec correctly stretched out learned.." -- no it doesn't, visually it seems wrapped too. 
(9) There exists literature in RL to combine the 2 step metric learning process to 1 step. This is relevant. [3].
(10) What is the action space of these domains? Based on visualization in Figure 3 (2), are the actions continuous? What is the action space for Figure 7? These details details are missing.
(11) Description of the StreetLearn dataset would be useful. Further an example of why Plan2Vec generalizes (last para, Section 4.3) would be useful. Just statement based claims seem rather vacuous.
(12) The last statement in Conclusion - why? The paper has made an argument against utilizing generative modelling in an unsupervised manner. So why would including it improve it? Such unexplained statements reflect poorly.

Questions:
(1) What do you see as the contribution of the work? Why is it new/different from existing literature?
(2) How exactly do you generate a plan?
(3) How is the UVFA V used?
(4) When to use UVFA? When to use Dijkstra? Why are the choices in the experiments as made?


Some typos to help future version of the paper:
(1) Section 2 -- We now overview --> We now review.
(2) Section 2, UVFAs -- expected discounted future value --> expected sum of discounted future rewards.
(3) Section 2 completely ignores the discount factor/horizon of an MDP, although the utility here I suppose relies on the horizon aspect.
(4) Figure 6 explanation is very sloppy (description and body).
(5) The Zhang et. al. reference in Section 4.2 is unclear.

While I am not from the planning community, I am from the RL community - and as presented the paper is ignoring a lot of details, and was extremely difficult to piece together for me.

[1] Srinivas, Aravind, et al. "Universal planning networks." arXiv preprint arXiv:1804.00645 (2018).
[2] Tamar, Aviv, et al. "Value iteration networks." Advances in Neural Information Processing Systems. 2016.
[3] Wu, Yifan, George Tucker, and Ofir Nachum. "The laplacian in rl: Learning representations with efficient approximations." arXiv preprint arXiv:1810.04586 (2018).

review_title: Official Blind Review #3

Meta-review: The paper proposes a representation learning objective that makes it 
amenable to planning, 

The initial submission contained clear holes, such as missing related work and only containing very simplistic baselines. The authors have substantially updated the paper based on this feedback, resulting in a clear improvement.

Nevertheless, while the new version is a good step in the right direction, there is some additional work needed to fully address the reviewers' complaints. For example, the improved baselines are only evaluated in the most simple domain, while the more complex domains still only contain simplistic baselines that are destined to fail. There are also some unaddressed questions regarding the correctness of Eq. 4. Finally, the substantial rewrites have given the paper a less-than-polished feel.

In short, while the work is interesting, it still needs a few iterations before it's ready for publication.
Decision: Reject