Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper presents HTM, an extension of the semiparametric topological memory method that augments the approach with hallucinated nodes and an energy cost function. The hallucination is enabled by a CVAE, conditioned on an image of the environment, and allows the method to generalize to unseen environments. The energy cost function is trained as a contrastive loss and acts as a robustness score for connecting the two samples. The underlying graph is then used to plan for several top view planning problems.

The paper is well written and clear. I believe such latent representations are an interesting approach to solving visual navigation and general planning. HTM provides an interesting and useful extension to SPTM, allowing both generalization to unseen environments and a more robust loss function. The 

My primary concern is the lack of rigorous experimentation to validate the concept and push it’s limits. The results in Table 1 show HTM outperforms baselines clearly on the given problems, but how it performs on more complex problems is unclear. These problems are dynamically simple and the obstacles are easily identified. Some more difficult problems may be:
- The mazes in SPTM or environments from https://arxiv.org/pdf/1612.03801.pdf.
- The original SPTM paper focuses on visual navigation from first person views. How does this method apply to such situations? How does the context translate to this scenario?
- Planning in real environments with real images, as done in [6].

Other comparisons and notes:
- Can the method be applied to higher dimensional problems (dimensionality of the underlying space) where planning may be more difficult? E.g. SE(2), robot arms or other agents from UPN [26]. Application with actual 3D workspace problems too would be interesting as the image context may underspecify the environment.
- The energy cost function acts as a proxy for connection probability when traversing an edge. This may also be useful for dynamical systems (e.g., the mujoco ant navigating a maze). Are there limitations for the method on such problems, e.g., edges may no longer be symmetric?
- How does SPTM compare when the space has been explored already?
- Can more quantitative results been shown such as solution path cost?
- Provide definitions for Fidelity, feasibility, and completeness and the source of data (polling human’s) in the Table 1 caption.
- “As shown in Table 5.2, “, should be renamed to Table 1.

review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: The paper propose a novel visual planning approach which constructs explicit plans from "hallucinated" states of the environment. To hallucinate states, it uses a Conditional Variational Autoencoder (which is conditioned on a context image of the domain). To plan, it trains a Contrastive Predictive Coding (CPC) model for judging similarities between states, then applies this model to hallucinated states + start/end states, then runs Dijkstra on the edges weighted by similarities.

I vote for accepting this paper as it tackles two important problems: where to get subgoals for visual planning and what similarity function to use for zero-shot planning. Furthermore, the paper is clearly written, the experiments are well-conducted and analyzed.

Detailed arguments:
1. Where to get subgoals for visual planning is an important question persistently arising in control tasks. SPTM-style solution is indeed limited because it relies on an exploration sequence as a source of subgoals. Every time the environment changes, data would need to be re-collected. Getting subgoals from a conditional generative model is a neat solution.
2. Benchmarking similarity functions is crucial. One productive way to approach zero-shot problems is to employ similarity functions, but the question arises: what algorithm to use for training them? The paper compares two popular choices: CPC and Temporal Distance Classification (in particular, R-network). It thus provides guidance that CPC might be a better algorithm for training similarity functions.
3. The paper is well-positioned in the related work and points to the correct deficiencies of the existing methods. It also features nice experimental design with controlled complexity of the tasks, ablation studies and two relevant baselines.

I would encourage the authors to discuss the following questions:
1) Fidelity in Table 3 - why is it lower for SPTM compared to HTM if both methods rely on the same generated samples? Is it because HTM selects betters samples than SPTM for its plans?
2) Why is fidelity larger for SPTM in a more complex task 2?
3) Same question about fidelity/feasibility for HTM1/2?
4) Are there any plans to open-source the code?
review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: The paper presents a method for learning agents to solve visual planning, in particular to navigate to a desired goal position in a maze, with a learned topological map, i.e. a graph, where nodes correspond to positions in the maze and edges correspond accessibility (reachability in a certain number of steps). The work extends previous work (semi parametric topological memory, ref. [22]) in several ways. It claims to address a shortcoming of [22], namely the fact that the graph is calculated offline from random rollouts, by using a conditional variational auto-encoder to predict a set of observed images which could lie between the current position and the goal position, and, most importantly from a context image which describes the layout of the environment. These predicted images are then arranged in a graph through a connectivity predictor, which is trained from rollouts through a contrastive loss. Training is performed on multiple environments, and the context vector provides enough information for this connectivity network to generalize to unseen environments.  At test time, the agent navigates using a planner and a policy. The planner calculates the shortest path on a graph where edges are connectivity probabilities, and the policy is an inverse model trained on the output of the planner.

While the idea of a topological memory with dynamic graph creation is certainly interesting, the work is unfortunately not well enough executed and the paper structured written in a way which makes it up to impossible to grasp what has been really done, as much information is missing which would be required for understanding. 

As a first example, we are never really told what the observations are, which the agent sees. The different figures of the paper show very small images with a 2D maze from a bird’s eye view consisting of a walls arranged in a single connected component (mostly 1 to 3 strokes) in red color and an agent shown as a position indicated as a green dot. Are these the observed images? In absence of any other information, this is what we need to assume, and then this problem is fully observable and does not seem to be very challenging. Given the figures, even a handcrafted algorithm should be able to calculate the optimal solution with Dijkstra’s algorithm on a graph calculated from the pixel grid.

This important missing information alone makes it difficult to assess the paper, but the rest of the writing is similarly confusing. The authors focus on very short and dense descriptions of mathematical properties, but seem to have forgotten to ground the different symbols and to connect them to physical entities of the problem. The technical writing is in large parts disconnected from the problem which is addressed by it.

Further examples are:

-	“the data (…) is collected in a self-supervised manner”: what does this mean? Self-supervision is way of creating loss from data without labels, but I am not sure what is meant by collecting data this way.
-	The paragraph on CPC in section 2 can only be understood if the contrastive loss is known. To make the paper self-consistent, this should be properly explained, and tied to a training procedure which details how exactly the positive and negative samples are defined … and collected.
-	The CPC objective in section 2 is only loosely connected to its usage in section 3.2. Barely writing “we optimize a CPC” objective is not sufficient for understanding how this objective is really tied to the different entities of the problem. This paper contains maths (which is always a pleasure to read), but it is not a purely and abstract mathematical problem - a real task is addressed, so it needs to be connected to it. This connection has certainly been done by the authors while they were working on the problem, but they should also communicate it to the reader.
-	The section on ML trajectory is too dense and should be rewritten. I don’t understand what the authors want to tell us here. Basically, a (generalized) Dijkstra is run on a graph, where edge weights are the density or density ratios learned by the CPC objective, and if the edge weights are probabilities, that the shortest path corresponds to a trajectory likelihood. This is known, and this information is buried in a dense set of equations which are difficult to decipher and do not add any further value to the paper.
-	The connection between the planner (generalized Dijkstra) and the policy is never explained. We don’t know how the policy is trained and how it works.

One of the downsides of the method is that it requires a context image. This image is responsible for the generalization to unseen environments, but it is a major drawback, as the image must be created beforehand. The authors claim that the context image must only contain the layout in any format which makes it possible to extract information about navigational space from it, but in the experiments the context image corresponds to the full map – and it is probably equivalent to the observed images, but we can’t be sure as we haven’t been told. In any case, it is far from sure how this could generalize to more complex environments, let alone 3D navigation as is currently addressed in standard simulators like VizDoom, GIBSON, Matterport, Deepmind Lab, Habitat AI etc. 

The authors’ claim that the proposed environment requires long-term planning, but looking at the images this does not seem to be the case. 

The paper claims to perform zero-shot generalization and to adapt to changes in the environment, like the slight changes in camera motion, variations in lightning, but it unclear how the solution solves this claim.

How does the agent determine that a goal has been reached, without ground truth information? 

What happens, if the hallucinated images are disconnected (form several connected components) or are disconnected from the current position and/or from the goal position?

As mentioned, the method is evaluated on an environment, which is too simple. The experiments are difficult to assess, as we don’t really know what the agent observes. An information asymmetry is mentioned (visual foresight having the object’s (=agent’s) position and the others not) … but if the proposed method observes the bird’s eye view, it can infer the agent’s position (as the position of the green dot).

Subjective evaluation by humans on this kind of simple data does not seem to be meaningful, in particular with a very low number of observers (5 people).

review_title: Official Blind Review #1

Meta-review: The submission presents an approach to visual planning. The work builds on semi-parametric topological memory (SPTM) and introduces ideas that facilitate zero-shot generalization to new environments. The reviews are split. While the ideas are generally perceived as interesting, there are significant concerns about presentation and experimental evaluation. In particular, the work is evaluated in extremely simple environments and scenarios that do not match the experimental settings of other comparable works in this area. The paper was discussed and all reviewers expressed their views following the authors' responses and revision. In particular, R1 posted a detailed justification of their recommendation to reject the paper. The AC agrees that the paper is not ready for publication in a first-tier venue. The AC recommends that the authors seriously consider R1's recommendations.
Decision: Reject