Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper presents for learning to visually navigate using a topological map is presented. The method combines an algorithmic approach to generate topological graphs, which is indexed by observations with Dijkstra's algorithm to determine a global path from which a waypoint observation is selected. The waypoint along with the current observation is fed to a learned local planner that transitions the agent to the target waypoint. A learned observation similarity mapping localizes the agent and indexes targets in the graph.

The novelty of the approach in context of prior visual navigation and landmark-based robotics research is the hybrid algorithmic and learned approach that builds a graph purely from observations without ego motion or direct state estimation. Most of the individual components have previously appeared in the literature including graph search (also roadmap methods), learned similarity metrics, and learned observation-based local planners. However, the combination of these approaches along with some of the presented nuanced enhancements including the connectivity shortcuts (a simple form of loop closure) to visual topological navigation and are compelling contributions. The results while not thorough do support the ability of the method effectively plan on novel and unseen environments.

The approach has potential limitations including the linear growth in the size of the memory, and it is unclear how the method handles degenerate observations (e.g., similar looking hallways on opposite sides of the environment). The authors should consider including examples or analysis illustrating the method’s performance in such scenarios.

The impact of the proposed approach would be better supported if compared against stronger baselines including recent research that address issues with learning long sequences in RNNs. Furthermore, additional experiments over a greater number and more diverse set of environments along with additional results showing the performance of the method based on variation environment parameters including number of exploration steps and heuristic values would help the reader understand the sensitivity and stability of the method.

The work in “Control of Memory, Active Perception, and Action in Minecraft” by Oh et al. have a learned memory that is used recall previous observations for planning. This method’s memory architecture can be considered to be nonparametric, and, while different from the proposed method, this similarity merits additional discussion and consideration for empirical comparison. 

Some of the the details for the baseline methods are unclear. The reviewer assumes that when the authors state they use the same architecture as Mnih et al. and Mirowski et al. that this also includes all hyper parameters including the size of each layer and training method. However, Mirowski et al. use RGB while the stated baseline is grayscale. 

The reviewer wonders whether the baselines may be disadvantaged compared to the proposed method. The input for the baselines are restricted to a 84x84 input image in addition to being grayscale vs the proposed methods 160x120 RGB image. It appears the proposed method is endowed with a much greater capacity with a ResNet-18 in the retrieval network compared to the visual feature layers (two layers of CNNs) of the baseline networks. Finally the proposed method is provided with demonstrations (the human exploration) that effectively explore the environment. The authors should consider bootstrapping the baseline methods with similar experience. 
review_title: Interesting approach with compelling results but is contrasted against weaker baselines 

final_score: rating: 7: Good paper, accept
review_content: *** Revision: based on the author's work, we have switched the score to accept (7) ***

Clever ideas but not end-to-end navigation.

This paper presents a hybrid architecture that mixes parametric (neural) and non-parametric (Dijkstra's path planning on a graph of image embeddings) elements and applies it to navigation in unseen 3D environments (Doom). The path planning in unseen environments is done in the following way: first a human operator traverses the entire environment by controlling the agent and collecting a long episode of 10k frames that are put into a chain graph. Then loop closures are automatically detected using image similarity in feature space, using a localization feed-forward ResNet (trained using a DrLIM-like triplet loss on time-similar images), resulting in a topological graph where edges correspond to similar viewpoints or similar time points. For a given target position image and agent start position image, a nearest neighbor search-powered Dijkstra path planning is done on the graph to create a list of waypoint images. The pairs of (current image, next waypoint) images are then fed to a feed-forward locomotion (policy) network, trained in a supervised manner.

The paper does not discuss at all the problems arising when the images are ambiguous: since the localisation network is feed-forward, surely there must be images that are ambiguously mapped to different graph areas and are closing loops erroneously? The problem is mitigated by the fact that a human operator controls the agent, making sure that the agent's viewpoint is clear, but the method will probably fail if the agent is learning to explore the maze autonomously, bumping into walls and facing walls. The screenshots on Figure 3 suggest that the walls have a large variety of textures and decorations, making each viewpoint potentially unique, unlike the environments in (Mirowski et al, 2017), (Jaderberg et al, 2017) and (Mnih et al, 2016).

Most importantly, the navigation is not based on RL at all, and ignores the problem of exploration altogether. A human operator labels 10k frames by playing the game and controlling the agent, to show it how the maze looks like and what are the paths to be taken. As a consequence, comparison to end-to-end RL navigation methods is unclear, and should be stressed upon in the manuscript. This is NOT a proper navigation agent.

Additional baselines should be evaluated: 1) a fully Dijkstra-based baseline where the direction of motion of the agent along the edge is retrieved and used to guide the agent (i.e., the policy becomes a lookup table on image pairs) and 2) the same but the localization network replaced by image similarities in pixel space or some image descriptor space (e.g., SURF, ORB, etc…). It seems to me that those baselines would be very strong.

Another baseline is missing: (Oh et al, 2016, “Control of Memory, Active Perception, and Action in Minecraft”).

The paper is not without merit: the idea of storing experiences in a graph and in using landmark similarity rather than metric embeddings is interesting. Unfortunately, that episodic memory is not learned (e.g., Neural Turing Machines or Memory Networks).

In summary, just like the early paper released in 2010 about Kinect-based RGBD SLAM: lots of excitement but potential disappointment when the method is applied on an actual mobile robot, navigating in normal environments with visual ambiguity and white walls. The paper should ultimately be accepted to this conference to provide a baseline for the community  (once the claims are revised), but I street that the claims of learning to navigate in unseen environments are unsubstantiated, as the method is neither end-to-end learned (as it relies on human input and heuristic path planning) nor capable of exploring unseen environments with visual ambiguity.
review_title: Exciting idea but disappointing implementation (revised)

final_score: rating: 3: Clear rejection
review_content: The paper introduces a graph based memory for navigation agents. The memory graph is constructed using nearest neighbor heuristics based on temporal adjacency and visual similarity. The agent uses Dijkstra's algorithm to plan a a path through the graph in order to solve the navigation task. 

There are several major problems with this paper. My overall impression is that the the proposed agent is a nearly hard-coded solution (which I think might be the correct approach to such problems), but a poorly implemented one. Specific points: 1-There are only 5 test mazes, and the proposed agent doesn't even solve all of them. 2-The way in which the maze is traversed in the exploration phase determines the accuracy of the graph that is constructed (i.e. traversing each location exactly once using a space-filling curve). 3-Of the two heuristics used in Equation 1 how many edges are actually constructed using the visual similarity heuristic? 4-How does the visual similarity heuristic handle visually similar map elements that correspond to distinct locations? 5- The success criteria of solving a maze is arbitrarily defined -- why exactly 2.4 min?   
review_title: Right approach -- poorly executed 

Meta-review: Important problem (navigation in unseen 3D environments, Doom in this case), interesting hybrid approach (mixing neural networks and path-planning). Initially, there were concerns about evaluation (proper baselines, ambiguous environments, etc). The authors have responded with updated experiments that are convincing to the reviewers. R1 did not participate in the discussion and their review has been ignored. I am supportive of this paper. 
Decision: Accept (Poster)