Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: 
## Summary 

 - The authors propose a method for "zero-shot navigation" that learns to navigate mazes from a map of the maze and the start and goal location in the maze. 

## Strengths

 - This work builds on previous work in similar environment setups such as "Learning to Navigate in Complex Environments" (Mirowski et. al. 2017) to the "zero-shot" case, i.e., where the agent need not do any exploration when presented with a new environment but can immediately navigate to the goal. 

## Weaknesses

 - My major concern about this work is that I just don't understand why this is a hard problem. You argue that "one cannot simply perform graph search on the 2-D map" but I fail to see why. Any classical planning method would have no trouble generating a plan for agent navigation from the 2-D map provided. The problem then, it seems to me, is reconciling the plan on the 2-D map with the actions that are needed to be taken in the actual environment. But given that there is a deterministic scaling between occupancy grid map and the agent world ("Each cell on the abstract map corresponds to 100 units in the agent world.") AND your agent's transition function is deterministic, then this problem is solvable zero-shot with a traditional planner (e.g. one based that generates a roadmap and then searches it but others would work)  and then executing the plan. Learning the dynamic model is also straightforward because you assume that you can directly observe the the joint state $o_t$.  

 - Related to the above, you refer to the 2-D map as "abstract" but I fail to see what's abstract about it. It is a metric occupancy grid map that gives you full information about the layout of the environment. 

 - As a result, my impression is that the solution is "over-engineered". There are many complex components and "hyper" models when something much simpler would have easily solved this problem. 

## Minor Questions/Comments

 - At the onset you frame with this work with the classical SLAM literature, but this seems puzzling since you are providing the agent with both the map, and its initial location within the map, so both the mapping and localization are solved. 
review_title: ICLR 2021

final_score: rating: 4: Ok but not good enough - rejection
review_content: This submission tackles the Point Goal navigation task given access to the agent’s starting location, current state (position and velocity), the goal location and a top-down map. The submission presents two approaches for tackling this task. First is MMN (Map-conditioned Multi-task Navigator) which is model-based approach which learns to hyper-network to convert map input to a transition function. Second approach is MAH (Map-conditioned Ape-X HER DQN) which is model-free approach using Ape-X DQN with Hindsight experience replay.

The submission has several weaknesses:
- I believe the key challenge in solving the point goal navigation task given access to a top-down 2-D map is localization. However, the authors assume access to the agent’s position and orientation which makes localization trivial. This assumption simplifies the task a lot. In fact, I believe there’s a deterministic solution to this task which would achieve ~100% success rate. The starting location, the goal location, and the map are given as input to the agent. The agent can use any planning algorithm like Dijkstra to plan the path to the goal location. The only minor challenge is that the action space is different from moving cells on the map, but the sequence of actions required to go to the next map cell is easy to obtain given that the actions and transitions are deterministic and the agent has access to its location and velocity at all time steps.
- There are several existing methods for the PointGoal navigation task in unseen environments without having access to the map (for example [1-4]), which seem to achieve much better performance (90-99% success) in more realistic environments. One or more of these methods should be used as baselines. A lot of these works have also open-sourced the code.
- The submission uses mazes in the DeepMind Lab for evaluation. I believe the key challenge of understanding abstract maps is highly simplified in this environment. I am not convinced that performance in this environment is indicative of performance in realistic environments as realistic maps are more complex and very different from mazes. There are several more realistic simulation environments available for navigation such as Habitat [5], Gibson [6], AI2Thor [7], etc.
- One of the desirables when learning to navigate with access to a 2-D map as opposed to navigating without maps is efficiency. Although the authors report success trajectory length, I believe Success weighted by path length (SPL) is a more informative metric to measure efficiency. SPL is commonly used in navigation papers in recent years (for example, [1-5]) and it makes it easier to compare the performance of different methods.

It is very difficult to access the efficacy of the method since the task is simplistic and there exists a deterministic solution to the task which would achieve ~100% success rate. The submission also lacks competitive baselines.

Suggestions for improvement:
- The agent should not have access to the true pose and velocity. This assumption makes the task trivial in my opinion.
- The actions and motion should not be deterministic.
- Empirical evaluation in a more realistic simulation environment would be more indicative of the efficacy of the proposed method.

[1] Ramakrishnan et al. Occupancy Anticipation for Efficient Exploration and Navigation. ECCV, 2020

[2] Wijmans et al. DDPPO. ICLR, 2020

[3] Chaplot et al. Learning to Explore using Active Neural SLAM. ICLR, 2020

[4] Sax et al. Learning to Navigate Using Mid-Level Visual Priors. CoRL, 2019

[5] Savva et al. Habitat: A Platform for Embodied AI Research. ICCV, 2019

[6] Xia et al. Gibson env: real-world perception for embodied agents. CVPR 2018

[7] Kolve et al. AI2-THOR: An Interactive 3D Environment for Visual AI. 2017

[8] Anderson et al. On Evaluation of Embodied Navigation Agents. 2018

review_title: Assumptions make the task very simplistic

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper tackles the task of going to a point-goal using an abstract 2D map of a given environment. The central idea is to use the given abstract 2D map to predict parameters for the transition function in the environment depicted by the map. This predicted transition function is used to search for actions to execute via planning.

Strengths: The paper tackles an interesting problem, that of how to use an abstract 2D map to navigate. It proposes interesting neural architectures for solving this task.

Shortcomings:
1. While the paper uses 3D environments from DeepMind lab, actual experiments in the paper employ a 12D agent state as input to the policies. Thus, in my view, the paper is actually only solving a 2D problem (even though Fig 1 shows a first-person image). If my understanding about the problem setup is correct, I wonder how would a hand-crafted transition function based on the map do -- we already know where is the free space, the starting location, the goal location, and the fixed scaling between the actual space and the map, can't we build a conservative transition model based on this information? How will such a hand-crafted solution compare? Conceptually, what is the utility of learning on top of such a hand-crafted transition function?

2. I found it challenging to understand the precise model. Section 3.2 is somewhat confusing: are we learning a policy \pi_t that mimics the outcome of a monte-carlo tree search? How are the functions $h_\psi$ and $f_\phi$ trained? What are $W_\phi$ in Figure 2? Without these details it is hard to fully understand the proposed model.

In summary, I am not convinced of the experimental setup being used to study the proposed approach. The proposed approach hasn't been described clearly enough to allow for a proper review.

Update: I thanks the authors for preparing a response and for providing additional experiments. I believe these additional experiments will benefit the paper. However, these experiments will require a full re-evaluation of the paper which, in my view, is beyond the scope of a response to a rebuttal.
review_title: Concerns about task settings, lack of clarity in approach description

final_score: rating: 6: Marginally above acceptance threshold
review_content: [EDIT AFTER DICUSSIONS] I thank the authors for their answer to my comments. I agree with the summary of the Area Chair and do not wish to modify my score.
[/EDIT]

##########################################################################
Summary:
This paper presents addresses the problem of zero-shot navigation in environments with novel layouts.  It introduces two approaches (MMN a model-based approach based on Monte-Carlo Tree Search, and MAH, a model-free approach based on Deep-Q networks). The paper also introduces n-step relabelling as a way to leverage failed trials and make learning more efficient. Experiments on the DeepMind Lab environment show that both methods perform well against a random baseline and that MAH extends better to larger maps.

##########################################################################
Reasons for score: 
This paper presents a novel approach to an interesting problem. The method is sound and the approach rigorous. On the downside, an important claim of the paper is that the method is more generalizable than the latest work on this topic (Brunner et al. 2018) but this claim would have been stronger if supported with evidence, in particular with stronger baselines and more tasks.

 ##########################################################################
Pros:

- Clarity: the paper is well-structured, clear, and easy to read

- Impact: the paper addresses an interesting problem, in particular trying to use a general approach that is not specific to map-based navigation

- Rigor: the work presented in this paper is detailed and follows a clear methodology. Equations seem correct. The experimentation study is fairly detailed and the appendix provides significant details about the methods.

##########################################################################
Cons: 

- No code is provided with the paper, making it hard for future work to use it as a baseline

- The paper mentions Brunner et al. 2018 as reference work. This work seems to achieve significantly better performance, at the "cost" of using a more task-specific, map-based navigation approach.This raises a few questions:
1. Could the authors have used Brunner et al. as a baseline for this work? Was the code available?
2. The authors (rightfully) claim that their approach is more generic and may be more readily generalizable to other problems. This statement would be a lot stronger if they actually proved it in the paper, i.e. if they used the same technic to solve a different problem.
3. The methodology used in Brunner et al. uses a larger variety of map sizes. The authors could have used this approach too to better evaluate their method.

- The paper provides a simple external baseline (random).  Could other (external) stronger baselines have been used, such as asynchronous advantage actor-critic ((Mnih et al.,2016) or model-free episodic control (Blundell et al., 2016)?

- The paper evaluates on a single task and could have been evaluated on more tasks to illustrate robustness to domain shifts, in particular in the light of the comment above regarding the generalization of the method. Examples include Jaco Arm, CoinRun, or the Surreal framework.

#########################################################################
Some typos: 

- Page 2: "betweend" in the first paragraph
- Figures on page 6 are hardly legible on a printed version of the paper.  Try to make them larger maybe?
- missing upper cases in some references (e.g. POMDPs)

review_title: Interesting contribution to zero-shot navigation, falls a bit short on generalization

Meta-review: The paper considers the problem of 2D point-goal navigation in novel environments given access to an abstract occupancy grid map of the environment, together with knowledge of the agent's state and the goal location typical of point-goal navigation. The paper proposes learning a navigation policy in a model-based fashion, whereby the architecture predicts the parameters of the transition function and then uses this learned transition function to plan the agent's actions. The authors also describe a model-free approach that extends a version of DQN to reason over the 2D maps.

The paper was reviewed by four knowledgeable referees, who read the author response. The general problem of learning to navigate a priori unknown environments to reach a desired goal is an interesting problem that has received significant attention of-late in the learning community. In its current form, however, the paper does not adequately convey why this is a difficult problem that can not be solved using existing planning techniques or why it benefits from learning, particularly given access to an abstract map. These concerns apply more generally to point-goal navigation, namely the assumption that the pose of the agent and goal are fully known throughout (or the agent-relative pose of the goal) and that there is no uncertainty in the agent's motion. The practicality of these assumptions is unclear, and they are inconsistent with decades of research in robotics and robot learning, which addresses the more realistic setting in which there is uncertainty in pose and motion. The author response helps to clarify some of these questions, but it is still not fully clear why existing methods are insufficient for this task, whether they use traditional planning methods or are learned. Revisiting the discussion of why this is a hard problem would strengthen the paper, as would a more thorough evaluation that compares against other baselines.
Decision: Reject