Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: The paper describes ANM, active neural mapping, to learn policies for efficiently exploring 3d environments. The paper combines classical methods with learning based approaches, allowing the final system to work competitively with raw sensory inputs without requiring unreasonable amounts of training samples.

I think this is a well-written "ML-systems paper" and I'm especially happy that real-world aspects of mobile robots are taken into account. I was able to follow the overall idea of the approach as well as the description of the three components. I also think that the experiments are well done, showing convincingly ANMs competitive performance and demonstrate, through the ablation studies, the importance of its constituting parts.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper proposes a new architecture and policy for coverage maximization (which the authors call exploration).  Overall the paper is well written, but I have some major concerns. However I am not an expert in navigation / robotics so i have given myself the lowest confidence for this paper.

My highest level concern is that this approach seems extremely complicated (eg Figs 1 and 2), as well as employing several sub-algorithms as part of the procedure (eg Fast Marching Method). It's not clear to me why any of the components are necessary, though I do appreciate the ablation study. But even within that ablation not all components are ablated (e.g., why GRU units?).  My experience suggests that extremely complicated architectures such as this one are brittle and don't generalize (and it goes against Sutton's 'bitter lesson'). The fact that the experiments are so small does not help. Perhaps more challenging domains would yield negative results. Further, how tuned are the baselines? And it seems that the baselines are general RL agents and not optimized for coverage maximization like this architecture. The authors say " We will also open-source the code", has this been done? Open-sourcing would help others reproduce the results since as it stands I think this is too complicated to be reproduced. The level of intricacy makes me think that perhaps this paper is more suited to a robotics conference.

Secondly, the paper mentions exploration a lot, but it's not clear to me how this is a principled exploration strategy. Exploration is not in fact defined as "visit as much area as possible" or "maximize the coverage in a fixed time budget", as the authors suggest. In fact the sentences "We follow the exploration task setup proposed by Chen et al. 2019 where the objective is to maximize the coverage in a fixed time budget. [The] coverage is defined as the total area in the map known to be traversable" appears twice in this manuscript. Exploration is better defined within the context of the explore-exploit tradeoff, whereby an agent must sometimes take sub-optimal actions in order to learn more about the environment in the hope of possibly increasing it's long-term return. Conflating 'coverage-maximization' and exploration is confusing. I think the paper should be rewritten to de-emphasize exploration and instead talk about coverage-maximization, which is more accurate.

"Exploration has also been studied more generally in RL for faster training (Schmidhuber, 1991)." I certainly would *not* cite Schmidhuber 91 as the canonical reference of exploration in RL. Far, far, more appropriate would be either the Sutton+Barto RL book (which doesn't do a great job covering exploration but is at least a decent overall reference) or the works of Auer 2002 and Jaksch et al 2010, and related papers. The Schmidhuber citation should be removed and replaced with a few that actually make sense in this context.

I don't understand how the goals (especially long-term) are generated and trained. Is the long-term goal trained using the reward signal? This is not properly explained.

"and summarize major these below" typo, probably should be themes or theses?

"agnet pose" typo.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The paper describes a method for visual robot navigation in simulated environments. In terms of overall objectives and targeted reasoning, the current approaches can be roughly divided into two groups: i) learning tasks requiring high-level reasoning for navigation involving the detection and discovery of objects and their affordances and eventually also requiring to process language input, and ii) simpler navigation task involving geometry and the detection of free (navigable space): point goal, maximizing coverage etc. The former target more complex problems but the agents are more difficult (currently up to impossible) to transfer to real environments, whereas the latter directly target problems which can currently realistically used in real world scenarios.

The paper is of the second group, and addresses one of the currently investigated problems in robot navigation and mapping, namely whether learned navigation is superior to traditional planning algorithms, and whether the two different approaches can be integrated. It proposes to separate the task into long-term and short-term goals, which is not new per se, but the proposed formulation is quite interesting. In particular, the integration of the “handcrafted” planar (front propagation) into the learned framework solves a couple of issues with sample efficiency of learned methods, while still keeping some flexibility of learning over the 100% traditional approaches.

I will be upfront – I already reviewed an earlier version of this paper for NeurIPS 2019, where this paper unfortunately did not pass. I was actually a favorable reviewer at this time and was defending it. The paper has been improved since and I would be happy to see it pass. I still have a couple of questions, some of which are similar to the ones I raised in the NeurIPS review (others have been addressed since).

While I do agree that the targeted tasks might be considered less exciting then tasks involving high level semantics, I do also think that these tasks are far from solved as soon as we try to implement them in real life scenarios. I do think that the proposed paper is an interesting step forward.

The advantages of the proposed method are “bought” with a couple of key design choices, in particular the handcrafted non-differentiable long-term path planner. The downside of this is that the loss signals can’t be backpropagated through the planner, which restricts the mapping module to very simple mapping information, basically free /navigational space. End-to-end training of navigation could in principle learn to map objects and affordances which are discovered through the task and not hardcoded or even learned with supervision, which also must be known in advance. This means that the contribution is limited to simpler navigational tasks like the tested exploration and PointGoal. In contrast, other work from the literature uses differentiable planners (eg cited CMP (Gupta et al 2017), using value Iteration Networks (cited Talmar et al. 2016) which allows fine-tuning.

The mapping network, which is learned with supervision, is a general encoder-decoder network which needs to translate from projective first person views to ego-centric bird’s eye views. It thus needs to learn projective geometry from data, although projective geometry could be used as structure for the network, given camera calibration, which has been done in other work:

-	Chen et al., 2019
-	Gupta et la., 2017
-	Henriques et al., 2018
And a couple of others.

Several improvements have been made since the NeurIPS submission, some of which I had addressed in my review. The experiments are quite convincing in their comparisons with the state of the art, in particular the generalization performances:
- generalization from Gibson (training) to Matterport (testing)
- generalization from exploration (training) to PointGoal (testing).
A couple of the results have been removed from the NeurIPS submission, unfortunately, I think they should be kept in.

I appreciated the realistic sensor model fitted to real data measured with a Locobot robot, and the ablation studies, which indicated the contributions of the different planner modules and of pose estimation. The role of the short term planner has been made clearer in the new paper.

I found it interesting that the stellar performance at the Habitat AI challenge was removed from the new paper – this method (or at least a preceding version) won the challenge. But I do understand that this choice was motivated by some remarks of the NeurIPS fellow reviewers regarding the simplicity of the PointGoal task of the challenge.

A couple of less positive aspects, and questions:

On the downside, and following the remarks on literature above, I still think that the results should be compared with CMP, the main competitor of this method. 
I think this is the main short coming of the paper, in particular since CMP is able to perform end-to-end training because the planner is differentiable (value iteration networks, NIPS 2016).

The literature w.r.t. to hierarchical planning is very far from exhaustive and lots of work is missing, consisting of recent work 

Embodied Question Answering, Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, Dhruv Batra, CVPR 2018 
(and several follow up papers)

but also quite classical work like the literature around the options framework, with the following starting point:

R.S. Sutton, D. Precup, and S. Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1):181–211, 1999.

And many other papers.

The figures 1 and 2 have been completely redone, but they are not completely clear. In particular, several intermediate representations/maps/Images are not commented or labeled, they should be annotated with the symbols from the text. 

The role of the sensor output is not clear. Sensors normally provide relative positions … but the text seems to indicate absolute pose. Some details are lacking.

In “… to predict the pose change between the two maps …” it is unclear what is done here. Is this self-supervision?

The authors mention that unexplored area is considered as free space for planning. What consequences did this have in case of unexplored obstacles? I guess the problem was delegated to the local policy, which needed coping with these issues?

The last paragraph before the conclusions briefly mentions experiments and comparisons but without giving any details. This is unfortunate, since there is still space available (the paper length is 8.5 pages).

review_title: Official Blind Review #2

Meta-review: The paper presents a method for visual robot navigation in simulated environments. The proposed method combines several modules, such as mapper, global policy, planner, local policy for point-goal navigation. The overall approach is reasonable and the pipeline can be modularly trained. The experimental results on navigation tasks show strong performance, especially in generalization settings. 
Decision: Accept (Poster)