Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper looks into offline data as the sole source of training actor policies for navigational agents. The proposed approach achieves this by a mix of learnable modules/networks and rule-based, heuristic ones. Moreover, the overarching goal is to reduce (to zero) the number of interactions in the simulator during training. The approach is compared with behavior cloning and end-to-end RL flat policy. The task chosen by the authors is image-goal navigation i.e. navigating to a specified image on data that they generate on the Habitat simulator.
main_review: **[Originality] (+) Model-based approaches are still under-explored for Habitat-like simulators:**
The rendering of RGB images is a computation bottleneck for current embodied AI research. While previous works have looked at distributed training (Wijmans et al. 2020), accelerating simulation (Savva et al. 2019), auxiliary losses (Ye et al. 2020), etc., this work investigates the promise of model-based approaches and with heuristic components.

**[Clarity] (+) The approach is clearly explained and figures are informative:**
The explanation of the approach is done in a clear top-down way which makes understanding quite easy. Relevant background from existing work is included, making the manuscript self-sufficient. Moreover, the figures and associated captions illustrate the approach and idea very effectively.

----

**[Originality] (-) The paper doesn’t connect/compare to research on ‘Offline Reinforcement Learning’ and ‘Learning from Demonstrations’ (or Offline IL):**

Offline RL, as thoroughly studied by multiple works, is aimed at utilizing passively collected offline data and needs no additional online interaction with the environment/simulator. Kindly refer to (Levine et al. 2020 arXiv:2005.01643) for a detailed tutorial, discussion of baselines, and open problems. While the submission is applying offline learning to visual navigation tasks with some existing neat modules on topological mapping and expansion, it fails to connect to the most relevant literature and claims offline learning in interactive environments as their key novelty. As mentioned in L245 “our key contribution is the ability to learn navigation agents from passive data.” This has been studied in existing work. Qualifying the claims of the work could be considered.

Particularly relevant are (a) Kahn et al. 2020 (BADGR) that runs on self-supervised off-policy data gathered in real-world environments needing ‘no simulation’ or human supervision; (b) Fu et al. 2020 (D4RL) that includes offline CARLA visual simulator for autonomous driving. Please see Levine et al. 2020 for more references on navigation and robot manipulation. The reward in a dataset of offline demonstrations could be any metric inversely proportional to the distance between states (or nodes, in the case of this submission). Also, the distance estimator $\mathcal{G}_{D}$ appears very similar to a critic estimating the value of a given state.

Additionally, the research in ‘learning from demonstrations/observations’ that learns from a mix of pre-collected demonstrations and interactions is relevant to the work. Hester et al. 2018 (DQfD),  Vecerik et al. 2017 (DDPGfD), and Nair et al. ICRA 2018 are relevant works for comparisons and grounding.

Considering domain shifts and using web/internet data as studied by Chang et al. [23] and also touched upon in Sec. 5.2 is a clear novel direction.

Q. The reviewer requests the authors to clearly relate and contrast their _key contribution_ to the offline RL/IL literature pointed above. Also, kindly highlight novelty differences from [23] (that tackle real-world youtube videos instead of traversals of simuators) beyond using memory in distance prediction (L78)

**[Originality] Request the authors to clearly distinguish what components in the _approach_ are novel.**

Discussing contributions beyond learning from passive offline data, several components in the approach are overlapping existing work on navigation in visual environments. The reviewer requests the authors clearly list the key contributions of the _approach_ with respect to [4,10]?

**[Quality] (-) Concern about benchmark chosen for evaluation:**
Since the demonstrations or training data used in the work is generated within the Habitat simulator, it’s straightforward to collect the same for the well-studied objectgoal or pointgoal navigation datasets. These tasks have been hosted within AIHabitat + (Gibson/MP3D scenes) with challenges run for 2 years at CVPR 2020 and 2021 workshops. Evaluating a new idea (of pure offline learning) on well-studied benchmarks and in the presence of reported results of previous baselines would add more credibility and also encourages the associated community to adopt this new idea. 

The reviewer shares the concern of using a (presently nonstandard) image-goal navigation task when it has no particular advantage for this study over the well-studied embodied tasks. While image-goal navigation task has been studied before by [4] and [15], both these worked introduce their own implementation of the task and their own data. This paper, again, introduces a new realization of the task and a new dataset. Hence, all the baselines reported in Tables 1 and 2 are implemented by the authors themselves. While introducing a dataset is a great feat, it would benefit from being grounded in performance to other datasets, accompanied by rigorous analysis, and tests for any data biases. These are missing in the submission. 

Q. The reviewer is happy to improve the rating if the authors can report results on well-studied benchmarks in the same Habitat simulator they use (examples: CVPR 20/21 pointgoal/objectgoal navigation challenges or RxR/VLNCE challenge) and compare to the metrics reported by previously published work on these well-studied benchmarks.

-----

**[Quality] Questions about training:**
* Q. **How are the predictions of the distance-prediction module made consistent with each other?** While $\mathcal{G}_D$ predicts distances of nodes to the goal, how is the triangle inequality enforced? Eg. $\mathcal{G}_D(\cdot, \cdot, n_1) = 3$; $\text{travelcost}(n1, n2) = 2$, then $\mathcal{G}_D(\cdot, \cdot, n_2) \leq 5$. 
* Q. **Could the authors include metrics for intermediate checkpoints to understand if the training of end-to-end RL baseline was close to saturation:** (L276) 20 Mn frames for PPO seems less given that 75Mn frames are needed for PPO to learn simpler PointGoal. 
* Q. **Fairness of comparisons in Tab. 1 and 2**. End-to-end RL sees 10 and 20 Mn samples/steps for Tab. 1 and 2 (that too from an impoverished exploration policy). The supplement suggests 10-15 and 20-25 epochs over 1 and 2.5 Mn steps from a perfect exploration policy. Could the authors comment about the comparison between these two approaches in terms of the number of experiences seen or the number of gradient updates made?
* Q. **Another way of no simulation**: Saving graphs with each pose was done in basic image-goal navigation in [11]. This too can train agents without simulating these observations. Could the authors comment on the disadvantages of this perfect model-based approach for training?

**[Clarity] Minor comments (no impact on reviewer rating, solely for constructive feedback):**
* Should the “navigate to sub-goal” step in Algorithm 1 have $\rho_{sg}$ and $\phi_{sg}$ instead of $\rho_{i}$ and $\phi_{i}$?
* Also in Alg. 1, "if $\eta_s$ then" suggests any non-zero probability leads to termination. Is there a threshold here?
* The grammar in L217 is off.
* Epochs of training should be included in L277.
* L139: subscript could be changed to ‘t’ to match Fig. 4’s notation.
* Figure 3: Notation for N is overloaded: -- the set of all nodes and the number of nodes.
* L52: check grammar in “predicts the if the goal”.
* Fig. 2: It could be clarified that “goal location” isn’t already known to the agent.
* Tab. 4: Caption “passive video data, on Image-Goal…” → “passive video data, and tested on Image-Goal…”

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper addresses image-goal navigation, where an agent is placed in a novel environment, and is tasked with navigating to an (unknown) goal position that is specified using an image taken from that position. The method builds a topological map and uses three learned networks for planning navigation. The first one, G_D, is a GNN which takes the current topological graph and goal image, and predicts distances to goal for all nodes. The second one predicts visual features for unexplored nodes in the graph. The third one, G_T, is a termination predictor which estimates probability of goal within sight, and its relative pose. Finally, the system uses non-learned components: shortest distance planner, a local heuristic navigation policy G_LP for reaching subgoals, a heuristic module G_EA for expanding graph with unexplored nodes. The experiments are conducted in 2 environments: Gibson and MP3D. The method shows improvement with respect to 2 BC and 1 RL baseline.
main_review: The scores below are consistent with "Rating" legend.

## Originality = 5/10
The proposed method is new. However, the paper would profit from a better embedding into the prior work. From the title "No RL, No Simulation: Learning to Navigate without Navigating" a reader could make a conclusion that it is the first time a passively learned navigation policy has shown success. However, there is prior work which already attempted to do this, albeit in a slightly more restricted setting. For example, SPTM [18] did learn its R and L networks passively, from data sampled by a random policy - so no RL-like feedback loop. It could also do it without a simulator, if it chose to have a local heuristic control policy instead of L-network. Of course, SPTM differs in test-time assumptions: it requires a short video of the test environment to begin with. This is the scientific "delta" of the proposed algorithm: it gets rid of this requirement by using a smart heuristic module G_EA for expanding the graph, and by a network which predicts visual features of newly expanded nodes. This is a nice research progress, but it requires a proper explanation of what is really new in this paper. If the authors choose to make such updates, I'm ready to increase my score.

## Quality = 7/10
The quality of experiments in this paper is high. There are enough experiments to support the main claims and there are exhaustive ablations to determine the value of each introduced component. Perhaps, one nice-to-have additional experiment would be to add a random walk baseline, and also SPTM + random-walk exploration.

## Clarity = 7/10
The paper is written in a nice and clear manner. A few questions though:
1. What are step limits for RL and BC?
2. [L111-112] "egocentric pose estimate" of what with respect to which origin?
3. [L166-167] "The local metric maps are discarded upon reaching the sub-goal" - how to determine that the subgoal is reached? What if it's not reached in reality? What if we think we reached on subgoal, but instead we reached another visually similar subgoal (i.e. perceptual aliasing)?
4. Could you please use bold font in tables to highlight the best number?

## Significance = 6/10
The submission does advance state-of-the-art by getting rid of annoying assumptions in prior work. The slight concern though is that the real frontier now lies more in real-world applicability of these algorithms than in conceptual improvements in simulated environments, because there have already been many navigation papers in simulation. "Scaling Local Control to Large-Scale Topological Navigation" https://arxiv.org/pdf/1909.12329.pdf scaled more restricted SPTM-like ideas to the real world. I would be curious to see if the ideas in this submission scale to the real world as well.
limitations_and_societal_impact: The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper introduces a new method for learning a navigation policy for indoor environments without RL and without simulation (which is, admittedly, fitting given the title). The method uses SLAM on passive video recordings to construct navigation graphs that are then used to train several networks including a goal-detection network that can be used in goal-based navigation.

I personally like the idea in this paper a lot and clearly, a lot of work went into the experiments and into the polish. That is why I'm upset that I have to give this paper a lower score than it deserves. The main thing that's hindering me from wholeheartedly recommending this is the lack of code and reproducibility. If code is not included, then I have to put the methods section under more scrutiny and there aren't enough details to reproduce this work. I'm happy to change my score if the authors address the issues listed below.
main_review: **Originality:** 

There are many works on using SLAM as part of an RL pipeline but all of these need to be trained interactively. Other works do indoor navigation from imitation and these, too, often need to be tuned via RL. The works that do learn from passive video, don't learn a distance estimator and only compare the current frame of a trajectory to the goal. Therefore I think the method presented in this work is sufficiently novel. 


**Quality:** 

The experiments sufficiently back up the method, the method is well-motivated, the ablations are insightful.
The only concerns I have with respect to quality are
1. The low performance of PPO - without listing the hyperparameters. I have used PPO for indoor navigation in Gibson for my own experiments and with a bit of tweaking, the performance can be brought up to a 10-15% success rate, especially on the "easy" task. I would recommend the authors spend a little more time with this baseline to establish a fair comparison.
2. Why is the agent with access to the metric map so bad? I don't think this was sufficiently justified in the paper. 
3. In the supplemental video, why does the second trajectory just fade out long before reaching the target? And the third trajectory seems to miss the target. I'm very confused by the video - it makes your method look significantly worse.


**Clarity:** 

As mentioned above, this is my main concern with this paper. Again, I'm only this strict because you didn't include any code with your submission. I can't accept any promise to submit the code later since I can't review non-existent future code. 
There are 4 subcomponents to your method,
- $G_D$, the global policy
- $G_{LP}$, the local policy
- $G_{T}$, the stop signal function
- $G_{EA}$, the explorable area function

$G_D$ is learned with a Graph Neural Network. There is a nice plot of the network in Figure 3 but is that all? What are the sizes of the GAT layers, their inputs and outputs? Is the MLP single-layer or multi-layer. You wrote "simple MLP" but what does that mean? Could we maybe get a network table where each layer is listed, together with their activation function and the number of parameters? And what exactly is the structure of the graph encoder? 
And for preparing the data, you wrote "First, we generate a stepwise trajectory based on odometry information from each frame." (line 210)... How exactly do you do that? What kind of SLAM system is employed here, and with which parameters? And likewise, "We adapt this trajectory to topological graph structure using affinity clustering on the stepwise graph via visual features and odometry information." - How? Which parameters, and which affinity clustering algorithm is used? Without this information, I cannot reproduce this.

$G_{LP}$ is "adapted from [10]" - but what does that mean? "Adapted" means "make (something) suitable for a new use or purpose; modify", so how was it modified to suit your needs?

"$G_{T}$ is simple MLP" (sic! line 186) - what does that mean? How many layers, activation functions, etc? Apparently, it has multiple output heads (for goal in sight and relative position of goal).

$G_{EA}$ - "This is untrained, heuristic function" (sic! line 175). It supposedly returns angles that don't have an obstacle within 3m. But what if there is a partial obstacle? Say half the frame has an obstacle, or a third or a fifth? What's the cutoff? Or what if the obstacle is 2m above ground like a doorframe? 

Also, from the text, line 148, you are using the visual features of unexplored nodes in your $G_D$. How does that work - which network generates visual features of unexplored nodes?

And in general, what are any of the loss functions you are training, what is your optimizer, what are the training parameters? There are superficial details in chapter 1 of the appendix, but the losses aren't included there and the detail there is not enough to warrant any reproduction.



**Significance:** 

I think if it were reproducible, this would be a significant contribution.

**Nitpicks and Questions:** 

1. The first third of the paper is well-written and mostly error-free. But as pointed out above, sections 4 and 5 are riddled with errors like missing articles and missing commas. Please proofread the document! For example lines 180, 228, 239, 259 twice, 279, 290, 305. Also, I believe that in general, if you write "x-based" like RL-based or Graph Neural Network-based, there needs to be a dash between "based" and the word before.
2. What's the PPO implementation and backend? I assume you're not using the standard Atari CNN, right?
3. In Table 1, for a 1.5-3m goal that's in a straight line from the agent, why is the performance so bad? 
limitations_and_societal_impact: The authors have neither assessed limitations nor possible societal impact in the main body of the paper.

Here are some ideas to discuss for limitations:
- The work is reliant on the video preprocessing. Right now this was demonstrated with video that's in-distribution and slightly out-of-distribution but not with video "in the wild". Given how sharply the performance drops when using non-Gibson data to train a Gibson agent, how would you expect this to perform with random youtube videos?
- Do you consider the task solved with a 68% success rate in straight 1.5-3m navigation tasks?

For societal impact: This work, like many others, enables indoor navigation agents. That could lead to the development of more capable robotic assistants in the long run, especially since this work aims to lower the bar for entry by removing simulation and RL.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work details a method for planning and navigation on the image-driven navigiation task in a photo-realistic simulator. The novelty of this approach is that the method is trained with sequences of (RGBD, pose) sampled from the environment, without the associated actions or rewards. The sequences are used to learn a predictive model of a distance function, which the authors claim can learn priors about the indoor layouts of indoor buildings, which enables the method to be transferred to buildings that were not observed during training.

main_review: The work presented is novel, of high quality and clearly written. The method is particularly of interest as it can be used to create a policy without the need to interact in a simulator, which means the method has the potential to scale to vast real world datasets. 
The method is compared against three baselines and outperforms them by a large margin. Detailed ablations are shown to break down the modular approach of this method.
Whilst the approach is novel, and outperforms the presented baselines by a large margin, I have a number of questions about the details of the implementation of the baselines.
 
Strengths:
1. The work outperforms baselines based on RL and behavioral cloning.
2. This approach decouples low-level acting and planning in interpretable manner .
3. The modular approach means that each component can be trained independently, which separates the problem into independant subproblems.

Weakness:

Baselines - I have a number of questions and concerns about the baselines:
0. It would have been good include a baseline with a policy of sampling actions randomly, in order to ensure the other baselines outperform purely random actions.
1. The baseline performs rather poorly in this setting, whilst the task is challenging I still find a success rate of 3.6% on goals within 1.5-3m suprisingly low.
2. How many seeds were evaluated when training the baseline method?
3. What is the reward function used for the RL method?
4. Input resolution: Given the instability and optimization challenge of training an RL method, an input resolution of 480x640 seems rather large, have the authors tested a down-sampled input?
5. On-policy Deep-RL methods such as PPO are known to be sample inefficient, do the authors consider the baseline to have converged after a "mere" 10M environment interactions?
6. Stopping criterion, it would be interesting to see comparisons of the baseline methods using the learned stopping criterion. As one reason an end to end RL method performs poorly in the setting could be related to the difficulty of exploring the action space when a STOP action is present, which presumable terminates the episode. In Neural Topological SLAM the approachs were compared with and without the need to a STOP actions. It would be good to have included this comparison.
7. Many recent works have shown scaling and distributed training of RL methods can outperform handcrafted learning based methods, for example [1] vs [2]. What is the authors opinion of scaling RL methods on a more complex task such as image-driven goal navigation?

Handcrafted learning based methods require time, tuning and domain expertise. How transferable is this approach to other tasks such as object goal navigation or coverage maximization?

For the local navigation policy, have the authors compared against an local pointgoal RL agent?

Missing references:
I draw the authors attention to [3] which is a recent related work on topological planning with graph neural networks in photorealistic envrionments. 

[1] DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames. Wijmans et al. ICLR 2020
[2] LEARNING TO EXPLORE USING ACTIVE NEURAL SLAM. Chaplot et al. ICLR 2020
[3] Learning to plan with uncertain topological maps. Beeching et al. ECCV 2020

UPDATE AFTER AUTHOR RESPONSE: Thank you to the authors for addressing my questions. I will leave my review at 7. 
limitations_and_societal_impact: One limitation is the requirement of pose estimates. The authors show a comparison where nose is added to the pose estimates. A more real world comparison where SLAM is used directly on the sequences to infer relative position and orientation would be interesting, but is perhaps beyond the scope of an already extensive conference paper.

A second interestesting extension would be sim2real transfer of policies trained with such a method.
review_title: null

Meta-review: This paper presents a method for learning robot navigation policies avoiding simulations and reinforcement learning.

The paper was initially on the fence, generally appreciated by reviewers but with one of the reviewers pointing several weaknesses, which were discussed with the authors and among the reviewers. The main problems were a complete omission of a discussion of offline RL, and the general responses to the reviewers ("please forget about this") was quite hand-wavy. However, the authors did provide convincing answers to many issues, and in particular explaining that their setting is sufficiently different from RL (no reward, most importantly). Of course, this still means that these parts should have covered it in the related works section, and I strongly urge the authors to modify the paper.

It was also discussed that several baselines are missing, in particular work on topological memory, and the standard ImageGoal environment was still considered a restriction.

Issues on presentation and reproducibility remained. 

However, there was a near consensus that the work is sufficiently novel and nicely executed.
I recommend acceptance.
Decision: Accept (Poster)