No RL, No Simulation: Learning to Navigate without Navigating
Authors: Meera Hahn
Devendra Singh Chaplot
Shubham Tulsiani
Mustafa Mukadam
James Matthew Rehg
Abhinav Gupta
Venue: NeurIPS 2021 Poster (2021)
Keywords: Navigation
Simulation
Graph Neural Networks
Embodied AI

Abstract: Most prior methods for learning navigation policies require access to simulation environments, as they need online policy interaction and rely on ground-truth maps for rewards. However, building simulators is expensive (requires manual effort for each and every scene) and creates challenges in transferring learned policies to robotic platforms in the real-world, due to the sim-to-real domain gap. In this paper, we pose a simple question: Do we really need active interaction, ground-truth maps or even reinforcement-learning (RL) in order to solve the image-goal navigation task? We propose a self-supervised approach to learn to navigate from only passive videos of roaming. Our approach, No RL, No Simulator (NRNS), is simple and scalable, yet highly effective. NRNS outperforms RL-based formulations by a significant margin. We present NRNS as a strong baseline for any future image-based navigation tasks that use RL or Simulation.

No RL, No Simulation: Learning to Navigate without Navigating
Meera Hahn Georgia Institute of Technology
Devendra Chaplot Facebook Al Research
Shubham Tulsiani Facebook Al Research
Mustafa Mukadam Facebook Al Research
James M. Rehg Georgia Institute of Technology
Abhinav Gupta
Facebook AI Research
Abstract
Most prior methods for learning navigation policies require access to simulation environments, as they need online policy interaction and rely on ground-truth maps for rewards. However, building simulators is expensive (requires manual effort for each and every scene) and creates challenges in transferring learned policies to robotic platforms the real-world, due to the sim-to-real domain gap. In this paper, pose simple question: Do we really need active interaction, ground -rutt maps or even reinforcement-learnnng (RL) in order to solve the image-goal navigation task? We propose self-supervised approach to learn to navigate from only passive videos of roaming. Our approach, No RL, No Simulator (NRNS), is simple and scalable, yet highly effective. NRNS outperforms RL-based formulations by significant margin. We present NRNS as a strong baseline for any future image- based navigation tasks that RL Simulation.
Introduction
In recent years, have significant advances in learning based approaches for indoor naviga- tion [1. Impressive performance gains have been obtained for a range tasks, from non-semantic point-goal navigation 3 semantic tasks such as image-goal 4 and object-goal navigation (5] [)
via methods that use reinforcement learning (RL). The effectiveness of RI. for these tasks can be attributed in part the emergence of powerful new simulators such Habitat (7]. Matterport [8] and A12Thor [9]. These simulators have helped scale learning to billions of frames by providing large-scale active interaction data and ground-truth maps for designing reward functions. But do we actually need simulation and RL to learn to navigate? Is there an alternative way to formulate the navigation problem, such that no ground-trute maps or active interaction are required? These are valuable questions to explore because learning navigation in simulation constrains the approach to limited set of environments, since the creation of 3D assets remains costly and time-comeen consuminnn
In this paper, we propose a self-supervised approach to learning how to navigate from passive egocentric videos. Our novel method is simple and scalable (no simulator required for training), and at the same time highly effective, as it outperforms RL-based formulations by a significant margin. To introduce our approach, let first examine the role of RL and simulation in standard navigation learning methods. In the standard RL formulation, an agent gets reward upon reaching the goal, followed by a credit assignment stage determine the most useful state-action pairs. But do actually need the reinforce function for action credit assignment? Going step further, do we even need learn a policy explicitly? navigation, we argue that the state space itself is highly structured via a distance function, and the structure itself could be leveraged for credit assignment. Simply put, states that help reduce the distance to the goal are better and therefore predicted distance can be
"Correspondence: meerahahn @ @gatech.edu
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Training From Videos
Testing on Image- Goal Navigation
Pessive Trajectory Video VV
to Predict Distance
Video Graph G
FL
Sampled Curren State
Sampled Goal
Goal Image
Chosen Goal
Figure 1: Left: Using passive videos we learn to predict distances for navigation. Our distance function learns the priors of the layouts of indoor buildings to estimate distances to goal location. Right: Image-Good Goal Navigation Task [T0]. Our model uses distance function to predict distances of unexplored nodes and uses greedy policy to choose the shortest-distane node.
used either the value function reduced to goal' in reward shaping. The key property of our approach is that we learn a generalizable distance estimator directly from passive videos, and as result we do not require any interaction. We demonstrate that an effective distance estimator can be learned directly from visual trajectories, without the need for an RL policy to map visual observations the action space, thereby obviating the need for extensive interaction in simulator and hand-designed rewards. However passive videos do not provide learning opportunities for obstacle avoidance since they rarely, if ever, consist of cameras bumping into walls. We forgo the need for active interaction to reason about collisions as we show that obstacle avoidance is only required locally and simple depth maps are sufficient to prune invalid actions and locations for navigation. More broadly, our approach can be considered as closely related to model-based control, which an alternative paradigm to RL-based policy learning, with the key insight that components of the model and cost functions can be learned from passive data.
proxy for it. In fact, RL formulations frequently use 'distance
No RL, No Simulator Approach (NRNS): Our NRNS algorithm can be described as follows. During training we learn two functions from passive videos: (a) a geodesic distance estimator: given the state features and goal image, this function predicts the geodesic distance of the unexplored frontiers of the graph to the goal location. This enables a greedy policy in which we select the node with the least distance; (b) a target prediction model: Given the goal image and the image from the agent's 's current location, this function predicts if the goal is within sight and can be reached without collisions, along with the exact location of the goal. The key is that both the distance model and the target prediction model can be learned from passive RGBD videos, with SLAM used to estimate relative poses. We believe our simple NRNS approach should act as a strong baseline for any future approaches that use RL and simulation. We show that NRNS outperforms end-to-end RL, even when RL is trained using 5x more data and 10x more compute. Furthermore, unlike RL methods which need to be trained in simulation because they require substantial numbers of interactions, NRNS can be trained directly on real-world videos alone, and therefore does not suffer from the sim-to-real gap.
2 Related Work
Navigation in simulators. Navigation tasks largely fall into two main categories (. ones in which a goal location is known [ 12 and limited exploration is required, and ones in which the goal location is not known and efficient exploration is necessary. In the second category, tasks range from finding the location of specific objects (5), rooms 14, or images [13), to the task of exploration itself [2]. The majority of current work [12 15
16 3 leverages simulators (7] and extensive interaction to learn end-to-end models for these tasks. In contrast, our work shows that
the semantic cues needed for exploration-based navigation tasks can be learned directly from video
trajectories.
Navigation using passive data. Several prior works have tackled the navigation task when there is
some passive experience available in the test environment (17 18 19 20
21
22 more limited
number of works train navigation policies without simulation environments
Unlike these works, we tackle the task of image goal navigation in unseen environments where there is no experience available to the agent in the test environments during training [19
19
22
23
22) and no map 23 Two works adopting similar requirements are Chaplot et al. [ and Chang et al. 24 Most closely related is Neural Topological SLAM (NTS) 44 which builds topological map and estimates distance to the target image using learned function. But NTS requires access to panoramic observations and ground- -oound-truth maps for training the distance
knowledge is given about the environments

---

function, making it much less scalable than our approach which can be trained with just video data
with urbitrary field of view. [Chang et all [24 adopt similar approach to NTS for object goal navigation, while also incorporating video data from YouTube for learning a Q function. A key
difference that our method learns a episodic distance function, utilizing all past observations to
estimate distances to the target image. In comparison, comparison.CCaploot et al.] [ and Chang et al. (4 use memory-less distance function operating only on the agent's 's current observation.
requires: (a) access to actions (e.g. teleoperated in simulator); and (b) optimal human demonstrations. The goal in LfD is to learn policies that match given demonstrations by either mimicking the actions or formulating rewards using the demonstrations. In contrast, NRNS does neither, since our passive training data does always contain optimal navigation trajectories and NRNS does not compute any rewards as it does not require RL or credit assignment. We make the same distinction between our work and offline RI. 28
Prior work work on LID 25 26 27
29 30,, which does not utilize online data gathering but still learns a policy using rewards from observed trajectories. In contrast, we do not use any rewards or learn a policy. Instead, our approach aims to learn a distance-to- goal model and uses a greedy policy based on the distance predictions. From this perspective, NRNS is performing model-based learning.
Map-Based Navigation. There are multiple spatial representations which can be leveraged in solving navigation tasks. Metric maps. which maintain precise information on the occupied space in an environment, are commonly used for visual navigation [2] 10 6 However, metric maps suffer from issues of scalability and consistency under sensor noise. As result, topological maps have recently gained traction [ 21 18, 24 3D as a means to combat these issues. A significant difference from our approach is that in these prior works a topological map given at the beginning of the navigation task, and is not created or changed during navigation. Specifically, [Savinov et al. [8]] create the
map from a given 5 minute video of the test environment and Chen et al. [2]]] assume access to ground truth map. In our image-goal navigation set up, the agent is given no information about the test environment. Chaplot et al. do not require experience in the test environment and build topological maps at test time, but still requires access to simulator for computing shortest-path distances between pairs of images. Additionally, topological maps for robotic navigation draw inspiration from both animal and human psychology. The cognitive map hypothesis proposes that the brain builds coarse internal spatial representations of environments [32] [3]. Multiple works argue that this internal representation relies on landmarks [44] 35 making human cognitive maps more similar to topologiaal maps as opposed to metric maps.
Graph Neural Networks. Graph Neural Networks (GNN) are specifically used for modeling relational data in a non-Euclidean space. We employ a GNN for estimating distance-to-goal over the agent's exploration frontier. Specifically we employ an augmented Graph Attention Network (36] which allows for a weighted aggregation of neighboring node information. Graph Networks are rarely-used for the task of topological map- map-based visual navigation. [Savinov et al. [[& use GNN for the sub-task of localizing the agent in a ground truth topological map. We believe we are the first to leverage graph neural networks in a visual navigation task.
3 Image Goal Navigation using Topological Graphs
We propose No RL, No Simulator "NRNS". hierarchical modular approach to image- navigation that consists of: a) high-level modules for maintaining a topological map and using visual and semantic reasoning to predict sub-goals, and b) heuristic low-level modules that use depth data to select low level navigation actions to reach sub-goals and determine geometrically-explor We first describe NRNS in detail and then show in Sec. that the high-level modules can be trained without using any simulation, interaction or even ground-truth scans - i.e. that passive video data alone is sufficient to learn the semantic and visual reasoning models needed for navigation.
areas.
3.1 Formulation and Representation
Task Definition. We tackle the task of image- -oage-goal navigation, where an agent is placed in a novel environment, and is tasked with navigating to an (unknown) goal position that is specified using an image taken from that position, as shown Fig. 2 More formally, an episode begins with an agent receiving an RGB observation (IG) corresponding to the goal position. At each time step, I, of the episode the agent receives set of observations 8t, and must take a navigation action at. The agents state observations, 8t, are defined narrow field of view RGBD image, Le, and egocentric pose
3

---

estimate, Pt. The agent must execute navigational actions to reach goal within a maximum number of steps.
Topological Map Representation. The NRNS agent maintains topological map in the form of graph G(N,)) .). defined by nodes N and edges E, which pro- vides a sparse representation of the environment. Con- cretely, a node n, E N is associated with a pose de- fined by location and orientation. Each node n, can either be 'explored' i.e. the agent has previously visited the pose and obtained a corresponding RGBD image It or 'unex- plored' e.g. unvisited positions at the exploration frontier which may be visited in the future. Each edge E F connects pair of adjacent nodes n; and nz. Nodes are deemed adjacent only a short and "simple" path exists be- tween the two nodes, further detailed in Sec. 3.3) Each edge between adjacent nodes then associated with the attribute AP the relative pose between the two nodes.
Gurrent Observation
Gool Image
Path Taken
Explored Node
Unexplored Node
Current Node
Goal Location (not given)
Figure 2: Image- Goal navigation task using topological graph.
3.2 Global Policy via Distance Prediction
Given representation of the environment topological graph, our global policy is tasked with identifying the next 'unexplored' node that the agent should visit, and the low-level policy (GLP) is then responsible for executing the precise actions. Intuitively, we want our agent to select as the next node the one that minimizes the total distance to goal. The global policy's inference task can thus be reduced to predicting distances from nodes in our graph to the goal location. To this end, our approach leverages a distance prediction network (Gp) which operates on top of G(N, E)) predict distance-to-goal for each unexplored node Aue G. Our global policy then simply selects the node with least total distance to goal which is defined as: distance the unexplored node from the agent's current position, plus the predicted distance from the unexplored node to the goal.
The input to the distance prediction network Gp is the current topological graph G(N, E); and Ig- While the explored nodes have an image associated with them, the unexplored nodes naturally do not. To allow prediction in this setup, we Graph Convolutional Network (GCN) architecture first. induce visual features for The E G, and then predict the distance to goal using an MLP.
As illustrated in Fig. the network first encodes the RGB images each explored node (n,) using a ResNet18 37 to obtain feature vector h; E 15122 Each edge eij is further represented by feature vector Mi.j which the flattened pose transformation matrix ('K, E 4244) The adjacency matrix A,, Ui.j and h are passed through a GCN comprising of two graph attention (GAT) layers [36] with intermediate nonlincarities. Note that we extend the graph attention layer architecture to additionally use edge features ujj when computing attention coefficients. The predicted visual features for unexplored nodes are then finally used to compute predicted distance-to-ge d, from each node Ig using simple MLP.
To select the most 'promising' Nue to explore, the distance from the agent's current location ny also needs to be accounted for. For Thue E G, the 'travel cost' is added to d,, calculated using shortest path on from The + T4, Our global policy then selects the unexplored node with the minimum total distance score as the next sub-goal.
3.3 Local Navigation and Graph Expansion
The NRNS global policy selects the sub-goal that the agent should pursue, and the precise lowelevel actions to reach the sub-goal are executed by a heuristic local policy. After the local policy finishes execution, the NRNS agent updates the graph to include the new observations and expands the graph with unexplored nodes at the exploration frontier.
Local Policy. The NRNS local policy, denoted as GLP, receives a target position, defined by distance and angle (p:, Φ)) with respect to the agent's current location. When Gn outputs a sub-goal node, Pi, di are calculated from the current position and passed to Gr.p. Low level navigation actions are selected and executed using simplistic point navigation model based on the agent's egocentric RGBD observations and (noisy) pose estimation. To navigate towards its sub-goal, the agent builds

---

Graph G(N.E)
G(N,E)
GAT Layer
pouuts < MTW
Graph Encoder
DIXIM
9
Lix
10X1
5
subgoa1
nu Th
12X
25
Arghin
Resnel 16
05
(mix)))ne))
Goal Image
400 X.44
Travel
Figure 3: The Global Policy and GD architecture used to model distance-to-goal prediction. GD employs Resnet18 encoder, Graph Attention layers, and a multi-layer perceptron with sigmoid.
and maintains local metric map using the noisy pose estimator and depth input. This effectively allows it to reach local goals and avoid obstacles. The local metric maps are discarded upon reaching the sub-goal, as they are based on noisy pose estimator. This policy adapted from [TO] and also used for Image-Gord Goal Navigation in [4.
Explorable Area Prediction for Graph Expansion. We incorporate 'graph expansion' step after the agent reaches a sub-goal via 9LP and before the agent selects a new sub-goal. First, the agent updates G(N, F.) to record the current location RGBD observation. Second, the agent creates additional 'unexplored" nodes, Tlue, adjacent to the current node, n, based on whether the corresponding directions deemed 'explorable" We use explorable area prediction module, GEA. to determine which adjacent areas to the current location are geometrically explorable. This is untrained, heuristic module takes the egocentric depth image I tarpen and tests angles in the direction 9 from the current position and returns the angles 0 that are not blocked by obstacles within depth meters. The NRNS agent tests at 1,, these angles are chosen based on the agent's 's turn radius of 15t and 120º FOV. For all @ determined to be explorable, the agent updates G(N, F) by adding an 'unexplored' node at position p 1m and Φ relative to the node the agent is at, with corresponding edge linking the current node to the new node. a node already exists in one of the explorable areas at nearby position, only an edge added to G(N, EE) and not a new node.
explored node and store the associated
3.4 Putting it Together
Stopping Criterion. The above described modules (GD. GLP, GEA) allow the NRNS agent to incrementally explore its environment while making progress towards the location of the target image. To allow the agent to terminate navigating when it is near the goal, we additionally learn target prediction network Gr that provides stopping criterion. Given Ia and current image 4, Gr simple MIP that predicts: so, b) the relative position (distance, direction) of the goal Pg. dg.
probability 8g E [0, 1] indicating whether the goal is within sight, and
Algorithm. We outline our navigation algorithm in Fig. 4 An example result on the image-goal navigation task is shown Fig. Among the modules used, the local policy module GLP and the explorable area module GEA do not require any learning. As we show in Sec. Gp and Gr can both be learned using only passive data.
Algorithm 1: NRNS Image Navigation
initialize graph
= (P==o (Ng,ng): no):
(mg); (ne):
G E);
loop until reached goal or nax steps
while steps token
steps do
ni I: Bit GgA (T:): // deternine valid explorable areas // add unexplored nodes edges Reg (peg (xg) aggmin ( (G, (c) +TrawdCossG n.)): // select sub-gosl
le+t,Pa41 P++1 -..p(Ppgg dag): // navigate sub-goal
(Pt-1.1141// update graph with observations 9. (Pg. (r(si,,
stopping criterion
9. 2.5 then
Gr d,): // navigate to target break:
end
end
5

---

Goall RGB
Global Policy 6g
Subgoa1 (R:
Local Policy Sy
May
Graph
Update
Prediction s.
Target
R3
Graph Map G(NE)).
Graph Map G(NE):
Current Depth
Current
Figure 4: The hierarchical modular NRNS approach to the Image-Goal Navigation task, for a single time step in the episode. The global policy Gp selects unexplored node n, sub-goal. The sub-goal position (ps, Φ;) is passed to the local navigation policy 9LP which takes in the current RGBD observations and outputs low level actions until the agent reaches n4. The graph then updated with the current observations 4441 and new unexplored nodes and edges generated by GEA-
4 Learning from Passive Data
The learned high-level policies of NRNS are the Distance Network GD and Target Prediction Network Gr. A key contribution of our work is showing that these functions can be learned from passive data alone. This eliminates the need for online interaction and ground-truth maps, allowing us to train the NRNS algorithm without using RL or simulation.
Learning Distance Prediction. First, we describe how learn the function Gp. Given a topological graph (consisting of both explored and unexplored nodes) and a goal image as input, Gp predicts the geodesic distance from all unexplored nodes the location of the goal image. Our training data therefore consists of triplets of the form (G. Ig. Du), where G is a topological graph, Ig is goal image, and Dy is the ground-trute distances from set of unexplored nodes the goal location (we use Lo-Loss to train the distance function).
We generate training graphs using passive videos two-step process. In the first step, we create topological graph, Gy; for every video V; in the passive dataset. Each graph contains both explored and unexplored nodes. We approximate distance unexplored nodes using the geodesic distance along the trajectory. In the second step, we uniformly sample sub-graphs and goal locations over each video's topological graph.
Step 1: Video to Topological Graph. First, we generate a step-wise trajectory graph in which each frame is node with odometry information. We then process this step-wise graph into topological graph using affinity clustering [38]. We use the visual features from each frame concatenated with odometry information the feature vector for clustering. Visual features for each frame extracted via a Places365 339 pretrained Resnet18. Each cluster represents a single node in the topological graph Gy, and the stepwise visual features of frames in the cluster average pooled. The topological graphs are then expanded to have unexplored nodes, used in training the distance function. These are created by applying GEA the RGBD of each node centroid in Gv. Fig, illustraaees the process by which an example video is converted into a topological graph for training data.
Step 2: Sampling training datapoints. Individual data instances are selected via uniform sampling without replacement. This means selecting a random node in Gy as the goal location and a random sub-grap sub-graph of Gy as the observed trajectory and current location. Distance along the trajectory is used as the ground truth distance between nodes in the sub-graph and the goal image: these distance labels used to train the network Gp. Due to non-optimal long term paths of the video trajectories, distances may overestimate distance to the goal. In other words while each node along the generated trajectory graphs are legitimate paths to the goal, shorter paths may exist that not covered in the training trajectory graph, making this a challenging problem. In total, Gp is trained using ~ 1k training instances per scan for both Gibson and MP3D.
Learning Target Direction Prediction The GT prediction network receives goal image and the current node image input, and predicts whether the goal is within sight of the current image. To gather training instances for Gr, we use the RGB frame at randomly selected node as the goal image. We make simplifying assumption that any adjacent pair of nodes (similar features and odometry) in the topological graph are positive examples and any other pair of nodes are negative examples.

---

Passive Trajectory Video V/j
Trajectory Graphs Got
Exporade Area va
R
Trijeckory
Stepwise
Affinity Clustered
Adapted Graph for Training 5.
Trajectory
Trajectory
Figure 5: Example from the passive video dataset. Frames of video trajectory V; are shown on the left. The stepwise trajectory is then turned into trajectory graph Gy; via affinity clustering (38 of node image and pose features. Gy, adapted to train the Global Policy Gn and target direction prediction A.A. An example of an adapted Gy, for training is shown on the left.
5 Experiments
Image-Goal Navigation Task Setup. At the beginning of each episode, the agent is placed in an unseen environment. The agent receives observations from the current state, 3 x odometry pose reading and RGBD image, and the RGB goal image. Observation and goal images are FO0 and of size 480 x 640. An episode is considered a success if the agent is able to reach within Im of the goal location and do so within maximum episode length of 500 steps. Each episode is also evaluated by the efficiency of the navigational path from start to goal, which is quantitatively measured by Success weighted by inverse Path Length (SPL.) [ Note, the narrow field of the agent in this task definition differs from past works which use panoramic views [4]. The decision to use narrow field based on our method of training only on passive data. Current passive video datasets of indoor trajectories such as YouTube Tours (24]. RealEstate 10k not contain panoramas.
[40 and our NRNS passive video dataset, do
Action Space. The agent's action space contains four actions: forward by .25m, rotate left by 15,, rotate right by 15, and stop. In our experiments, we consider two cases for pose estimation and action transition. In the first condition the agent has access to ground truth pose and the navigation actions are deterministic. In the second condition, noise is added to the pose estimation and the actuation of the agent. We utilize the realistic pose and actuation noise models from (10), which are similarly used in [4] The actuation noise adds stochastic rotational and translations transitions to the agent's 's navigational actions.
Training Data. Our key contribution is the ability to learn navigation agents from passive data. In theory, our approach can be trained from any passive data source, and we test this in Sec. 5.2 using RealEstate 10K [40] However, since RL-based baselines are trained in the Habitat Simulator 17), generate our NRNS dataset, of egocentric trajectory videos, using the same Habitat training scenes to provide direct comparison and isolate domain gap issues. Specifically, the trajectory videos are created as follows. A set of 4 points are randomly selected from the environment using uniform sampling. A video is then generated of the concatenated RGBD frames of the shortest path between consecutive points. Note that the complete video trajectory (from first source to final target) is not step-wise optimal. Frames in the videos are of size 480 X 640 and have a FOV 120%and each frame is associated with a 3 x I odometry pose reading. In the noisy setting discussed in Sec. [ sensor and actuation noise is injected into training trajectories. We create 19K, 43K video trajectories, containing 1, 2.5 million frames respectively, on the Gibson and MP3D datasets. We then use this data to train the NRNS modules, as described in Sec.
Test Environments.
We evaluate our approach on the task of image-goal navigation. For testing, we use the Habitat Simulator 77. We evaluate on the standard test-split for both the Gibson [41] and Matterport3D (MP3D) 42 datasets. For MP3D, we evaluate on 18 environments and for Gibson, we evaluate on 14 environments.
Baselines. We consider a number of baselines to contextualize our Image-Goal Navigation results:
BC w/ ResNet + GRU. Behavioral Cloning (BC) policy where I and Ig are encoded using pretrained ResNet- 18. Both image encodings and the previous action az-1 are passed through a two layer Gated Recurrent Unit (GRU) with softmax, which outputs the next action at.
BC w/ ResNet + Metric Map. BC policy: I, and IG are encoded with a ResNet, same as the above policy. This policy keeps metric map built from the depth images. The metric map is encoded with a linear layer. The metric map encoding and encodings of I, and IG are concatenated and passed into an MLP with softmax, which outputs the next navigational action ac.
7

---

Goal Image
Current Opservation 1,
Cureen Graph G.(.V)
Explored Node
Unexplored Node
Gumont Node
Goal Location
Stops
Figure 6: Example of an Image-Goal Navigation episode on MP3D. Shows the agent's observations and internal topological graph at different time steps.
End to End RL with DDPPO. An agent is trained end to end with proximal policy optimiza- tion 13 in the Habitat Simulator 7 for the image-goal navigation task.
We use and adapt code for baseline algorithms from Chaplot et al. [4. However, as our setup uses narrow-view cameras instead of panoramas, these adapted baselines perform worse compared to their previously reported performance. This difference in setup also makes direct comparison with 4 infeasible, as they critically rely on panoramic views for localization. The baseline behavioral cloning policies are also trained only using the passive dataset described in Sec. 4 This makes the BC baseline policies directly comparable to the NRNS model.
The end-to-end RL policy is trained using a Habitat implementation of DDPPO [3]. For training we use GPUs and 16 processes per GPU. We first train with Ik episodes per house, which identical to how NRNS is trained, and we train for 10M steps. To test the scalability of the RL method, we additionally train model using 5k training episodes per environment (providing the RL agent 5x more data than our NRNS agent) and report the performance of the agent trained on these episodes at 50M steps (5x more compute than NRNS) and 100M steps (10x more compute than NRNS). Training the Gn model takes 20 epochs, requiring ~ hours on single GPU. Training the Gr model takes ~ 10 epochs, requiring -44 hours on a single GPU.
Episode Settings. To provide an in-depth understanding of the successes and limitations of our approach, we sub-divide test episodes into two categories: 'straight' and 'curved'. In 'straight" episodes, the ratio of shortest path geodesic-distande to suclidean-distane goal locations is < 1.2 and rotational difference between the orientation of the start position and goal image is < 45% All other start-goal location pairs are labeled as 'curved' episodes. We make this distinction due to the nature of the narrow field of view of our agent, which strongly affects performance on curved episodes, since the agent must learn to turn both as part of navigating and part of seeking new information about the target location. Also, while a greedy policy being successful on 'straight" episodes might be expected, competitive performance on even 'curved' episodes will highlight how effective our simple model and policy is. We further subdivide each of these categories into 3 sub-categories of difficulty: 'easy', 'medium' and 'hard'. Difficulty is determined by the length of the shortest path between the start and goal locations. Following 4 the 'easy', 'medium" and 'hard' settings are (1.5 3m), (3 5m), and (5 10m) respectively. To generate test episodes we uniformly sample the test scene for start-goal location pairs, to create approximately 1000 episodes per setting.
between the start and
5.1 Results
Tables 2 show the performance of our NRNS model and relevant baselines on the test splits of the Gibson and Matterport datasets. In 6 we visualize an episode of the NRNS agent as it navigates to the goal-image.
NRNS outperforms baselines. Our NRNS algorithm outperforms the BC and end to end RL policies in terms of Success and SPL @ Im on both datasets. NRNS improves upon the best offline baseline, a Behavioral Cloning (BC) policy with a ResNet and GRU, across splits of Gibson by an absolute 20+% on Straight episodes and 10+% on Curved episodes. We find that BC policy using a GRU for memory outperforms using only a metric map. We attribute this to a spatial memory (metric map) being less informative for agent exploration than memory of the visual features and previous
8

---

steps (GRU). We observe that an end-to-end RL policy trained for 10M steps with 10k epsiodes per house, in simulation performs much weaker than all baselines. With increased data and steps RL baselines unsurprisingly increase in performance, however we find with 5X more data and 10x more compute baselines still are outperformed by NRNS. The low performance of behavioral cloning and RI. methods for image-goal navigation is unsurprising 4 [5]. This demonstrates the difficulty of learning rewards on low level actions instead value learning on possible exploration directions, exacerbating the difficulty of exploration in image-goal navigation. Adding the challenges of the task, all policies must learn the stop action. Previous works 4 have found that adding oracle stopping, target-driven RL agent, leads to large gains in performance
image- -gaa navigation. The limitations of all approaches are seen on the 'hard' and 'curved' episode settings, showing the overall difficulty of the exploration problem and the challenge of using narrow field of view.
NRNS is robust noise. Even with the injection of sensor and actuation noise TQ], in both the passive training data and test episodes, NRNS maintains superior near comparable performance all baselines. In fact, we find that the addition of noise leads to only an absolute drop success between 8-8% Gibson 41] and 1-5% MP3D [42]. An interesting observation is small increase in performance (w/noise) for the hard-case. We believe this is because gi-distance for hard cases are more error prone and noise during training provides regularization.
Table Comparison of our model (NRNS) with baselines on Image-Goal Navigation Gibson 41
We report average Success and Success weighted by inverse Path Length (SPL) @ 1m. Noise refers injection of sensor & actuation noise into the train videos and test episodes. denotes using simulator.
Easy
Medium
Hard
Path Type
Model
Suce f SPL
Suce f SPL f
Suec f
SPL f
RL (10M steps)
10.50
6.70
18.10
16.17
11.79
10.85
RL (extra data 50M steps)"
13
36.30
34.93
35.70
33.98
5.94
6.33
Straight
RL. (extra data 100M steps)
13
43.20
38.54
36.40
34.89
7.44
7.20
BC w/ ResNet Metric Map
24.80
23.94
11.50
11.28
1,36
1.26
BC w/ ResNet GRU
34.90
33.43
17.60
17.05
6.08
5.93
NRNS w/ noise
64.10
55.43
47.90
39.54
25.19
18.09
NRNS w/out noise
68.00
61.62
44.56
23.82
18.28
RL (10M steps) 13
7.90
3.27
9.50
7.11
5.50
4.72
RL (extra data 50M steps)
33
18.10
15.42
16.30
14.46
2.60
2.23
Curved
RL (extra data + 100M steps)
13
22.20
16.51
20.70
18.52
4.20
3.71
BC w/ ResNet Metric Map
3.10
2.53
0.80
0.71
0.20
0.16
BC w/ ResNet GRU
3.60
2.86
1.10
0.91
0.50
0.36
NRNS w/ noise
27.30
10.55
23.10
10.35
10.50
5.61
NRNS w/out noise
35.50
18.38
23.90
12.08
12.50
6.84
Table 2: Comparison of our model (NRNS) with baselines on Image-Gou Navigation MP3D 42
Easy Suec
Medium
Hard
Path Type
Model
SPL
Succ t SPL
Suec t SPL 1
RI. (10M steps)
7.50
4.00
3.50
1.73
1.00
0.55
RL (extra data 50M steps)"
13
34.50
30.08
35.70
33.80
10.40
10.07
Straight
(extra data 100M steps)
36.40
30.84
33.80
31.42
12.00
11.56
BC ResNet Metric Map
25.80
24.82
11.30
10.65
3.00
2.93
BC w/ ResNet GRU
30.20
29.57
12.70
12.48
4.40
4.34
NRNS w/ noise
63.80
53.12
36.20
26.92
24.10
16.93
NRNS w/out noise
64.70
58.23
39.70
32.74
22.30
17.33
(10M steps)" 13
1.78
3.20
1.10
0.46
RL (extra data 50M steps)'
13
15.70
11.34
10.70
9.03
3,90
3.57
Curved
(extra data 100M steps)
17.90
13.24
15.00
12.17
5.90
4.87
BC w/ ResNet + Metric Map
4.90
4.23
1.40
1.29
0.40
0.34
BC w/ ResNet GRU
3.10
2.61
0.80
0.77
0.10
0.02
NRNS w/ noise
21.40
8.19
15.40
6.83
10.0
4.86
NRNS w/out noise
23.70
12.68
16.20
8.34
9.10
5.14

---

NRNS Module Ablations. Tab. reports detailed ablations of NRNS on the Gibson dataset (see appendix for ablation results on MP3D). We ablate the NRNS approach by testing each module individually. In the ablation experiments, we replace the module output with the ground truth labels or numbers in order to evaluate the affect of each module on the performance of the overall approach. For simplicity, all ablations are trained and tested without sensor or actuation noise. Unsurprisingly, we find that the Global Policy, GD. has a large affect on performance (Row 4 and 8). We find that the largest affects are seen in the 'hard' and 'curved' test episodes. This is unsurprising because as the distance to the goal increases, the path increases in complexity and the search space of GD increases.
Table 3: Ablations of NRNS with baselines on Image-Goal Navigation on Gibson 41
We report
average Success and Success weighted by inverse Path Length (SPL) @ 1m. denotes a module being replaced by the ground truth labels and a / denotes the NRNS module being used.
NRNS Ablation GEA
Easy
Medium
Hard
Path Type
Gr
GD
Suce f SPL f
Succ f SPL f
Suce f SPL f
x
x
x
100.00
99.75
100.00
99.62
100.00
99.57
/
x
x
99.90
99.05
98.20
95.06
95.91
90.53
Straight
<
x
79.40
73.48
71.30
67.48
62.16
58.06
v
/
68.00
61.62
49.10
44.56
23.45
18.84
x
x
100.00
97.62
100.00
97.47
100.00
98.18
V
X
99.70
95.93
97.50
90.14
89.30
79.95
Curved
/
X
65.00
56.70
58.10
52.51
47.70
42.28
/
V
35.50
18.38
23.90
12.08
12.50
6.84
5.2 Training on Passive Videos in the Wild
Finally, we demonstrate that our model can be learned from passive videas in the wild. Towards this end, we train our NRNS model using the RealEstate 10K dataset [40] which contains YouTube videos of real estate tours. This dataset has 80K clips with poses estimated via SLAM. Note that the average trajectory length is smaller than test time trajectories in Gibson or MP3D, and we therefore only evaluate on "easy' and 'medium' settings. Tab, | shows the performance. Note there is drop in performance compared to training using Gibson videos, which we attribute to domain shift. Even after this drop, our approach, trained on real-world passive videos, outperforms BC baselines and performs competitively against RL baselines. RL baselines have an advantage as they are both trained in the simulator and trained on the Gibson dataset. We believe that the delta between RL and the performance of NRNS trained on RealEstate would close both were tested on real-world data. We find these results to be strong indication of the effectiivnnss of training passive data.
Table 4: Comparison of our model (NRNS) trained with different sets of passive video data and tested on Image-Gort Goal Navigation on Gibson [41. We report average Success and Success weighted by inverse Path Length (SPI.) @ 1m. Results shown are tested without sensor & actuation noise.
Easy
Medium Suce t SPL
Path Type
Training Data
Model
Suce t
SPL
RealEstate 10k
40
NRNS
56.42
48.01
30.30
25.67
Straight
MP3D
NRNS
59.80
52.35
37.00
31.89
Gibson
NRNS
68.00
61.62
49.10
44.56
Gibson
BC w/ ResNet + GRU
30,20
29.57
12.70
12.48
RealEstate 10k (40
NRNS
21.10
15.76
12.90
5.57
Curved
MP3D
NRNS
28,26
13.59
11.00
5.10
Gibson
NRNS
35.50
18.38
23.90
12.08
Gibson
BC ResNet + GRU
3.10
2.61
0.80
0.77
Conclusion
We have presented a simple yet effective approach for learning navigation policies from passive videos. While simulators have become fast, the diversity and scalability of environments still remains a challenge. Our presented approach, NRNS, neither requires access to ground-truth maps nor online policy interaction and hence forgoes the need for simulator to learn policy functions. We demonstrate that NRNS can outperform RI. and behavioral cloning policies by significant margins. We show that NRNS can be trained on passive videos in the wild and still outperform all baselines.
10

---

Acknowledgement
The authors would like to thank Saurabh Gupta for the discussions. We would also like thank the Gibson and RealEstate IOK dataset authors for sharing their datasets for scientific research.
Licenses for referenced datasets.
Gibson: http://oddd.ssaaao....do..d.... httpa///.aa..ttooco/oooo////aaaa
pdf
Matterport3 http: MateepprttD..attpp///aaaaia vc. in. tum wtt..d///mateerprrr///e-eaa .TS.:ttteeP___PPpppp.p__ pdf
References
[1] Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi. Manolis Savva, et al. On evaluation of embodied navigation agents. arXiv preprint arXiv: (tfi::/80700677. 2018.
[2] Tao Chen, Saurabh Gupta, and Abhinav Gupta. Learning exploration policies for navigation. International Conference on Learning Representations, 2019.
[3] Santhosh K Ramakrishnan, Ziad Al-Halah, and Kristen Grauman. Occupancy anticipation for efficient exploration and navigation. In European Conference on Computer Vision, 2020.
[4] Devendra Singh Chaplot, Ruslan Salakhutdinov, Abhinav Gupta, and Saurabh Gupta. Neural topological slam for visual navigation. In Computer Vision and Pattern Recognition, 2020.
[5] Dhruv Batra, Aaron Gokaslan, Aniruddha Kembhavi, Oleksandr Maksymets, Roozbeh Mottaghi, Manolis Savva, Alexander Toshev, and Erik Wijmans. ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects. In arXiv:2006.131 13171, 2020.
[6] Devendra Singh Chaplot, Dhiraj Prakashchand Gandhi, Abhinav Gupta, and Russ R Salakhut- dinov. Object goal navigation using goal-oriented semantic exploration. Neural Information Processing Systems, 2020.
[7] Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, al. Habitat: A platform for embodied ai research. In International Conference on Computer Vision, 2019.
[8] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. lision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments. In Computer Vision and Pattern Recognition, 2018
[9] Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. AI2-THOR: An Interactive 3D Environment for Visual AI. Computer Vision and Pattern Recognition, 2017.
[10] Devendra Singh Chaplot, Dhiraj Gandhi, Saurabh Gupta, Abhinav Gupta, and Ruslan Salakhut- dinov. Learning to explore using active neural slam. In International Conference on Learning Representations, 2020.
[11] Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In International Conference on Robotics and Automation, 2017.
[12] Saurabh Gupta, James Davidson, Sergey Levine, Rahul Sukthankar, and Jitendra Malik. Cogni- tive mapping and planning for visual navigation. In Computer Vision and Pattern Recognition, 2017.
[13] Erik Wijmans, Abhishck Kadian, Ari Morcos. Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra. Dd-ppo: Learning near-perfect pointgoal navigators from 2.5 billion frames. In International Conference Learning Representations, 2020.
[14] Yi Wu, Yuxin Wu, Aviv Tamar, Stuart Russell, Georgia Gkioxari, and Yuandong Tian. Bayesian relational memory for semantic visual navigation. In International Conference on Computer Vision, 2019.
11

---

[15] Lina Mezghani. Sainbayar Sukhbaatar, Thibaut Lavril, Oleksandr Maksymets, Dhruv Batra, Piotr Bojanowski, and Karteek Alahari. Memory-augmennt reinforcement learning for image- goal navigation. arXiv preprint 1xXiv:2101.05/4 2021.
[16] Wei Yang, Xiaolong Wang, Ali Farhadi, Abhinav Gupta, and Roozbeh Mottaghi. Visual semantic navigation using scene priors. International Conference on Learning Representations, 2018.
[17] Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. Search on the replay buffer: Bridging planning and reinforcement learning. In Neural Information Processing Systems, 2019.
[18) Nikolay Savinov, Alexey Dosovitskiy, and Vladlen Koltun. Semi-parametric topological memory for navigation. In International Conference on Learning Representations. 2018.
[19] Piotr Mirowski, Matthew Koichi Grimes, Mateusz Malinowski, Karl Moritz Hermann, Keith Anderson, Denis Teplyashin, Karen Simonyan, Koray Kavukcuoglu, Andrew Zisserman, and Raia Hadsell. Learning navigate in cities without map. In Neural Information Processing Systems, 2018
[20] Scott Emmons, Ajay Jain, Michael Laskin, Thanard Kurutach, Pieter Abbeel, and Deepak Pathak. Sparse graphical memory for robust planning. In Neural Information Processing Systems, 2020.
[21] Kevin Chen, Juan Pablo de Vicente, Gabriel Sepulveda, Fei Xia, Alvaro Soto, Marynel Vázquez. and Silvio Savarese. A behavioral approach to visual navigation with graph localization networks. In Robotics: Science and Systems, 2019.
[22] Dhruv Shah, Benjamin Eysenbach, Gregory Kahn, Nicholas Rhinchart, and Sergey Levine. Ving:
Learning world navigation with visual goals. International Conference on Robotics and Automation, 2020.
[23] Samarth Brahmbhatt and James Hays. Deepnav: Learning to navigate large cities. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, 2017.
[24] Matthew Chang, Arjun Gupta, and Saurabh Gupta. Semantic visual navigation by watching youtube videos. In Neural Information Processing Systems, 2020.
[25] Harish Ravichandar, Athanasios S Polydoros. Sonia Chernova, and Aude Billard. Recent advances in robot learning from demonstration. Annual Review of Control, Robotics, and Autonomous Systems, 2020.
[26] David Silver, J Andrew Bagnell, and Anthony Stentz. Active learning from demonstration for robust autonomous navigation. In International Conference on Robotics and Automation, 2012.
[27] Xinlei Pan, Tingnan Zhang, Brian Ichter, Aleksandra Faust, Jie Tan, and Sehoon Ha. Zero-shot imitation learning from demonstrations for legged robot visual navigation. In International Conference on Robotics and Automation, 2020.
[28) Noah Y Siegel, Jost Tobias Springenherg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprint arXiv:2002.08396. 2020.
[29] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint trXiv:2005.06645 2020.
[30] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint htt::2000000779, 2020.
[31] Edward Beeching, Jilles Dibangoye, Olivier Simonin, and Christian Wolf. Learning to plan with uncertain topological maps. In European Conference on Computer Vision, 2020.
[32] Russell A Epstein, Eva Zita Patai, Joshua Julian, and Hugo J Spiers. The cognitive map in humans: spatial navigation and beyond. Nature Neuroscience, 2017.
12

---

[33] Edward C Tolman. Cognitive maps in rats and men. Psychological Review, 1948.
[34] Patrick Foo, William H Warren, Andrew Duchon, and Michael J Tarr. Do humans integrate routes into a cognitive map? map-versus landmark-based navigation novel shortcuts. Journal of Experimental Psychology: Learning, Memory, and Cognition, 2005.
[35] Ranxiao Frances Wang and Elizabeth S Spelke. Human spatial representation: Insights from animals. Trends in Cognitive Sciences, 2002.
[36] Petar Veličković, Guillem Cucurall, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. International Conference on Learning Representations. 2017.
[37] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Computer Vision and Pattern Recognition, 2016.
[38] Brendan Frey and Delbert Dueck. Clustering by passing messages between data points. Science, 2007.
[39] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. Transactions on Pattern Analysis and Machine Intelligence, 2017.
[40] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnifi- cation: Learning view synthesis using multiplane images. In SIGGRAPH, 2018.
[41] Fei Xia, Amir R Zamir, Zhiyang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: Real-world perception for embodied agents. In Computer Vision and Pattern Recognition, 2018.
[42] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environment. International Conference 3D Vision, 2017.
13


Reviews:
[{"review_id": "micfuFB2oM2", "reviewer": "Reviewer_szNL", "summary": "This work details a method for planning and navigation on the image-driven navigiation task in a photo-realistic simulator. The novelty of this approach is that the method is trained with sequences of (RGBD, pose) sampled from the environment, without the associated actions or rewards. The sequences are used to learn a predictive model of a distance function, which the authors claim can learn priors about the indoor layouts of indoor buildings, which enables the method to be transferred to buildings that were not observed during training.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The work presented is novel, of high quality and clearly written. The method is particularly of interest as it can be used to create a policy without the need to interact in a simulator, which means the method has the potential to scale to vast real world datasets. \nThe method is compared against three baselines and outperforms them by a large margin. Detailed ablations are shown to break down the modular approach of this method.\nWhilst the approach is novel, and outperforms the presented baselines by a large margin, I have a number of questions about the details of the implementation of the baselines.\n \nStrengths:\n1. The work outperforms baselines based on RL and behavioral cloning.\n2. This approach decouples low-level acting and planning in interpretable manner .\n3. The modular approach means that each component can be trained independently, which separates the problem into independant subproblems.\n\nWeakness:\n\nBaselines - I have a number of questions and concerns about the baselines:\n0. It would have been good include a baseline with a policy of sampling actions randomly, in order to ensure the other baselines outperform purely random actions.\n1. The baseline performs rather poorly in this setting, whilst the task is challenging I still find a success rate of 3.6% on goals within 1.5-3m suprisingly low.\n2. How many seeds were evaluated when training the baseline method?\n3. What is the reward function used for the RL method?\n4. Input resolution: Given the instability and optimization challenge of training an RL method, an input resolution of 480x640 seems rather large, have the authors tested a down-sampled input?\n5. On-policy Deep-RL methods such as PPO are known to be sample inefficient, do the authors consider the baseline to have converged after a \"mere\" 10M environment interactions?\n6. Stopping criterion, it would be interesting to see comparisons of the baseline methods using the learned stopping criterion. As one reason an end to end RL method performs poorly in the setting could be related to the difficulty of exploring the action space when a STOP action is present, which presumable terminates the episode. In Neural Topological SLAM the approachs were compared with and without the need to a STOP actions. It would be good to have included this comparison.\n7. Many recent works have shown scaling and distributed training of RL methods can outperform handcrafted learning based methods, for example [1] vs [2]. What is the authors opinion of scaling RL methods on a more complex task such as image-driven goal navigation?\n\nHandcrafted learning based methods require time, tuning and domain expertise. How transferable is this approach to other tasks such as object goal navigation or coverage maximization?\n\nFor the local navigation policy, have the authors compared against an local pointgoal RL agent?\n\nMissing references:\nI draw the authors attention to [3] which is a recent related work on topological planning with graph neural networks in photorealistic envrionments. \n\n[1] DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames. Wijmans et al. ICLR 2020\n[2] LEARNING TO EXPLORE USING ACTIVE NEURAL SLAM. Chaplot et al. ICLR 2020\n[3] Learning to plan with uncertain topological maps. Beeching et al. ECCV 2020\n\nUPDATE AFTER AUTHOR RESPONSE: Thank you to the authors for addressing my questions. I will leave my review at 7. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "One limitation is the requirement of pose estimates. The authors show a comparison where nose is added to the pose estimates. A more real world comparison where SLAM is used directly on the sequences to infer relative position and orientation would be interesting, but is perhaps beyond the scope of an already extensive conference paper.\n\nA second interestesting extension would be sim2real transfer of policies trained with such a method.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "i_adEcGyrn", "reviewer": "Reviewer_dPyV", "summary": "The paper introduces a new method for learning a navigation policy for indoor environments without RL and without simulation (which is, admittedly, fitting given the title). The method uses SLAM on passive video recordings to construct navigation graphs that are then used to train several networks including a goal-detection network that can be used in goal-based navigation.\n\nI personally like the idea in this paper a lot and clearly, a lot of work went into the experiments and into the polish. That is why I'm upset that I have to give this paper a lower score than it deserves. The main thing that's hindering me from wholeheartedly recommending this is the lack of code and reproducibility. If code is not included, then I have to put the methods section under more scrutiny and there aren't enough details to reproduce this work. I'm happy to change my score if the authors address the issues listed below.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality:** \n\nThere are many works on using SLAM as part of an RL pipeline but all of these need to be trained interactively. Other works do indoor navigation from imitation and these, too, often need to be tuned via RL. The works that do learn from passive video, don't learn a distance estimator and only compare the current frame of a trajectory to the goal. Therefore I think the method presented in this work is sufficiently novel. \n\n\n**Quality:** \n\nThe experiments sufficiently back up the method, the method is well-motivated, the ablations are insightful.\nThe only concerns I have with respect to quality are\n1. The low performance of PPO - without listing the hyperparameters. I have used PPO for indoor navigation in Gibson for my own experiments and with a bit of tweaking, the performance can be brought up to a 10-15% success rate, especially on the \"easy\" task. I would recommend the authors spend a little more time with this baseline to establish a fair comparison.\n2. Why is the agent with access to the metric map so bad? I don't think this was sufficiently justified in the paper. \n3. In the supplemental video, why does the second trajectory just fade out long before reaching the target? And the third trajectory seems to miss the target. I'm very confused by the video - it makes your method look significantly worse.\n\n\n**Clarity:** \n\nAs mentioned above, this is my main concern with this paper. Again, I'm only this strict because you didn't include any code with your submission. I can't accept any promise to submit the code later since I can't review non-existent future code. \nThere are 4 subcomponents to your method,\n- $G_D$, the global policy\n- $G_{LP}$, the local policy\n- $G_{T}$, the stop signal function\n- $G_{EA}$, the explorable area function\n\n$G_D$ is learned with a Graph Neural Network. There is a nice plot of the network in Figure 3 but is that all? What are the sizes of the GAT layers, their inputs and outputs? Is the MLP single-layer or multi-layer. You wrote \"simple MLP\" but what does that mean? Could we maybe get a network table where each layer is listed, together with their activation function and the number of parameters? And what exactly is the structure of the graph encoder? \nAnd for preparing the data, you wrote \"First, we generate a stepwise trajectory based on odometry information from each frame.\" (line 210)... How exactly do you do that? What kind of SLAM system is employed here, and with which parameters? And likewise, \"We adapt this trajectory to topological graph structure using affinity clustering on the stepwise graph via visual features and odometry information.\" - How? Which parameters, and which affinity clustering algorithm is used? Without this information, I cannot reproduce this.\n\n$G_{LP}$ is \"adapted from [10]\" - but what does that mean? \"Adapted\" means \"make (something) suitable for a new use or purpose; modify\", so how was it modified to suit your needs?\n\n\"$G_{T}$ is simple MLP\" (sic! line 186) - what does that mean? How many layers, activation functions, etc? Apparently, it has multiple output heads (for goal in sight and relative position of goal).\n\n$G_{EA}$ - \"This is untrained, heuristic function\" (sic! line 175). It supposedly returns angles that don't have an obstacle within 3m. But what if there is a partial obstacle? Say half the frame has an obstacle, or a third or a fifth? What's the cutoff? Or what if the obstacle is 2m above ground like a doorframe? \n\nAlso, from the text, line 148, you are using the visual features of unexplored nodes in your $G_D$. How does that work - which network generates visual features of unexplored nodes?\n\nAnd in general, what are any of the loss functions you are training, what is your optimizer, what are the training parameters? There are superficial details in chapter 1 of the appendix, but the losses aren't included there and the detail there is not enough to warrant any reproduction.\n\n\n\n**Significance:** \n\nI think if it were reproducible, this would be a significant contribution.\n\n**Nitpicks and Questions:** \n\n1. The first third of the paper is well-written and mostly error-free. But as pointed out above, sections 4 and 5 are riddled with errors like missing articles and missing commas. Please proofread the document! For example lines 180, 228, 239, 259 twice, 279, 290, 305. Also, I believe that in general, if you write \"x-based\" like RL-based or Graph Neural Network-based, there needs to be a dash between \"based\" and the word before.\n2. What's the PPO implementation and backend? I assume you're not using the standard Atari CNN, right?\n3. In Table 1, for a 1.5-3m goal that's in a straight line from the agent, why is the performance so bad? ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have neither assessed limitations nor possible societal impact in the main body of the paper.\n\nHere are some ideas to discuss for limitations:\n- The work is reliant on the video preprocessing. Right now this was demonstrated with video that's in-distribution and slightly out-of-distribution but not with video \"in the wild\". Given how sharply the performance drops when using non-Gibson data to train a Gibson agent, how would you expect this to perform with random youtube videos?\n- Do you consider the task solved with a 68% success rate in straight 1.5-3m navigation tasks?\n\nFor societal impact: This work, like many others, enables indoor navigation agents. That could lead to the development of more capable robotic assistants in the long run, especially since this work aims to lower the bar for entry by removing simulation and RL.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "SuRVU-jQf-h", "reviewer": "Reviewer_ncbG", "summary": "The paper addresses image-goal navigation, where an agent is placed in a novel environment, and is tasked with navigating to an (unknown) goal position that is specified using an image taken from that position. The method builds a topological map and uses three learned networks for planning navigation. The first one, G_D, is a GNN which takes the current topological graph and goal image, and predicts distances to goal for all nodes. The second one predicts visual features for unexplored nodes in the graph. The third one, G_T, is a termination predictor which estimates probability of goal within sight, and its relative pose. Finally, the system uses non-learned components: shortest distance planner, a local heuristic navigation policy G_LP for reaching subgoals, a heuristic module G_EA for expanding graph with unexplored nodes. The experiments are conducted in 2 environments: Gibson and MP3D. The method shows improvement with respect to 2 BC and 1 RL baseline.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The scores below are consistent with \"Rating\" legend.\n\n## Originality = 5/10\nThe proposed method is new. However, the paper would profit from a better embedding into the prior work. From the title \"No RL, No Simulation: Learning to Navigate without Navigating\" a reader could make a conclusion that it is the first time a passively learned navigation policy has shown success. However, there is prior work which already attempted to do this, albeit in a slightly more restricted setting. For example, SPTM [18] did learn its R and L networks passively, from data sampled by a random policy - so no RL-like feedback loop. It could also do it without a simulator, if it chose to have a local heuristic control policy instead of L-network. Of course, SPTM differs in test-time assumptions: it requires a short video of the test environment to begin with. This is the scientific \"delta\" of the proposed algorithm: it gets rid of this requirement by using a smart heuristic module G_EA for expanding the graph, and by a network which predicts visual features of newly expanded nodes. This is a nice research progress, but it requires a proper explanation of what is really new in this paper. If the authors choose to make such updates, I'm ready to increase my score.\n\n## Quality = 7/10\nThe quality of experiments in this paper is high. There are enough experiments to support the main claims and there are exhaustive ablations to determine the value of each introduced component. Perhaps, one nice-to-have additional experiment would be to add a random walk baseline, and also SPTM + random-walk exploration.\n\n## Clarity = 7/10\nThe paper is written in a nice and clear manner. A few questions though:\n1. What are step limits for RL and BC?\n2. [L111-112] \"egocentric pose estimate\" of what with respect to which origin?\n3. [L166-167] \"The local metric maps are discarded upon reaching the sub-goal\" - how to determine that the subgoal is reached? What if it's not reached in reality? What if we think we reached on subgoal, but instead we reached another visually similar subgoal (i.e. perceptual aliasing)?\n4. Could you please use bold font in tables to highlight the best number?\n\n## Significance = 6/10\nThe submission does advance state-of-the-art by getting rid of annoying assumptions in prior work. The slight concern though is that the real frontier now lies more in real-world applicability of these algorithms than in conceptual improvements in simulated environments, because there have already been many navigation papers in simulation. \"Scaling Local Control to Large-Scale Topological Navigation\" https://arxiv.org/pdf/1909.12329.pdf scaled more restricted SPTM-like ideas to the real world. I would be curious to see if the ideas in this submission scale to the real world as well.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical issues with this paper.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3h"}, "extra_text": {}}, {"review_id": "1UYAbisaT84", "reviewer": "Reviewer_D3yc", "summary": "The paper looks into offline data as the sole source of training actor policies for navigational agents. The proposed approach achieves this by a mix of learnable modules/networks and rule-based, heuristic ones. Moreover, the overarching goal is to reduce (to zero) the number of interactions in the simulator during training. The approach is compared with behavior cloning and end-to-end RL flat policy. The task chosen by the authors is image-goal navigation i.e. navigating to a specified image on data that they generate on the Habitat simulator.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**[Originality] (+) Model-based approaches are still under-explored for Habitat-like simulators:**\nThe rendering of RGB images is a computation bottleneck for current embodied AI research. While previous works have looked at distributed training (Wijmans et al. 2020), accelerating simulation (Savva et al. 2019), auxiliary losses (Ye et al. 2020), etc., this work investigates the promise of model-based approaches and with heuristic components.\n\n**[Clarity] (+) The approach is clearly explained and figures are informative:**\nThe explanation of the approach is done in a clear top-down way which makes understanding quite easy. Relevant background from existing work is included, making the manuscript self-sufficient. Moreover, the figures and associated captions illustrate the approach and idea very effectively.\n\n----\n\n**[Originality] (-) The paper doesn’t connect/compare to research on ‘Offline Reinforcement Learning’ and ‘Learning from Demonstrations’ (or Offline IL):**\n\nOffline RL, as thoroughly studied by multiple works, is aimed at utilizing passively collected offline data and needs no additional online interaction with the environment/simulator. Kindly refer to (Levine et al. 2020 arXiv:2005.01643) for a detailed tutorial, discussion of baselines, and open problems. While the submission is applying offline learning to visual navigation tasks with some existing neat modules on topological mapping and expansion, it fails to connect to the most relevant literature and claims offline learning in interactive environments as their key novelty. As mentioned in L245 “our key contribution is the ability to learn navigation agents from passive data.” This has been studied in existing work. Qualifying the claims of the work could be considered.\n\nParticularly relevant are (a) Kahn et al. 2020 (BADGR) that runs on self-supervised off-policy data gathered in real-world environments needing ‘no simulation’ or human supervision; (b) Fu et al. 2020 (D4RL) that includes offline CARLA visual simulator for autonomous driving. Please see Levine et al. 2020 for more references on navigation and robot manipulation. The reward in a dataset of offline demonstrations could be any metric inversely proportional to the distance between states (or nodes, in the case of this submission). Also, the distance estimator $\\mathcal{G}_{D}$ appears very similar to a critic estimating the value of a given state.\n\nAdditionally, the research in ‘learning from demonstrations/observations’ that learns from a mix of pre-collected demonstrations and interactions is relevant to the work. Hester et al. 2018 (DQfD),  Vecerik et al. 2017 (DDPGfD), and Nair et al. ICRA 2018 are relevant works for comparisons and grounding.\n\nConsidering domain shifts and using web/internet data as studied by Chang et al. [23] and also touched upon in Sec. 5.2 is a clear novel direction.\n\nQ. The reviewer requests the authors to clearly relate and contrast their _key contribution_ to the offline RL/IL literature pointed above. Also, kindly highlight novelty differences from [23] (that tackle real-world youtube videos instead of traversals of simuators) beyond using memory in distance prediction (L78)\n\n**[Originality] Request the authors to clearly distinguish what components in the _approach_ are novel.**\n\nDiscussing contributions beyond learning from passive offline data, several components in the approach are overlapping existing work on navigation in visual environments. The reviewer requests the authors clearly list the key contributions of the _approach_ with respect to [4,10]?\n\n**[Quality] (-) Concern about benchmark chosen for evaluation:**\nSince the demonstrations or training data used in the work is generated within the Habitat simulator, it’s straightforward to collect the same for the well-studied objectgoal or pointgoal navigation datasets. These tasks have been hosted within AIHabitat + (Gibson/MP3D scenes) with challenges run for 2 years at CVPR 2020 and 2021 workshops. Evaluating a new idea (of pure offline learning) on well-studied benchmarks and in the presence of reported results of previous baselines would add more credibility and also encourages the associated community to adopt this new idea. \n\nThe reviewer shares the concern of using a (presently nonstandard) image-goal navigation task when it has no particular advantage for this study over the well-studied embodied tasks. While image-goal navigation task has been studied before by [4] and [15], both these worked introduce their own implementation of the task and their own data. This paper, again, introduces a new realization of the task and a new dataset. Hence, all the baselines reported in Tables 1 and 2 are implemented by the authors themselves. While introducing a dataset is a great feat, it would benefit from being grounded in performance to other datasets, accompanied by rigorous analysis, and tests for any data biases. These are missing in the submission. \n\nQ. The reviewer is happy to improve the rating if the authors can report results on well-studied benchmarks in the same Habitat simulator they use (examples: CVPR 20/21 pointgoal/objectgoal navigation challenges or RxR/VLNCE challenge) and compare to the metrics reported by previously published work on these well-studied benchmarks.\n\n-----\n\n**[Quality] Questions about training:**\n* Q. **How are the predictions of the distance-prediction module made consistent with each other?** While $\\mathcal{G}_D$ predicts distances of nodes to the goal, how is the triangle inequality enforced? Eg. $\\mathcal{G}_D(\\cdot, \\cdot, n_1) = 3$; $\\text{travelcost}(n1, n2) = 2$, then $\\mathcal{G}_D(\\cdot, \\cdot, n_2) \\leq 5$. \n* Q. **Could the authors include metrics for intermediate checkpoints to understand if the training of end-to-end RL baseline was close to saturation:** (L276) 20 Mn frames for PPO seems less given that 75Mn frames are needed for PPO to learn simpler PointGoal. \n* Q. **Fairness of comparisons in Tab. 1 and 2**. End-to-end RL sees 10 and 20 Mn samples/steps for Tab. 1 and 2 (that too from an impoverished exploration policy). The supplement suggests 10-15 and 20-25 epochs over 1 and 2.5 Mn steps from a perfect exploration policy. Could the authors comment about the comparison between these two approaches in terms of the number of experiences seen or the number of gradient updates made?\n* Q. **Another way of no simulation**: Saving graphs with each pose was done in basic image-goal navigation in [11]. This too can train agents without simulating these observations. Could the authors comment on the disadvantages of this perfect model-based approach for training?\n\n**[Clarity] Minor comments (no impact on reviewer rating, solely for constructive feedback):**\n* Should the “navigate to sub-goal” step in Algorithm 1 have $\\rho_{sg}$ and $\\phi_{sg}$ instead of $\\rho_{i}$ and $\\phi_{i}$?\n* Also in Alg. 1, \"if $\\eta_s$ then\" suggests any non-zero probability leads to termination. Is there a threshold here?\n* The grammar in L217 is off.\n* Epochs of training should be included in L277.\n* L139: subscript could be changed to ‘t’ to match Fig. 4’s notation.\n* Figure 3: Notation for N is overloaded: -- the set of all nodes and the number of nodes.\n* L52: check grammar in “predicts the if the goal”.\n* Fig. 2: It could be clarified that “goal location” isn’t already known to the agent.\n* Tab. 4: Caption “passive video data, on Image-Goal…” → “passive video data, and tested on Image-Goal…”\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8-10 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a method for learning robot navigation policies avoiding simulations and reinforcement learning.

The paper was initially on the fence, generally appreciated by reviewers but with one of the reviewers pointing several weaknesses, which were discussed with the authors and among the reviewers. The main problems were a complete omission of a discussion of offline RL, and the general responses to the reviewers ("please forget about this") was quite hand-wavy. However, the authors did provide convincing answers to many issues, and in particular explaining that their setting is sufficiently different from RL (no reward, most importantly). Of course, this still means that these parts should have covered it in the related works section, and I strongly urge the authors to modify the paper.

It was also discussed that several baselines are missing, in particular work on topological memory, and the standard ImageGoal environment was still considered a restriction.

Issues on presentation and reproducibility remained. 

However, there was a near consensus that the work is sufficiently novel and nicely executed.
I recommend acceptance.
Decision: Accept (Poster)