No RL, No Simulation: Learning to Navigate without Navigating

1 Introduction . In recent years , we have seen significant advances in learning-based approaches for indoor navigation [ 1 , 2 ] . Impressive performance gains have been obtained for a range of tasks , from non-semantic point-goal navigation [ 3 ] to semantic tasks such as image-goal [ 4 ] and object-goal navigation [ 5 , 6 ] , via methods that use reinforcement learning ( RL ) . The effectiveness of RL for these tasks can be attributed in part to the emergence of powerful new simulators such as Habitat [ 7 ] , Matterport [ 8 ] and AI2Thor [ 9 ] . These simulators have helped scale learning to billions of frames by providing large-scale active interaction data and ground-truth maps for designing reward functions . But do we actually need simulation and RL to learn to navigate ? Is there an alternative way to formulate the navigation problem , such that no ground-truth maps or active interaction are required ? These are valuable questions to explore because learning navigation in simulation constrains the approach to a limited set of environments , since the creation of 3D assets remains costly and time-consuming . In this paper , we propose a self-supervised approach to learning how to navigate from passive egocentric videos . Our novel method is simple and scalable ( no simulator required for training ) , and at the same time highly effective , as it outperforms RL-based formulations by a significant margin . To introduce our approach , let us first examine the role of RL and simulation in standard navigation learning methods . In the standard RL formulation , an agent gets a reward upon reaching the goal , followed by a credit assignment stage to determine the most useful state-action pairs . But do we actually need the reinforce function for action credit assignment ? Going a step further , do we even need to learn a policy explicitly ? In navigation , we argue that the state space itself is highly structured via a distance function , and the structure itself could be leveraged for credit assignment . Simply put , states that help reduce the distance to the goal are better – and therefore predicted distance can be ∗Correspondence : meerahahn @ gatech.edu 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . used either as the value function or as a proxy for it . In fact , RL formulations frequently use ‘ distance reduced to goal ’ in reward shaping . The key property of our approach is that we learn a generalizable distance estimator directly from passive videos , and as a result we do not require any interaction . We demonstrate that an effective distance estimator can be learned directly from visual trajectories , without the need for an RL policy to map visual observations to the action space , thereby obviating the need for extensive interaction in a simulator and hand-designed rewards . However passive videos do not provide learning opportunities for obstacle avoidance since they rarely , if ever , consist of cameras bumping into walls . We forgo the need for active interaction to reason about collisions as we show that obstacle avoidance is only required locally and simple depth maps are sufficient to prune invalid actions and locations for navigation . More broadly , our approach can be considered as closely related to model-based control , which is an alternative paradigm to RL-based policy learning , with the key insight that components of the model and cost functions can be learned from passive data . No RL , No Simulator Approach ( NRNS ) : Our NRNS algorithm can be described as follows . During training we learn two functions from passive videos : ( a ) a geodesic distance estimator : given the state features and goal image , this function predicts the geodesic distance of the unexplored frontiers of the graph to the goal location . This enables a greedy policy in which we select the node with the least distance ; ( b ) a target prediction model : Given the goal image and the image from the agent ’ s current location , this function predicts if the goal is within sight and can be reached without collisions , along with the exact location of the goal . The key is that both the distance model and the target prediction model can be learned from passive RGBD videos , with SLAM used to estimate relative poses . We believe our simple NRNS approach should act as a strong baseline for any future approaches that use RL and simulation . We show that NRNS outperforms end-to-end RL , even when RL is trained using 5x more data and 10x more compute . Furthermore , unlike RL methods which need to be trained in simulation because they require substantial numbers of interactions , NRNS can be trained directly on real-world videos alone , and therefore does not suffer from the sim-to-real gap . 2 Related Work . Navigation in simulators . Navigation tasks largely fall into two main categories [ 1 ] , ones in which a goal location is known [ 11 , 12 , 13 ] and limited exploration is required , and ones in which the goal location is not known and efficient exploration is necessary . In the second category , tasks range from finding the location of specific objects [ 5 ] , rooms [ 14 ] , or images [ 15 ] , to the task of exploration itself [ 2 ] . The majority of current work [ 12 , 15 , 16 , 3 ] leverages simulators [ 7 ] and extensive interaction to learn end-to-end models for these tasks . In contrast , our work shows that the semantic cues needed for exploration-based navigation tasks can be learned directly from video trajectories . Navigation using passive data . Several prior works have tackled the navigation task when there is some passive experience available in the test environment [ 17 , 18 , 19 , 20 , 21 , 22 ] . A more limited number of works train navigation policies without simulation environments [ 19 , 22 , 23 ] . Unlike these works , we tackle the task of image goal navigation in unseen environments where there is no experience available to the agent in the test environments during training [ 19 , 22 ] and no map knowledge is given about the environments [ 23 ] . Two works adopting similar requirements are Chaplot et al . [ 4 ] and Chang et al . [ 24 ] . Most closely related is Neural Topological SLAM ( NTS ) [ 4 ] , which builds a topological map and estimates distance to the target image using a learned function . But NTS requires access to panoramic observations and ground-truth maps for training the distance function , making it much less scalable than our approach which can be trained with just video data with arbitrary field of view . Chang et al . [ 24 ] adopt a similar approach to NTS for object goal navigation , while also incorporating video data from YouTube for learning a Q function . A key difference is that our method learns a episodic distance function , utilizing all past observations to estimate distances to the target image . In comparison , Chaplot et al . [ 4 ] and Chang et al . [ 24 ] use a memory-less distance function operating only on the agent ’ s current observation . Prior work work on LfD [ 25 , 26 , 27 ] requires : ( a ) access to actions ( e.g . teleoperated in simulator ) ; and ( b ) optimal human demonstrations . The goal in LfD is to learn policies that match given demonstrations by either mimicking the actions or formulating rewards using the demonstrations . In contrast , NRNS does neither , since our passive training data does always contain optimal navigation trajectories and NRNS does not compute any rewards as it does not require RL or credit assignment . We make the same distinction between our work and offline RL [ 28 , 29 , 30 ] , which does not utilize online data gathering but still learns a policy using rewards from observed trajectories . In contrast , we do not use any rewards or learn a policy . Instead , our approach aims to learn a distance-to-goal model and uses a greedy policy based on the distance predictions . From this perspective , NRNS is performing model-based learning . Map-Based Navigation . There are multiple spatial representations which can be leveraged in solving navigation tasks . Metric maps , which maintain precise information on the occupied space in an environment , are commonly used for visual navigation [ 2 , 10 , 6 ] . However , metric maps suffer from issues of scalability and consistency under sensor noise . As a result , topological maps have recently gained traction [ 4 , 21 , 18 , 24 , 31 ] as a means to combat these issues . A significant difference from our approach is that in these prior works a topological map is given at the beginning of the navigation task , and is not created or changed during navigation . Specifically , Savinov et al . [ 18 ] create the map from a given 5 minute video of the test environment and Chen et al . [ 21 ] assume access to a ground truth map . In our image-goal navigation set up , the agent is given no information about the test environment . Chaplot et al . [ 4 ] do not require experience in the test environment and build topological maps at test time , but still requires access to a simulator for computing shortest-path distances between pairs of images . Additionally , topological maps for robotic navigation draw inspiration from both animal and human psychology . The cognitive map hypothesis proposes that the brain builds coarse internal spatial representations of environments [ 32 , 33 ] . Multiple works argue that this internal representation relies on landmarks [ 34 , 35 ] , making human cognitive maps more similar to topological maps as opposed to metric maps . Graph Neural Networks . Graph Neural Networks ( GNN ) are specifically used for modeling relational data in a non-Euclidean space . We employ a GNN for estimating distance-to-goal over the agent ’ s exploration frontier . Specifically we employ an augmented Graph Attention Network [ 36 ] which allows for a weighted aggregation of neighboring node information . Graph Networks are rarely-used for the task of topological map-based visual navigation . Savinov et al . [ 18 ] use a GNN for the sub-task of localizing the agent in a ground truth topological map . We believe we are the first to leverage graph neural networks in a visual navigation task . 3 Image Goal Navigation using Topological Graphs . We propose No RL , No Simulator “ NRNS ” , a hierarchical modular approach to image-goal navigation that consists of : a ) high-level modules for maintaining a topological map and using visual and semantic reasoning to predict sub-goals , and b ) heuristic low-level modules that use depth data to select low level navigation actions to reach sub-goals and determine geometrically-explorable areas . We first describe NRNS in detail and then show in Sec . 4 that the high-level modules can be trained without using any simulation , interaction or even ground-truth scans – i.e . that passive video data alone is sufficient to learn the semantic and visual reasoning models needed for navigation . 3.1 Formulation and Representation . Task Definition . We tackle the task of image-goal navigation , where an agent is placed in a novel environment , and is tasked with navigating to an ( unknown ) goal position that is specified using an image taken from that position , as shown in Fig . 2 . More formally , an episode begins with an agent receiving an RGB observation ( IG ) corresponding to the goal position . At each time step , t , of the episode the agent receives a set of observations st , and must take a navigation action at . The agents state observations , st , are defined as a narrow field of view RGBD image , It , and egocentric pose estimate , Pt . The agent must execute navigational actions to reach goal within a maximum number of steps . Topological Map Representation . The NRNS agent maintains a topological map in the form of a graph G ( N , E ) , defined by nodes N and edges E , which provides a sparse representation of the environment . Concretely , a node ni ∈ N is associated with a pose pi , defined by location and orientation . Each node ni can either be ‘ explored ’ i.e . the agent has previously visited the pose and obtained a corresponding RGBD image Ii , or ‘ unexplored ’ e.g . unvisited positions at the exploration frontier which may be visited in the future . Each edge e ∈ E connects a pair of adjacent nodes ni and nj . Nodes are deemed adjacent only if a short and ‘ simple ’ path exists between the two nodes , as further detailed in Sec . 3.3 . Each edge between adjacent nodes is then associated with the attribute ∆P – the relative pose between the two nodes .

Summary: The paper looks into offline data as the sole source of training actor policies for navigational agents. The proposed approach achieves this by a mix of learnable modules/networks and rule-based, heuristic ones. Moreover, the overarching goal is to reduce (to zero) the number of interactions in the simulator during training. The approach is compared with behavior cloning and end-to-end RL flat policy. The task chosen by the authors is image-goal navigation i.e. navigating to a specified image on data that they generate on the Habitat simulator.