Learning To Explore Using Active Neural SLAM

1 INTRODUCTION . Navigation is a critical task in building intelligent agents . Navigation tasks can be expressed in many forms , for example , point goal tasks involve navigating to specific coordinates and semantic navigation involves finding the path to a specific scene or object . Irrespective of the task , a core problem for navigation in unknown environments is exploration , i.e. , how to efficiently visit as much of the environment . This is useful for maximizing the coverage to give the best chance of finding the target in unknown environments or for efficiently pre-mapping environments on a limited time-budget . Recent work from Chen et al . ( 2019 ) has used end-to-end learning to tackle this problem . Their motivation is three-fold : a ) learning provides flexibility to the choice of input modalities ( classical systems rely on observing geometry through the use of specialized sensors , while learning systems can infer geometry directly from RGB images ) , b ) use of learning can improve robustness to errors in explicit state estimation , and c ) learning can effectively leverage structural regularities of the real world , leading to more efficient behavior in previously unseen environments . This lead to their design of an end-to-end trained neural network-based policy that processed raw sensory observations to directly output actions that the agent should execute . While the use of learning for exploration is well-motivated , casting the exploration problem as an end-to-end learning problem has its own drawbacks . Learning about mapping , state-estimation , and path-planning purely from data in an end-to-end manner can be prohibitively expensive . Consequently , past end-to-end learning work for exploration from Chen et al . ( 2019 ) relies on the use of imitation learning and many millions of frames of experience , but still performs worse than classical methods that don ’ t require any training at all . This motivates our work . In this paper , we investigate alternate formulations of employing learning for exploration that retains the advantages that learning has to offer , but doesn ’ t suffer from the †Correspondence : chaplot @ cs.cmu.edu ∗Equal Contribution drawbacks of full-blown end-to-end learning . Our key conceptual insight is that use of learning for leveraging structural regularities of indoor environments , robustness to state-estimation errors , and flexibility with respect to input modalities , happens at different time scales and can thus be factored out . This motivates the use of learning in a modular and hierarchical fashion inside of what one may call a ‘ classical navigation pipeline ’ . This results in navigation policies that can work with raw sensory inputs such as RGB images , are robust to state estimation errors , and leverage the regularities of real-world layouts . This results in extremely competitive performance over both geometry-based methods and recent learning-based methods ; at the same time requiring a fraction of the number of samples . More specifically , our proposed exploration architecture comprises of a learned Neural SLAM module , a global policy , and a local policy , that are interfaced via the map and an analytical path planner . The learned Neural SLAM module produces free space maps and estimates agent pose from input RGB images and motion sensors . The global policy consumes this free-space map with the agent pose and employs learning to exploit structural regularities in layouts of real-world environments to produce long-term goals . These long-term goals are used to generate short-term goals for the local policy ( using a geometric path-planner ) . This local policy uses learning to directly map raw RGB images to actions that the agent should execute . Use of learning in the SLAM module provides flexibility with respect to input modality , learned global policy can exploit regularities in layouts of real-world environments , while learned local policies can use visual feedback to exhibit more robust behavior . At the same time , hierarchical and modular design and use of analytical planning , significantly cuts down the search space during training , leading to better performance as well as sample efficiency . We demonstrate our proposed approach in visually and physically realistic simulators for the task of geometric exploration ( visit as much area as possible ) . We work with the Habitat simulator from Savva et al . ( 2019 ) . While Habitat is already visually realistic ( it uses real-world scans from Chang et al . ( 2017 ) and Xia et al . ( 2018 ) as environments ) , we improve its physical realism by using actuation and odometry sensor noise models , that we collected by conducting physical experiments on a real mobile robot . Our experiments and ablations in this realistic simulation reveal the effectiveness of our proposed approach for the task of exploration . A straightforward modification of our method also tackles point-goal navigation tasks , and won the AI Habitat challenge at CVPR2019 across all tracks . 2 RELATED WORK . Navigation has been well studied in classical robotics . There has been a renewed interest in the use of learning to arrive at navigation policies , for a variety of tasks . Our work builds upon concepts in classical robotics and learning for navigation . We survey related works below . Navigation Approaches . Classical approaches to navigation break the problem into two parts : mapping and path planning . Mapping is done via simultaneous localization and mapping ( Thrun et al. , 2005 ; Hartley and Zisserman , 2003 ; Fuentes-Pacheco et al. , 2015 ) , by fusing information from multiple views of the environment . While sparse reconstruction can be done well with monocular RGB images ( Mur-Artal and Tardós , 2017 ) , dense mapping is inefficient ( Newcombe et al. , 2011 ) or requires specialized scanners such as Kinect ( Izadi et al. , 2011 ) . Maps are used to compute paths to goal locations via path planning ( Kavraki et al. , 1996 ; Lavalle and Kuffner Jr , 2000 ; Canny , 1988 ) . These classical methods have inspired recent learning-based techniques . Researchers have designed neural network policies that reason via spatial representations ( Gupta et al. , 2017 ; Parisotto and Salakhutdinov , 2018 ; Zhang et al. , 2017 ; Henriques and Vedaldi , 2018 ; Gordon et al. , 2018 ) , topological representations ( Savinov et al. , 2018 ; 2019 ) , or use differentiable and trainable planners ( Tamar et al. , 2016 ; Lee et al. , 2018 ; Gupta et al. , 2017 ; Khan et al. , 2017 ) . Our work furthers this research , and we study a hierarchical and modular decomposition of the problem and employ learning inside these components instead of end-to-end learning . Research also focuses on incorporating semantics in SLAM ( Pronobis and Jensfelt , 2012 ; Walter et al. , 2013 ) . Exploration in Navigation . While a number of works focus on passive map-building , path planning and goal-driven policy learning , a much smaller body of work tackles the the problem of active SLAM , i.e. , how to actively control the camera for map building . We point readers to FuentesPacheco et al . ( 2015 ) for a detailed survey , and summarize the major themes below . Most such works frame this problem as a Partially Observable Markov Decision Process ( POMDP ) that are approximately solved ( Martinez-Cantin et al. , 2009 ; Kollar and Roy , 2008 ) , and or seek to find a sequence of actions that minimizes uncertainty of maps ( Stachniss et al. , 2005 ; Carlone et al. , 2014 ) . Another line of work explores by picking vantage points ( such as on the frontier between explored and unexplored regions ( Dornhege and Kleiner , 2013 ; Holz et al. , 2010 ; Yamauchi , 1997 ; Xu et al. , 2017 ) ) . Recent works from Chen et al . ( 2019 ) ; Savinov et al . ( 2019 ) ; Fang et al . ( 2019 ) attack this problem via learning . Our proposed modular policies unify the last two lines of research , and we show improvements over representative methods from both these lines of work . Exploration has also been studied more generally in RL in the context of exploration-exploitation trade-off ( Sutton and Barto , 2018 ; Kearns and Singh , 2002 ; Auer , 2002 ; Jaksch et al. , 2010 ) . Hierarchical and Modular Policies . Hierarchical RL ( Dayan and Hinton , 1993 ; Sutton et al. , 1999 ; Barto and Mahadevan , 2003 ) is an active area of research , aimed at automatically discovering hierarchies to speed up learning . However , this has proven to be challenging , and thus most work has resorted to using hand-defining hierarchies . For example in the context of navigation , Bansal et al . ( 2019 ) and Kaufmann et al . ( 2019 ) design modular policies for navigation , that interface learned policies with low-level feedback controllers . Hierarchical and modular policies have also been used for Embodied Question Answering ( Das et al. , 2018a ; Gordon et al. , 2018 ; Das et al. , 2018b ) . 3 TASK SETUP . We follow the exploration task setup proposed by Chen et al . ( 2019 ) where the objective is to maximize the coverage in a fixed time budget . The coverage is defined as the total area in the map known to be traversable . Our objective is to train a policy which takes in an observation st at each time step t and outputs a navigational action at to maximize the coverage . We try to make our experimental setup in simulation as realistic as possible with the goal of transferring trained policies to the real world . We use the Habitat simulator ( Savva et al. , 2019 ) with the Gibson ( Xia et al. , 2018 ) and Matterport ( MP3D ) ( Chang et al. , 2017 ) datasets for our experiments . Both Gibson and Matterport datasets are based on real-world scene reconstructions are thus significantly more realistic than synthetic SUNCG dataset ( Song et al. , 2017 ) used for past research on exploration ( Chen et al. , 2019 ; Fang et al. , 2019 ) . In addition to synthetic scenes , prior works on learning-based navigation have also assumed simplistic agent motion . Some works limit agent motion on a grid with 90 degree rotations ( Zhu et al. , 2017 ; Gupta et al. , 2017 ; Chaplot et al. , 2018 ) . Other works which implement fine-grained control , typically assume unrealistic agent motion with no noise ( Savva et al. , 2019 ) or perfect knowledge of agent pose ( Chaplot et al. , 2016 ) . Since the motion is simplistic , it becomes trivial to estimate the agent pose in most cases even if it is not assumed to be known . The reason behind these assumptions on agent motion and pose is that motion and sensor noise models are not known . In order to relax both these assumptions , we collect motion and sensor data in the real-world and implement more realistic agent motion and sensor noise models in the simulator as described in the following subsection . 3.1 ACTUATION AND SENSOR NOISE MODEL . We represent the agent pose by ( x , y , o ) where x and y represent the xy co-ordinate of the agent measured in metres and o represents the orientation of the agent in radians ( measured counterclockwise from x-axis ) . Without loss of generality , assume agents starts at p0 = ( 0 , 0 , 0 ) . Now , suppose the agent takes an action at . Each action is implemented as a control command on a robot . Let the corresponding control command be ∆ua = ( xa , ya , oa ) . Let the agent pose after the action be p1 = ( x ? , y ? , o ? ) . The actuation noise ( act ) is the difference between the actual agent pose ( p1 ) after the action and the intended agent pose ( p0 + ∆u ) : act = p1 − ( p0 + ∆u ) = ( x ? − xa , y ? − ya , o ? − oa ) Mobile robots typically have sensors which estimate the robot pose as it moves . Let the sensor estimate of the agent pose after the action be p′1 = ( x ′ , y′ , o′ ) . The sensor noise ( sen ) is given by the difference between the sensor pose estimate ( p′1 ) and the actual agent pose ( p1 ) : sen = p ′ 1 − p1 = ( x′ − x ? , y′ − y ? , o′ − o ? ) In order to implement the actuation and sensor noise models , we would like to collect data for navigational actions in the Habitat simulator . We use three default navigational actions : Forward : move forward by 25cm , Turn Right : on the spot rotation clockwise by 10 degrees , and Turn Left : on the spot rotation counter-clockwise by 10 degrees . The control commands are implemented as uForward = ( 0.25 , 0 , 0 ) , uRight : ( 0 , 0 , −10 ∗ π/180 ) and uLeft : ( 0 , 0 , 10 ∗ π/180 ) . In practice , a robot can also rotate slightly while moving forward and translate a bit while rotating on-the-spot , creating rotational actuation noise in forward action and similarly , a translation actuation noise in on-the-spot rotation actions . We use a LoCoBot1 to collect data for building the actuation and sensor noise models . We use the pyrobot API ( Murali et al. , 2019 ) along with ROS ( Quigley et al. , 2009 ) to implement the control commands and get sensor readings . For each action a , we fit a separate Gaussian Mixture Model for the actuation noise and sensor noise , making a total of 6 models . Each component in these Gaussian mixture models is a multi-variate Gaussian in 3 variables , x , y and o . For each model , we collect 600 datapoints . The number of components in each Gaussian mixture model is chosen using cross-validation . We implement these actuation and sensor noise models in the Habitat simulator for our experiments . We have released the noise models , along with their implementation in the Habitat simulator in the open-source code .

Summary: The paper describes a method for visual robot navigation in simulated environments. In terms of overall objectives and targeted reasoning, the current approaches can be roughly divided into two groups: i) learning tasks requiring high-level reasoning for navigation involving the detection and discovery of objects and their affordances and eventually also requiring to process language input, and ii) simpler navigation task involving geometry and the detection of free (navigable space): point goal, maximizing coverage etc. The former target more complex problems but the agents are more difficult (currently up to impossible) to transfer to real environments, whereas the latter directly target problems which can currently realistically used in real world scenarios.