Learning Exploration Policies for Navigation

1 INTRODUCTION . Imagine your first day at a new workplace . If you are like most people , the first task you set for yourself is to become familiar with the office so that the next day when you have to attend meetings and perform tasks , you can navigate efficiently and seamlessly . To achieve that goal , you explore your office without the task context of target locations you have to reach and build a generic understanding of space . This step of task-independent exploration is quite critical yet often ignored in current approaches for navigation . When it comes to navigation , currently there are two paradigms : ( a ) geometric reconstruction and path-planning based approaches ( Hartley & Zisserman , 2003 ; Thrun et al. , 2005 ; LaValle , 2006 ) , and ( b ) learning-based approaches ( Mirowski et al. , 2017 ; Gupta et al. , 2017 ; Savinov et al. , 2018 ; Zhu et al. , 2017 ) . SLAM-based approaches , first build a map and then use localization and path planning for navigation . In doing this , one interesting question that is often overlooked is : How does one build a map ? How should we explore the environment to build this map ? Current approaches either use a human operator to control the robot for building the map ( e.g . Thrun et al . ( 1999 ) ) , or use heuristics such as frontier-based exploration ( Yamauchi , 1997 ) . On the other hand for approaches that use learning , most only learn policies for specific tasks ( Zhu et al. , 2017 ; Mirowski et al. , 2017 ; Gupta et al. , 2017 ) , or assume environments have already been explored ( Savinov et al. , 2018 ) . Moreover , in context of such learning based approaches , the question of exploration is not only important at test time , but also at train time . And once again , this question is largely ignored . Current approaches either use sample inefficient random exploration or make impractical assumptions about full map availability for generating supervision from optimal trajectories . Thus , a big bottleneck for both these navigation paradigms is an exploration policy : a task-agnostic policy that explores the environment to either build a map or sample trajectories for learning a navigation policy in a sample-efficient manner . But how do we learn this task-independent policy ? What should be the reward for such a policy ? First possible way is to not use learning and use heuristic based approaches ( Yamauchi , 1997 ) . However , there are four issues with non-learning based approaches : ( a ) these approaches are brittle and fail when there is noise in ego-estimation or localization ; ( b ) they make strong assumptions about free-space/collisions and fail to generalize when navigation requires interactions such as opening doors etc . ; ( c ) they fail to capture semantic priors that can reduce search-space significantly ; and ( d ) they heavily rely on specialized sensors such as range scanners . Another possibility is to learn exploration policies on training environments . One way is to use reinforcement learning ( RL ) with intrinsic rewards . Examples of intrinsic rewards can be “ curiosity ” where prediction error is used as reward signal or “ diversity ” which discourages the agent from revisiting the same states . While this seems like an effective reward , such approaches are still sample inefficient due to blackbox reward functions that can ’ t be differentiated to compute gradients effectively . So , what would be an effective way to learn exploration policies for navigation ? In this paper , we propose an approach for learning policies for exploration for navigation . We explore this problem from multiple perspectives : ( a ) architectural design , ( b ) reward function design , and ( c ) reward optimization . Specifically , from perspective of reward function and optimization , we take the alternative paradigm and use supervision from human explorations in conjunction with intrinsic rewards . We notice that bootstrapping of learning from small amount of human supervision aids learning of semantics ( e.g . doors are pathways ) . It also provides a good initialization for learning using intrinsic rewards . From the perspective of architecture design , we explore how to use 3D information and use it efficiently while doing exploration . We study proxy rewards that characterize coverage and demonstrate how this reward outperforms other rewards such as curiosity . Finally , we show how experience gathered from our learned exploration policies improves performance at down-stream navigation tasks . 2 RELATED WORK . Our work on learning exploration policies for navigation in real world scenes is related to active SLAM in classical robotics , and intrinsic rewards based exploration in reinforcement learning . As we study the problem of navigation , we also draw upon recent efforts that use learning for this problem . We survey related efforts in these three directions . Navigation in Classical Robotics . Classical approaches to navigation operate by building a map of the environment , localizing the agent in this map , and then planning paths to convey the agent to desired target locations . Consequently , the problems of mapping , localization and path-planning have been very thoroughly studied ( Hartley & Zisserman , 2003 ; Thrun et al. , 2005 ; LaValle , 2006 ) . However , most of this research starts from a human-operated traversal of the environment , and falls under the purview of passive SLAM . Active SLAM , or how to automatically traverse a new environment for building spatial representations is much less studied . Cadena et al . ( 2016 ) present an excellent review of active SLAM literature , we summarize some key efforts here . Past works have formulated active SLAM as Partially Observable Markov Decision Processes ( POMDPs ) ( MartinezCantin et al. , 2009 ) , or as choosing actions that reduce uncertainty in the estimated map ( Carrillo et al. , 2012 ) . While these formulations enable theoretical analysis , they crucially rely on sensors to build maps and localize . Thus , such approaches are highly susceptible to measurement noise . Additionally , such methods treat exploration purely as a geometry problem , and entirely ignore semantic cues for exploration such as doors . Learning for Navigation . In order to leverage such semantic cues for navigation , recent works have formulated navigation as a learning problem ( Zhu et al. , 2017 ; Gupta et al. , 2017 ; Mirowski et al. , 2017 ; Savinov et al. , 2018 ) . A number of design choices have been investigated . For example , these works have investigated different policy architectures for representing space : Zhu et al . ( 2017 ) use feed-forward networks , Mirowski et al . ( 2017 ) use vanilla neural network memory , Gupta et al . ( 2017 ) use spatial memory and planning modules , and Savinov et al . ( 2018 ) use semi-parametric topological memory . Different training paradigms have also been explored : Gupta et al . ( 2017 ) learn to imitate behavior of an optimal expert , Mirowski et al . ( 2017 ) and Zhu et al . ( 2017 ) use extrinsic reward based reinforcement learning , Pathak et al . ( 2018 ) learn an inverse dynamics model on the demonstrated trajectory way-points from the expert , while Savinov et al . ( 2018 ) use selfsupervision . In our work here , we build on insights from these past works . Our policy architecture and reward definition use spatial memory to achieve long-horizon exploration , and we imitate human exploration demonstrations to boot-strap policy learning . However , in crucial distinction , instead of studying goal-directed navigation ( either in the form of going to a particular goal location , or object of interest ) , we study the problem of autonomous exploration of novel environments in a taskagnostic manner . In doing so , unlike past works , we do not assume access to human demonstrations in the given novel test environment like Savinov et al . ( 2018 ) , nor do we assume availability of millions of samples of experience or reward signals in the novel test environment like Zhu et al . ( 2017 ) or Mirowski et al . ( 2017 ) . Moreover , we do not depend on extrinsically defined reward signals for training . We derive them using on-board sensors . This makes our proposed approach amenable to real world deployment . Learning based approaches ( Gandhi et al. , 2017 ; Sadeghi & Levine , 2017 ) have also been used to learn low-level collision avoidance policies . While they do not use task-context , they learn to move towards open space , without the intent of exploring the whole environment . Zhang et al . ( 2017 ) uses a differentiable map structure to mimic the SLAM techniques . Such works are orthogonal to our effort on exploration as our exploration policy can benefit from their learned maps instead of only using reconstructed occupancy map . Exploration for Navigation . A notable few past works share a similar line of thought , and investigate exploration in context of reinforcement learning ( Schmidhuber , 1991 ; Stadie et al. , 2015 ; Pathak et al. , 2017 ; Fu et al. , 2017 ; Lopes et al. , 2012 ; Chentanez et al. , 2005 ) . These works design intrinsic reward functions to capture novelty of states or state-action transitions . Exploration policies are then learned by optimizing these reward functions using reinforcement learning . Our work is most similar to these works . However , we focus on this problem in the specific context of navigation in complex and realistic 3D environments , and propose specialized policy architectures and intrinsic reward functions . We experimentally demonstrate that these specializations improve performance , and how learning based exploration techniques may not be too far from real world deployment . Jayaraman & Grauman ( 2018 ) use a related reward function ( pixel reconstruction ) to learn policies to look around ( and subsequently solve tasks ) . However , they do it in context of 360◦ images , and their precise reward can ’ t be estimated intrinsically . Xu et al . ( 2017 ) generate smooth movement path for high-quality camera scan by using time-varying tensor fields . Bai et al . ( 2016 ) propose an information-theoretic exploration method using Gaussian Process regression and show experiments on simplistic map environments . Kollar & Roy ( 2008 ) assume access to the groundtruth map and learn an optimized trajectory that maximizes the accuracy of the SLAM-derived map . In contrast , our learning policy directly tells the action that the agent should take next and estimates the map on the fly . System Identification . Finally , the general idea of exploration prior to goal-driven behavior , is also related to the classical idea of system identification ( Ljung , 1987 ) . Recent interest in this idea with end-to-end learning ( Yu et al. , 2017 ; Duan et al. , 2016 ; Mishra et al. , 2018 ) has been shown to successfully adapt to novel mazes . In comparison , we tackle navigation in complex and realistic 3D environments , with very long spatio-temporal dependencies . 3 APPROACH . Let us consider a mobile agent that is equipped with an RGB-D camera and that can execute basic movement macro-actions . This agent has been dropped into a novel environment that it has never been in before . We want to learn an exploration policy πe that enables this agent to efficiently explore this new environment . A successful exploration of the new environment should enable the agent to accomplish down-stream tasks in this new environment efficiently . We formulate estimation of πe as a learning problem . We design a reward function that is estimated entirely using on-board sensors , and learn πe using RL . Crucially , πe is learned on a set of environments Etrain , and tested on a held-out set of environments Etest , i.e. , Etrain ⋂ Etest = φ . The key novelties in our work are : a ) the design of the policy πe , b ) the design of the reward function R , and c ) use of human demonstrations to speed up training of πe . The design of the policy and the reward function depend on a rudimentary map of the environment that we maintain over time . We first describe the map construction procedure , and then detail the aforementioned aspects .

Summary: This is a well explained and well executed paper on using classical SLAM-like 2D maps for helping a standard Deep RL navigation agent (convnet + LSTM) explore efficiently an environment and without the need for extrinsinc rewards. The agent relies on 3 convnets, one processing RGB images, one the image of a coarse map in egocentric referential, and one of the image of a fine-grained map in egocentric referential (using pre-trained ResNet-18 convnets). Features produced by the convnets are fed into a recurrent policy trained using PPO. Two rewards are used: the increase in the map's coverage and an obstacle avoidance penalty. The agent is further bootstrapped through imitation learning in a goal-driven task executed by a human controlling the agent. The authors analyze the behavior of the navigation algorithm by various ablations, a baseline consisting of Pathak's (2017) Intrinsic Curiosity Module-based navigation and, commendably, a classical SLAM baseline with path planning to empty, unexplored spaces.