Year: 2019

Abstract:
Numerous past works have tackled the problem of task-driven navigation. But, how to effectively explore a new environment to enable a variety of down-stream tasks has received much less attention. In this work, we study how agents can autonomously explore realistic and complex 3D environments without the context of task-rewards. We propose a learning-based approach and investigate different policy architectures, reward functions, and training paradigms. We find that use of policies with spatial memory that are bootstrapped with imitation learning and finally finetuned with coverage rewards derived purely from on-board sensors can be effective at exploring novel environments. We show that our learned exploration policies can explore better than classical approaches based on geometry alone and generic learning-based exploration techniques. Finally, we also show how such task-agnostic exploration can be used for down-stream tasks. Videos are available at: https://sites.google.com/view/exploration-for-nav/.

heading: 1 INTRODUCTION
text: Imagine your first day at a new workplace. If you are like most people, the first task you set for yourself is to become familiar with the office so that the next day when you have to attend meetings and perform tasks, you can navigate efficiently and seamlessly. To achieve that goal, you explore your office without the task context of target locations you have to reach and build a generic understanding of space. This step of task-independent exploration is quite critical yet often ignored in current approaches for navigation.
When it comes to navigation, currently there are two paradigms: (a) geometric reconstruction and path-planning based approaches (Hartley & Zisserman, 2003; Thrun et al., 2005; LaValle, 2006), and (b) learning-based approaches (Mirowski et al., 2017; Gupta et al., 2017; Savinov et al., 2018; Zhu et al., 2017). SLAM-based approaches, first build a map and then use localization and path planning for navigation. In doing this, one interesting question that is often overlooked is: How does one build a map? How should we explore the environment to build this map? Current approaches either use a human operator to control the robot for building the map (e.g. Thrun et al. (1999)), or use heuristics such as frontier-based exploration (Yamauchi, 1997). On the other hand for approaches that use learning, most only learn policies for specific tasks (Zhu et al., 2017; Mirowski et al., 2017; Gupta et al., 2017), or assume environments have already been explored (Savinov et al., 2018). Moreover, in context of such learning based approaches, the question of exploration is not only important at test time, but also at train time. And once again, this question is largely ignored. Current approaches either use sample inefficient random exploration or make impractical assumptions about full map availability for generating supervision from optimal trajectories.
Thus, a big bottleneck for both these navigation paradigms is an exploration policy: a task-agnostic policy that explores the environment to either build a map or sample trajectories for learning a navigation policy in a sample-efficient manner. But how do we learn this task-independent policy? What should be the reward for such a policy? First possible way is to not use learning and use heuristic based approaches (Yamauchi, 1997). However, there are four issues with non-learning based approaches: (a) these approaches are brittle and fail when there is noise in ego-estimation or localization; (b) they make strong assumptions about free-space/collisions and fail to generalize when navigation requires interactions such as opening doors etc.; (c) they fail to capture semantic priors that can reduce search-space significantly; and (d) they heavily rely on specialized sensors such as range scanners. Another possibility is to learn exploration policies on training environments. One
way is to use reinforcement learning (RL) with intrinsic rewards. Examples of intrinsic rewards can be “curiosity” where prediction error is used as reward signal or “diversity” which discourages the agent from revisiting the same states. While this seems like an effective reward, such approaches are still sample inefficient due to blackbox reward functions that can’t be differentiated to compute gradients effectively. So, what would be an effective way to learn exploration policies for navigation?
In this paper, we propose an approach for learning policies for exploration for navigation. We explore this problem from multiple perspectives: (a) architectural design, (b) reward function design, and (c) reward optimization. Specifically, from perspective of reward function and optimization, we take the alternative paradigm and use supervision from human explorations in conjunction with intrinsic rewards. We notice that bootstrapping of learning from small amount of human supervision aids learning of semantics (e.g. doors are pathways). It also provides a good initialization for learning using intrinsic rewards. From the perspective of architecture design, we explore how to use 3D information and use it efficiently while doing exploration. We study proxy rewards that characterize coverage and demonstrate how this reward outperforms other rewards such as curiosity. Finally, we show how experience gathered from our learned exploration policies improves performance at down-stream navigation tasks.

heading: 2 RELATED WORK
text: Our work on learning exploration policies for navigation in real world scenes is related to active SLAM in classical robotics, and intrinsic rewards based exploration in reinforcement learning. As we study the problem of navigation, we also draw upon recent efforts that use learning for this problem. We survey related efforts in these three directions.
Navigation in Classical Robotics. Classical approaches to navigation operate by building a map of the environment, localizing the agent in this map, and then planning paths to convey the agent to desired target locations. Consequently, the problems of mapping, localization and path-planning have been very thoroughly studied (Hartley & Zisserman, 2003; Thrun et al., 2005; LaValle, 2006). However, most of this research starts from a human-operated traversal of the environment, and falls under the purview of passive SLAM. Active SLAM, or how to automatically traverse a new environment for building spatial representations is much less studied. Cadena et al. (2016) present an excellent review of active SLAM literature, we summarize some key efforts here. Past works have formulated active SLAM as Partially Observable Markov Decision Processes (POMDPs) (MartinezCantin et al., 2009), or as choosing actions that reduce uncertainty in the estimated map (Carrillo et al., 2012). While these formulations enable theoretical analysis, they crucially rely on sensors to build maps and localize. Thus, such approaches are highly susceptible to measurement noise. Additionally, such methods treat exploration purely as a geometry problem, and entirely ignore semantic cues for exploration such as doors.
Learning for Navigation. In order to leverage such semantic cues for navigation, recent works have formulated navigation as a learning problem (Zhu et al., 2017; Gupta et al., 2017; Mirowski et al., 2017; Savinov et al., 2018). A number of design choices have been investigated. For example, these works have investigated different policy architectures for representing space: Zhu et al. (2017) use feed-forward networks, Mirowski et al. (2017) use vanilla neural network memory, Gupta et al. (2017) use spatial memory and planning modules, and Savinov et al. (2018) use semi-parametric topological memory. Different training paradigms have also been explored: Gupta et al. (2017) learn to imitate behavior of an optimal expert, Mirowski et al. (2017) and Zhu et al. (2017) use extrinsic reward based reinforcement learning, Pathak et al. (2018) learn an inverse dynamics model on the demonstrated trajectory way-points from the expert, while Savinov et al. (2018) use selfsupervision. In our work here, we build on insights from these past works. Our policy architecture and reward definition use spatial memory to achieve long-horizon exploration, and we imitate human exploration demonstrations to boot-strap policy learning. However, in crucial distinction, instead of studying goal-directed navigation (either in the form of going to a particular goal location, or object of interest), we study the problem of autonomous exploration of novel environments in a taskagnostic manner. In doing so, unlike past works, we do not assume access to human demonstrations in the given novel test environment like Savinov et al. (2018), nor do we assume availability of millions of samples of experience or reward signals in the novel test environment like Zhu et al. (2017) or Mirowski et al. (2017). Moreover, we do not depend on extrinsically defined reward
signals for training. We derive them using on-board sensors. This makes our proposed approach amenable to real world deployment. Learning based approaches (Gandhi et al., 2017; Sadeghi & Levine, 2017) have also been used to learn low-level collision avoidance policies. While they do not use task-context, they learn to move towards open space, without the intent of exploring the whole environment. Zhang et al. (2017) uses a differentiable map structure to mimic the SLAM techniques. Such works are orthogonal to our effort on exploration as our exploration policy can benefit from their learned maps instead of only using reconstructed occupancy map.
Exploration for Navigation. A notable few past works share a similar line of thought, and investigate exploration in context of reinforcement learning (Schmidhuber, 1991; Stadie et al., 2015; Pathak et al., 2017; Fu et al., 2017; Lopes et al., 2012; Chentanez et al., 2005). These works design intrinsic reward functions to capture novelty of states or state-action transitions. Exploration policies are then learned by optimizing these reward functions using reinforcement learning. Our work is most similar to these works. However, we focus on this problem in the specific context of navigation in complex and realistic 3D environments, and propose specialized policy architectures and intrinsic reward functions. We experimentally demonstrate that these specializations improve performance, and how learning based exploration techniques may not be too far from real world deployment. Jayaraman & Grauman (2018) use a related reward function (pixel reconstruction) to learn policies to look around (and subsequently solve tasks). However, they do it in context of 360◦ images, and their precise reward can’t be estimated intrinsically. Xu et al. (2017) generate smooth movement path for high-quality camera scan by using time-varying tensor fields. Bai et al. (2016) propose an information-theoretic exploration method using Gaussian Process regression and show experiments on simplistic map environments. Kollar & Roy (2008) assume access to the groundtruth map and learn an optimized trajectory that maximizes the accuracy of the SLAM-derived map. In contrast, our learning policy directly tells the action that the agent should take next and estimates the map on the fly.
System Identification. Finally, the general idea of exploration prior to goal-driven behavior, is also related to the classical idea of system identification (Ljung, 1987). Recent interest in this idea with end-to-end learning (Yu et al., 2017; Duan et al., 2016; Mishra et al., 2018) has been shown to successfully adapt to novel mazes. In comparison, we tackle navigation in complex and realistic 3D environments, with very long spatio-temporal dependencies.

heading: 3 APPROACH
text: Let us consider a mobile agent that is equipped with an RGB-D camera and that can execute basic movement macro-actions. This agent has been dropped into a novel environment that it has never been in before. We want to learn an exploration policy πe that enables this agent to efficiently
explore this new environment. A successful exploration of the new environment should enable the agent to accomplish down-stream tasks in this new environment efficiently.
We formulate estimation of πe as a learning problem. We design a reward function that is estimated entirely using on-board sensors, and learn πe using RL. Crucially, πe is learned on a set of environments Etrain, and tested on a held-out set of environments Etest, i.e., Etrain ⋂ Etest = φ.
The key novelties in our work are: a) the design of the policy πe, b) the design of the reward function R, and c) use of human demonstrations to speed up training of πe. The design of the policy and the reward function depend on a rudimentary map of the environment that we maintain over time. We first describe the map construction procedure, and then detail the aforementioned aspects.

heading: 3.1 MAP CONSTRUCTION
text: As the agent moves around, it uses its depth camera to build and update a map of the world around it. This is done by maintaining an estimate of the agent’s location over time and projecting 3D points observed in the depth image into an allocentric map of the environment.
More specifically, let us assume that the agent’s estimate of its current location at time step t is x̂t, and that the agent starts from origin (i.e. x̂0 = 0). When the agent executes an action at at time step t, it updates its position estimate through a known transition function f , i.e., x̂t+1 = f(x̂t, at).
The depth image observation at time step t+1, Dt+1 is back-projected into 3D points using known camera intrinsic parameters. These 3D points are transformed using the estimate x̂t+1, and projected down to generate a 2D map of the environment (that tracks what parts of space are known to be traversable or known to be non-traversable or unknown). Note that, because of slippage in wheels etc., x̂t+1 may not be the same as the true relative location of the agent from start of episode xt+1. This leads to aliasing in the generated map. We do not use expensive non-linear optimization (bundle adjustment) to improve our estimate x̂t+1, but instead rely on learning to provide robustness against this mis-alignment. We use the new back-projected 3D points to update the current map Mt and to obtain an updated map Mt+1.

heading: 3.2 POLICY ARCHITECTURE
text: Before we describe our policy architecture, let’s define what are the features that a good exploration policy needs: (a) good exploration of a new environment requires an agent to meaningfully move around by detecting and avoiding obstacles; (b) good policy also requires the agent to identify semantic cues such as doors that may facilitate exploration; (c) finally, it requires the agent to keep track of what parts of the environment have or have not been explored, and to estimate how to get to parts of the environment that may not have been explored.
This motivates our policy architecture. We fuse information from RGB image observations and occupancy grid-based maps. Information from the RGB image allows recognition of useful semantic cues. While information from the occupancy maps allows the agent to keep track of parts of the environment that have or have not been explored and to plan paths to unexplored regions without bumping into obstacles.
The policy architecture is shown in Fig. 1. We describe it in detail here:
1. Information from RGB images: RGB images are processed through a CNN. We use a ResNet18 CNN that has been pre-trained on the ImageNet classification task, and can identify semantic concepts in images.
2. Information from Occupancy Maps: We derive an occupancy map from past observations (as described in Sec. 3.1), and use it with a ResNet-18 CNN to extract features. To simplify learning, we do not use the allocentric map, but transform it into an egocentric map using the estimated position x̂t (such that the agent is always at the center of the map, facing upwards). This map canonicalization aids learning. It allows the CNN to not only detect unexplored space but to also locate it with respect to its current location. Additionally, we use two such egocentric maps, a coarse map that captures information about a 40m × 40m area around the agent, and a detailed map that describes a 4m × 4m neighborhood. Some of these design choices were inspired by
recent work from Gupta et al. (2017), Parisotto & Salakhutdinov (2018) and Henriques & Vedaldi (2018), though we make some simplifications.
3. Recurrent Policy: Information from the RGB image and maps is fused and passed into an RNN. Recurrence can allow the agent to exhibit coherent behavior.

heading: 3.3 COVERAGE REWARD
text: We now describe the intrinsic reward function that we use to train our exploration policy. We derive this intrinsic rewards from the map Mt, by computing the coverage. Coverage C(Mt) is defined as the total area in the map that is known to be traversable or known to be non-traversable. Reward Rcovint (t) at time step t is obtained via gain in coverage in the map: C(Mt+1) − C(Mt). Intuitively, if the current observation adds no obstacles or free-space to the map then it adds no information and hence no reward is given. We also use a collision penalty, that is estimated using the bump sensor, Rcollint (t) = −Bump(t + 1), where Bump(t + 1) denotes if a collision occurred while executing action at. Rcovint and R coll int are combined to obtain the total reward.

heading: 3.4 TRAINING PROCEDURE
text: Finally, we describe how we optimize the policy. Navigating in complex realistic 3D houses, requires long-term coherent behavior over multiple time steps, such as exiting a room, going through doors, going down hallways. Such long-term behavior is hard to learn using reinforcement learning, given sparsity in reward. This leads to excessively large sample complexity for learning. To overcome this large sample complexity, we pre-train our policy to imitate human demonstrations of how to explore a new environment. We do this using trajectories collected from AMT workers as they were answering questions in House3D (Das et al., 2018). We ignore the question that was posed to the human, and treat these trajectories as a proxy for how a human will explore a previosuly unseen environment. After this phase of imitation learning, πe is further trained via policy gradients (Williams, 1992) using proximal policy optimization (PPO) from Schulman et al. (2017).

heading: 4 EXPERIMENTS
text: The goal of this paper is to build agents that can autonomously explore novel complex 3D environments. We want to understand this in context of the different choices we made in our design, as well as how our design compares to alternate existing techniques for exploration. While coverage of the novel environment is a good task-agnostic metric, we also design experiments to additionally quantify the utility of generic task-agnostic exploration for downstream tasks of interest. We first describe our experimental setup that consists of complex realistic 3D environments and emphasizes the study of generalization to novel environments. We then describe experiments that measure task-agnostic exploration via coverage. And finally, we present experiments where we use different exploration schemes for downstream navigation tasks.

heading: 4.1 EXPERIMENTAL SETUP
text: We conducted our experiments on the House3D simulation environment Wu et al. (2018). House 3D is based on realistic apartment layouts from the SUNCG dataset Song et al. (2017) and simulates first-person observations and actions of a robotic agent embodied in these apartments. We use 20 houses each for training and testing. These sets are sampled from the respective sets in House 3D, and do not overlap. That is, testing is done on a set of houses not seen during training. This allows us to study generalization, i.e., how well our learned policies perform in novel, previously unseen houses. We made one customization to the House 3D environment: by default doors in House 3D are rendered but not used for collision checking. We modified House 3D to also not render doors, in addition to not using them for collision checking.
Observation Space. We assume that the agent has an RGB-D camera with a field of view of 60◦, and a bump sensor. The RGB-D camera returns a regular RGB image and a depth image that records the distance (depth information is clipped at 3m) of each pixel from the camera. The bump sensor returns if a collision happened while executing the previous action.
Action Space. We followed the same action space as in EmbodiedQA (Das et al., 2018), with 6 motion primitives: move forward 0.25m, move backward 0.25m, strafe left 0.25m, strafe right 0.25m, turn left 9◦, and turn right 9◦.
Extrinsic Environmental Reward. We do not use any externally specified reward signals from the environment: Rext(t) = 0.
Intrinsic Reward. As described in Sec. 3.3, the intrinsic reward of the agent is based upon the map that it constructs as it moves around (as described in Sec. 3.1), and readings from the bump sensor. RewardRint(t) = αRcovint(t)+βR coll int(t). HereR cov int is the coverage reward, andR coll int is the collision reward. α, β are hyper-parameters to trade-off how aggressive the agent is.
Training. As described in Sec. 3.4, we train policies using imitation of human trajectories and RL.
1. Imitation from Human Exploration Trajectories. We leverage human exploration trajectories collected from AMT workers as they answered questions for the EmbodiedQA task (Das et al., 2018). We ignore the question that was posed to the AMT worker, and train our policy to simply mimic the actions that the humans took. As the humans were trying to answer these questions in previously unseen environments, we assume that these trajectories largely exhibit exploration behavior. We used a total of 693 human trajectories for this imitation.
2. Reinforcement Learning: After learning via imitation, we further train the policy using reinforcement learning on the training houses. We use PPO (Schulman et al., 2017) to optimize the intrinsic reward defined above. At the start of each episode, the agent is initialized at a random location inside one of the training houses. Each episode is run for 500 time-steps. We run a total of 6400 episodes which amounts to a total of 3.2M steps of experience.
Baselines. We next describe the various baseline methods that we experimented with. We implemented a classical baseline that purely reasons using geometry, and a learning baseline that uses curiosity for exploration.
1. Frontier-based Exploration. As a classical baseline, we experimented with frontier-based exploration (Yamauchi, 1997; Dornhege & Kleiner, 2013). This is a purely geometric method that utilizes the built map Mt. Every iteration, it samples a point in currently unexplored space, and plans a path towards it from the current location (unobserved space is assumed to be free). As the plan is executed, both the map and the plan are updated. Once the chosen point is reached, this process is repeated.
2. Curiosity-based Exploration. The next baseline we tried was curiosity-based exploration. In particular, we use the version proposed by Pathak et al. (2017) that uses prediction error of a forward model as reward. We use the modifications proposed by Burda et al. (2019), and only train a forward model. We prevent degeneracy in forward model by learning it in the fixed feature space of a ResNet-18 model that has been pre-trained on the ImageNet classification task.

heading: 4.2 COVERAGE QUALITY
text: We first measure exploration by itself, by measuring the true coverage of the agent. We compute the true coverage using the map as described in Sec. 3.1, except for using the true location of the agent rather than the estimated location (i.e. xt instead of x̂t). We study the following three scenarios:
Without Estimation Noise: We first evaluate the performance of the agent that is trained and tested without observation noise, that is, x̂t = xt. Note that this setting is not very realistic as there is always observation error in an agent’s estimate of its location. It is also highly favorable to the frontier-based exploration agent, which very heavily relies on the accuracy of its maps. Fig. 2(left) presents the performance of different policies for this scenario. We first note that the curiosity based agent (IL + RL with Curiosity) explores better than a random exploration policy (that executes a random action at each time step). Straight is a baseline where the agent moves along a straight line and executes a random number of 9◦ turns when a collision occurs, which is a strategy used by many robot vacuums. Such strategy does not require RGB or depth information, and performs better than the curiosity based agent. However, both policies are worse than an RGB only version of our method, an RGB only policy that is trained with our coverage reward (IL + RL with RGB). Our full system (IL + RL with Map + RGB) that also uses maps as input performs even better.
Frontier-based exploration (Frontier-based Explr) has the best performance in this scenario. As noted, this is to expect as this method gets access to perfect, fully-registered maps, and employs optimal path planning algorithm to move from one place to another. It is also worth noting that, it is hard to use such classical techniques in situations where we only have a RGB images. In contrast, learning allows us to easily arrive at policies that can explore using only RGB images at test time.
With Estimation Noise: We next describe a more realistic scenario that has estimation noise, i.e. x̂t+1 is estimated using a noisy dynamics function. In particular, we add truncated Gaussian noise to the transition function f at each time step. The details of the noise generation is elaborated in Appendix C.4. The noise compounds over time. Even though such a noise model leads to compounding errors over time (as in the case of a real robot), we acknowledge that this simple noise model may not perfectly match noise in the real world. Fig. 2(center) presents the coverage area at the end of episode (1000 time steps) as a function of the amount of noise introduced1. When the system suffers from sensor noise, the performance of the frontier-based exploration method drops rapidly. In contrast, our learning-based agent that wasn’t even trained with any noise continues to performs well. Even at relatively modest noise of 4% our learning based method already does better than the frontier-based agent. We additionally note that our agent can even be trained when the intrinsic reward estimation itself suffers from state estimation noise: for instance performance with 10% estimation noise (for intrinsic reward computation and map construction) is 98.9m2, only a minor degradation from 117.4m2 (10% estimation noise for map construction at test time only).
Geometry and Affordance Mismatch: Next, to emphasize the utility of learning for this task, we experiment with a scenario where we explicitly create a mismatch between geometry and affordance of the environment. We do this by rendering doors, but not using them for collision checking (i.e. the default House 3D environment). This setting helps us investigate if learning based techniques go beyond simple geometric reasoning in any way. Fig. 2(right) presents performance curves. We see that there is a large drop in performance for the frontier-based agent. This is because it is not able to distinguish doors from other obstacles, leading to path planning failures. However, there is a relatively minor drop in performance of our learning-based agent. This is because it can learn about doors (how to identify them in RGB images and the fact that they can be traversed) from human demonstrations and experience during reinforcement learning.

heading: 4.2.1 ABLATION STUDY
text: We also conducted ablations of our method to identify what parts of our proposed technique contribute to the performance. We do these ablations in the setting without any estimation noise, and use coverage as the metric.
Imitation Learning: We check if pre-training with imitation learning is useful for this task. We test this by comparing to the models that were trained only with RL using the coverage reward. The left two plots in Fig. 3 shows performance of agents with following combinations: RL: policy trained
1A noise of η means we sample perturbations from a truncated Gaussian distribution with zero-mean, η standard deviation, and a total width of 2η. This sampled perturbation is scaled by the step length (25cm for x, y and 9◦ for azimuth θ) and added to the state at each time step.
with PPO only, IL: policy trained with imitation learning, Map: policy uses constructed maps as input, RGB: policy uses RGB images as input. In all settings, pre-training with imitation learning helps improve performance, though training with RL improves coverage further. Also, policies trained using RL have a fairly large variance. Imitation learning also helps reduce the variance.
RGB Observations and Map: Fig. 3 (left) and Fig. 3 (center) respectively show that both RGB images and map inputs improve performance.
Intrinsic Reward: We also compare our intrinsic reward design with extrinsic reward design, as shown in Fig. 3 (right). The extrinsic reward is setup as follows: we randomly generated a number of locations evenly distributed across the traversable area of the houses, where the agent will get a positive reward if the agent is close to any of these locations. Once the agent gets a reward from a location, this location will be no longer taken into account for future reward calculation. We tried two settings where we place 1 or 4 reward-yielding objects per m2. We can see that our coverage map reward provides a better reward signal and in fact can be estimated intrinsically without needing to instrument the environment.
4.3 USING EXPLORATION FOR DOWNSTREAM TASK
Now that we have established how to explore well, we next ask if task-agnostic exploration is even useful for downstream tasks. We show this in context of goal-driven navigation. We execute the learned exploration policy πe (for 1500 time steps) to explore a new house, and collect experience (set of images I along with their pose p). Once this exploration has finished, the agent is given different navigation tasks. These navigation tasks put the agent at an arbitrary location in the environment, and give it a goal image that it needs to reach. More specifically, we reset the agent to 50 random poses (positions and orientations) in each testing house (20 testing houses in total) and get the RGB camera view. The agent then needs to use the experience of the environment acquired during exploration to efficiently navigate to the desired target location. The efficiency of navigation on these test queries measures the utility of exploration.
This evaluation requires a navigation policy πn, that uses the exploration experience and the goal image to output actions that can convey the agent to the desired target location. We opt for a simple policy πn. πn first localizes the target image using nearest neighbor matching to the set of collected images I (in ImageNet pre-trained ResNet-18 feature space). It then plans a path to the this estimated target location using a occupancy map computed from I. We do this experiment in the setting without any state estimation noise.
We independently measure the effectiveness of exploration data for a) localization of the given target images, and b) path planning efficiency in reaching desired locations. Localization performance: We measure the distance between the agent’s estimate of the goal image location, and the true goal image location. Fig. 4 (top) plots the success at localization as a function of the success threshold
(distance at which we consider a localization as correct). We report top-1 and top-5 success rates. We compare to the random exploration baseline and curiosity-driven baseline which serve to measure the hardness of the task. We see our exploration scheme performs well. Path planning efficiency: Next, we measure how efficiently desired goal locations can be reached. We measure performance using the SPL metric as described by Anderson et al. (2018) (described in the appendix, higher is better). We compare against a baseline that does not have any prior experience in this environment and derives all map information on the fly from goal driven behavior (going to the desired test location). Both agents take actions based on the shortest-path motion planning algorithm. Once again these serve as a measure of the hardness of the topology of the underlying environment. As shown in Fig. 4 (bottom), using exploration data from our policy improves efficiency of paths to reach target locations.

heading: 5 DISCUSSION
text: In this paper, we motivated the need for learning explorations policies for navigation in novel 3D environments. We showed how to design and train such policies, and how experience gathered from such policies enables better performance for downstream tasks. We think we have just scratched the surface of this problem, and hope this inspires future research towards semantic exploration using more expressive policy architectures, reward functions and training techniques.

heading: APPENDIX A SPL METRIC
text: As defined by Anderson et al. (2018), Success weighted by (normalized inverse) Path Length or SPL is computed as follows. Here, `i is the shortest-path distance between the starting and goal position, pi is the length of the path actually executed by the agent, and Si is a binary variable that indicates if the agent succeeded. SPL is computed over N trials as follows:
SPL = 1
N N∑ i=1 Si `i max(pi, `i) . (1)

heading: APPENDIX B EXAMPLES OF POLICY INPUTS AND MAPS
text: We show an example of the policy inputs (RGB image and maps in two different scales) at time t and t+ 10 in Fig. B.1. Green area means the known(seen) traversable area, blue area means known non-traversable area, and the white area means unknown area. Fig. B.2 shows how the map evolves and the agent moves as it explores a new house.

heading: APPENDIX C EXPERIMENTAL DETAILS
text: C.1 ENVIRONMENT DETAILS
In this paper, we simulated the agent in House3D environment Wu et al. (2018) and used 20 houses for training and 20 new houses for testing. Fig. C.1 shows the distribution of the total traversable area of these houses. Fig. C.2 shows some examples of the top-down views for training and testing houses (white is traversable, black is occupied).
C.2 MAP RECONSTRUCTION
Given a sequence of camera poses (extrinsics) ([R1, t1], [R2, t2], ..., [RN , tN ]), and the corresponding depth images (D1,D2, ...,DN ) at each time step as well as the camera intrinsics K ∈ R3×3, we can reconstruct the point cloud of the scene based on the principle of mulitple-view geometry (Hartley & Zisserman, 2003). We can convert the points from pixel space to the camera space and then to the world space. More specifically, the formulas to achieve this can be summarized as follows:
λijxij = K [Ri, ti]wij ∀j ∈ {1, 2, ..., S}, i ∈ {1, ..., N} (2)
W = N⋃ i=1 S⋃ j=1 wij (3)
where S is the total number of pixels in each depth image, xij ∈ R3 is the homogeneous coordinates for jth pixel on ith depth image Di, λij ∈ R is the depth value of the jth pixel on ith depth image Di, and wij ∈ R4 is the homogeneous coordinates for the corresponding point in the world coordinate system. We can get wij from xij based on Equation (2). And we can merge points via Equation (3).
C.3 TRAINING DETAILS
The occupancy map generated by the agent itself uses a resolution of 0.05m. Our policy uses a coarse map, and a detailed map. The coarse map captures information about a 40m×40m area around the agent, at a resolution of of 0.5m. The occupancy map is down-sampled from 800× 800 to 80× 80 to generate the coarse map that is fed into the policy network. The detailed map captures information about a 4m×4m area around the agent at a 0.05m resolution. RGB images are rendered at 224 × 224 resolution in House3D and re-sized into 80 × 80 before they are fed into the policy network.
The size of the last fully-connected layer in ResNet18 architecture is modified to 128. Outputs of the three ResNet18 networks are concatenated into a 384-dimensional vector. This is transformed into a 128-dimensional vector via a fully-connected layer. Next, it’s fed into a single-layer RNN(GRU) layer with 128 hidden layer size. The output of RNN layer is followed by two heads. One head (the policy head) has two fully-connected layers (128 − 32 − 6). The other head (the value head) has two fully-connected layers (128− 32− 1). We use ELU as the nonlinear activation function for the fully-connected layers after ResNet18.
Each episode consists of 500 steps in training. RNN time sequence length is 20. Coverage area C(Mt) is measured by the number of covered grids in the occupancy map. Coefficient α for the coverage reward Rcovint(t) is 0.0005, coefficient β for R coll int(t) is 0.006. PPO entropy loss coefficient is 0.01. Network is optimized via Adam optimizer with a learning rate of 0.00001.
C.4 NOISE MODEL
Details of noise generation for experiments with estimation noise in Section 4.2:
1. Without loss of generality, we initialize the agent at the origin, that is x̂0 = x0 = 0. 2. The agent takes an action at. We add truncated Gaussian noise to the action primitive(e.g., move
forward 0.25m) to get the estimated pose x̂t+1, i.e., x̂t+1 = x̂t + ãt where x̂t is the estimated pose in time step t and ãt is the action primitive at with added noise.
3. Iterate the second step until the maximum number of steps is reached.
Thus, in this noise model, the agent estimates its new pose based on the estimated pose from the last time step and the executed action. Thus, we dont use oracle odometry in the noise experiments. This noise model leads to compounding errors over time (as in the case of a real robot), though we acknowledge that this simple noise model may not perfectly match noise in the real world.
C.5 IMITATION LEARNING DETAILS
We use behavioral cloning technique (Argall et al., 2009; Michalski et al., 1998; Abbott, 2007) to imitate the human behaviors in exploring the environments. We got the human demonstration trajectories from (Das et al., 2018). We ignored the question-answering part of the data and only used the exploration trajectories. We cleaned up the data by removing the trajectories that have less than 100 time steps. The trajectories are then converted into short sequences of trajectory segments ((si, ai, si+1, ai+1, ..., si+T , ai+T ), where T is based on the RNN sequence length). The policy is pretrained with behavioral cloning by imitating actions (ai, ai+1, ..., ai+T ) from the human demonstrations given the states (si, si+1, ..., si+T ).

Review summaries:
This paper proposes a method for learning how to explore environments. The paper mentions that the “exploration task” that is defined in this paper can be used for improving the well-known navigation tasks. For solving this task, a reward function a network architecture that uses RGBD images + reconstructed map + imitation learning + PPO is designed.

This is a well explained and well executed paper on using classical SLAM-like 2D maps for helping a standard Deep RL navigation agent (convnet + LSTM) explore efficiently an environment and without the need for extrinsinc rewards. The agent relies on 3 convnets, one processing RGB images, one the image of a coarse map in egocentric referential, and one of the image of a fine-grained map in egocentric referential (using pre-trained ResNet-18 convnets). Features produced by the convnets are fed into a recurrent policy trained using PPO. Two rewards are used: the increase in the map's coverage and an obstacle avoidance penalty. The agent is further bootstrapped through imitation learning in a goal-driven task executed by a human controlling the agent. The authors analyze the behavior of the navigation algorithm by various ablations, a baseline consisting of Pathak's (2017) Intrinsic Curiosity Module-based navigation and, commendably, a classical SLAM baseline with path planning to empty, unexplored spaces.