Learning Exploration Policies for Navigation

Abstract: Numerous past works have tackled the problem of task-driven navigation. But, how to effectively explore a new environment to enable a variety of down-stream tasks has received much less attention. In this work, we study how agents can autonomously explore realistic and complex 3D environments without the context of task-rewards. We propose a learning-based approach and investigate different policy architectures, reward functions, and training paradigms. We find that use of policies with spatial memory that are bootstrapped with imitation learning and finally finetuned with coverage rewards derived purely from on-board sensors can be effective at exploring novel environments. We show that our learned exploration policies can explore better than classical approaches based on geometry alone and generic learning-based exploration techniques. Finally, we also show how such task-agnostic exploration can be used for down-stream tasks. Videos are available at https://sites.google.com/view/exploration-for-nav/.

Reviews and author responses:

After a full discussion on a proper SLAM baseline and explaining its difference with an exploration policy (such as frontier) and clarifying that authors had not been correct about arguing that R1 has missed the frontier method, now authors argue that the reviewer has flipped their arguments. That is not correct. Please read the full discussion thoroughly. The authors arguments about exploration for SLAM or for navigation are not justified correctly and are not backed up with a proper baseline. 
 
As for collision avoidance, I have provided detailed explanations as to why the experiment of Figure C.4(a) in the Appendix C4 can not be taken as a avoidance baseline. Please look at my previous comments about this.  The related discussion is explained fully in 3 paragraphs in my previous comments.
 
Obviously, the use of term “real world” instead of “realistic 3D environment” is not a correct practice in a scientific writing. Also this paper does not provide any evidence as to how well it can work on real scenes, or on a real robot or to what extend and how it can be transferred to real world. Using improper rewordings and making such arguments without an empirical or theoretical backup in a scientific paper is not correct.
 
While I can provide more explanations to clarify more and more about the arguments authors have made, I leave the rest of this discussion to AC. My final decision for this paper, as I also mentioned in my previous comments, is “2: strong reject”.  

Before we respond to R1, we point out that R3 supports the paper over and above R1's comments. 

**SPTM**
We quote N. Savinov from his follow-up paper "Episodic Curiosity Through Reachibility" that is available on arXiv:  "SPTM (Savinov et al., 2018) does compare to the episodic memory buffer but solves a different task — given an already provided exploration video, navigate to a goal — which is complementary to the task in our work." Furthermore, SPTM paper does not describe their automated exploration policy in enough detail, and itself acknowledges: "Effective exploration is a challenging task in itself, and a comprehensive study of this problem is outside the scope of the present paper. However, as a first step, we experiment with providing our method with walkthrough sequences generated fully autonomously – by our baseline agents trained with reinforcement learning. This is only possible in simple mazes, where these agents were able to reach all goals. We used the best-performing baseline for each maze and repeated exploration multiple times, until all goals were located."

**ZSVI** 
First, we emphasize that we have consulted with one of the authors of ZSVI and they agree: "ZSVI does not attempt to solve long-term navigation problem" (more details on this below).

a) 20-30 steps vs 59 steps for ZSVI. R1 is not only wrong but is also making a petty point. We picked 20-30 from the text of ZSVI: see Sect. 3.2 > 1. Goal Finding: "To test the extrapolative generalization, we keep the Turtlebot approximately 20-30 steps away from the target location in a way that current and goal observations have no overlap as shown in Figure 4." 20-30 or 59, our argument still holds.
b) Intermediate waypoints necessary or not. Our previous response was in consultation with authors of ZSVI. We reiterate: “Thus, ZSVI does not attempt to solve long-term navigation problem by itself and requires an expert to break long-term navigation into several short-term navigation problems.” Once again, finding goal tasks in ZSVI are limited to when the goal is 20-30 (or 59 whichever number you want to use here) steps away. Read relevant portions of text from their paper.

** Frontier-based Method **
R1 has changed their arguments on this over their different responses:
a) In the first review, R1 missed frontier-based method all together. 
b) When we pointed that we already have much stronger baselines (frontier-based method) than R1s suggestion of using a "greedy" policy, R1 claimed frontier-based method is not state-of-the-art.
c) On reiterating that our implementation is indeed very strong, R1 has flipped their argument again, stating that we did not describe it accurately in the original version. This is again incorrect, Section 4.1 > Baselines > 1. Frontier-based Exploration was and is an accurate description of our implementation. R1 had missed the frontier-based method altogether in their original review, and we suggest R1 to read the relevant part of the paper again.

** Collision Avoidance **
Our action space permits the agent to be stationary, or move around in a small circle. Such a behavior maximizes collision avoidance reward. We explicitly experimented with it in our setup and reported what we found in Appendix C4. Given our setup, what we found makes perfect sense. We are not sure what more R1 wants here, is there a different experiment you want us to run?

**Real World Scenes** 
House3D environments are realistic layouts of houses (made by people on the Internet). Infact, their paper is itself called: “Building Generalizable Agents With a Realistic And Rich 3D Environment”. Computer vision algorithms trained on this dataset have been shown to transfer to the real world. See the original SUNCG paper.

<<Comments about Pathak et al. 2018 Zero-shot visual imitation>>  
 
Regarding the discussion about the method presented in Pathak et al. 2018 Zero-shot visual imitation:
 
-According to the manuscript of Pathak et al. 2018 and the reported experiments in the published paper (https://arxiv.org/pdf/1804.08606.pdf , https://openreview.net/pdf?id=BkisuzWRW
) this statement is not correct:
 
“When GSP is applied to the navigation task, first, it requires the **goal** positions to be within **20-30 steps**”
 To investigate this, please look at Figure 4 in the main manuscript. Figure 4 of Pathak et al. 2018 shows that the agent takes 42 steps to see the goal in its view and then takes 17 other steps until it reaches the goal in their settings. This is a total of 59 steps which is more that 20-30 steps.
 
-Also, as explained in the caption of Figure 4 in Pathak et al. 2018, the exploratory behavior has naturally emerged during learning. I had mentioned in my previous review (and I repeat it here again) one would want to see how much gain in the task of navigation would be obtained if an “exploration only policy” is learned in a separate step. As can be seen from prior work, some exploration behavior naturally emerges during training and it is not clear if explicitly learning exploration is needed.
 
-There are two testing scenarios in Pathak et al. 2018, one that can use a sequence of landmark images and one that can take one single image as a goal (Please look at Figure 4 and Table 1 in Pathak et al. 2018 for evaluations on a single image goal scenario). Both scenarios are valid can be equally used as a use case of Pathak et al. 2018. Therefore, this statement in the post-rebuttal is incorrect: “GSP requires **an expert** to provide a sequence of **landmark images**”
 
 
<<Comments about SPTM>> 
This statement by authors in response of SPTM is not correct:
Authors incorrectly say: “SPTM requires the expert demonstration trajectories even at test time.” 

SPTM paper mentioned their experimental setup at test time at page 7 of their manuscript (https://arxiv.org/pdf/1803.00653.pdf):

“When given a new maze, the agent is provided with an exploration sequence of the environment, with a duration of approximately 5 minutes of in-simulation time (equivalent to 10,500 simulation steps). In our experiments, we used sequences generated by a human subject aimlessly exploring the mazes. The same exploration sequences were provided to all algorithms the proposed method and the baselines.”

The exploration sequence in SPTM is NOT *expert demonstration trajectories* at test time. it is *aimlessly exploration* of the maze by an agent and used as a set of observations in producing waypoint observations. Then the locomotion network is used for navigating towards the waypoints.

Also, please note that SPTM does not have systemic dependency to human demonstration and experiments with non-human exploration is also provided in the SPTM paper in the supplemental material (Table S2). Thus it cannot be said that SPTM is *infeasible* without demonstrations. 

Please look at this paragraph from SPTM paper at Page 10:
“Additional experiments are reported in the supplement: performance in the validation environments, robustness to hyperparameter settings, an additional ablation study evaluating the performance of the R and L networks compared to simple alternatives, experiments in environments with homogeneous textures, and experiments with automated (non-human) exploration.”
 
 
Based on the points mentioned above, authors do not discuss about prior work correctly. The arguments they have made are not convincing. Authors seem to not be knowledgeable about prior work. Authors have not provided the requested baselines. After several rounds of discussion, the paper still lacks proper experimental evaluations with the baselines and state-of-the-art methods. Based on this, I lower my score to “2” and vote for “strong rejection” of this paper.

Authors do not discuss about prior work correctly. The arguments are not convincing, paper lacks proper comparison with the state-of-the-art baselines. I lower my score to “2” and vote for “strong rejection” of this paper.


<<Comparing with a learned collision avoidance policy>>

The arguments made about the reason for not comparing with a prior collision avoidance baseline is not convincing:
 
-This statement made by authors is incorrect: “Experiments that we included (going forward and randomly turning at collision) uses *ground truth collision checking*, and thus already has an advantage over a policy that uses a learned model for collision checking”
While one can use ground truth collision checking, that does not suffice for a good “collision-avoidance policy”. Ground truth collision checking in the form that authors have explained, only provides a noise-free observation representation, and it does not provide any intelligent policy for avoiding collisions. The policy that authors have used on top of the noise-free observation representation (obtained from ground truth sensing) is “going forward and randomly turning at collision” which is a *heuristic policy*. A learned policy may not always move forward as a means to not be trapped in dead-ends. Also, random turns at the time of collision is not optimal; at the time of predicting a near obstacle (and thus a possible future collision) a learned policy can choose actions based on previous observations so that it can lead the agent to places with less chance of collisions in the near future. Given that such behavior can provide exploration as a side product of collision avoidance, I asked for a comparison with a state-of-the-art learning-based collision avoidance policy.  However, the authors did not provide such comparison. 
 
 
-The arguments made by authors about the “Sadeghi and Levine 2017” are not correct. Please look at the video provided here: https://www.youtube.com/watch?v=nXBWmzFrj5s
At the minute 3:07-3:11, the agent moves into a room and then moves out of it without *keep turning in a circle*. Also, minutes 0:43-2:33 of the same video shows another example on how the agent explores a building with several rooms and how it moves out of the rooms from the doors and without *keep turning in a circle*.
 
 
-The experiment that the authors point to for a version of their method that only gets collision avoidance reward in Figure C.4(a) in the Appendix C4 cannot be used in lieu of a collision avoidance policy with the current. The reason is that, a version of the proposed method that only gets collision avoidance reward could only be used as a baseline of a learned collision avoidance policy if its performance for the task of “collision-avoidance” had been compared with a state-of-the-art collision avoidance policy and similar results had been obtained. In other words, it is not clear if the version of the proposed method that only gets collision avoidance reward can compete with any of the state-of-the-art collision avoidance policies presented in prior work.
With the current set of experiments conducted in this paper, the experiment of Figure C.4(a) in the Appendix C4 can be taken as an ablation study on the components of the proposed method and can not be referred as a state-of-the-art collision avoidance policy.


I still do not see the paper to be offering significant novelty or interesting results. The proposed method does not have major technical novelty and the experiments do not prove that the proposed method is a promising direction for the problems of interest such as “navigation” or “map-reconstruction” or “general vision-based policy learning”. I have provided response to the discussion made by the authors bellow. Given all these discussions, I change my initial rate for the paper; I lower my score to “2” and vote for “strong rejection” of this paper.
 
I would like to also mention that the authors have not properly discussed about the points raised previously and the repeated statements about “incorrect understanding” or “misunderstanding” of the reviewer are not valid. 
 
Below I point out to several of the unconvincing arguments made by the authors:
 
<<Frontier-based method>>
After several rounds of discussion, and after stating several times that the reviewer has misunderstanding, the authors have provided a list of more recent prior works (than the 1997 paper of frontier-based method) as the actual versions of frontier-based method that they have used, and have revealed more details about their in-house version of “frontier-based baseline” implementation. Why were these details, explanations and prior works missed from the main manuscript in the first place? How can this discussion be used as an evidence that the more recent version of the “frontier-based method” was used as the baseline? 
Not citing a prior work while being aware of its existence and at the same time using it is not acceptable.
Clearly, it is not right to say that the reviewer has misunderstood about something which was absent in the paper.
 
The claims about the proposed method written in the paper and rebuttal are not precise.
For example, here is a statement in the first line of Section 2 at Page 2 of the manuscript:
“Our work on learning exploration policies for navigation in real world scenes is related to active
SLAM in classical robotics”
The claim about a “learning exploration policies for navigation in real world scene” is obviously not correct. Because the paper does not provide any evidence as how good the proposed method can work in the “real world” or on “real scenes” . All experiments are conducted in simulation environment: no real image of scene, no real environment, and no real robot has been used in the entire paper. Prior state-of-the-art exploration works (that I also mentioned in my previous post-rebuttal comments) such as .” Xu, K., Zheng, L., Yan, Z., Yan, G., Zhang, E., Niessner, M., ... & Huang, H. Autonomous reconstruction of unknown indoor scenes guided by time-varying tensor fields. ACM Transactions on Graphics (TOG), 2017.” work on real scenes with a real robot and solve a real problem related to active SLAM.
 


The authors have addressed my concerns. I brought the score back up to 7. I think the paper should be accepted.

This paper proposes learning exploration policies for navigation. The problem is motivated well. The learning is conducted using reinforcement learning, bootstrapped by imitation learning. Notably, RL is done using sensor-derived intrinsic rewards, rather than extrinsic rewards provided by the environment. The results are good.

I like this paper a lot. It addresses an important problem. It is written well. The approach is not surprising but is reasonable and is a good addition to the literature.

One reservation is that the method relies on an oracle for state estimation. In some experiments, synthetic noise is added, but this is not a realistic noise model and the underlying data still comes from an oracle that would not be available in real-world deployment. I recommend that the authors do one of the following: (a) use a real (monocular, stereo, or visual-inertial) odometry system for state estimation, or (b) acknowledge clearly that the presented method relies on unrealistic oracle odometry.

Even with this reservation, I support accepting the paper.

Minor: In Section 3.4, "existing a room" -> "exiting a room"

Thanks for your suggestions and appreciation for our paper. In our initial response, we were trying to keep the original paper intact and add any changes in the appendix so as to make it easy for reviewers to see what we have changed. As we mentioned in the first response publicly, we promise to make the changes you requested in the final version. We sincerely hope the AC and reviewer do not penalize us for a well-intentioned but not aligned update since we did not realize the update to main paper is a necessity.

We have an updated paper on the website linked in the abstract with these changes. We are listing the changes we made in this update below. These will be included in the final version. We will additionally also include references to the related works that came up in discussion with R1 to the final paper as well. 

(1) We will add the following text to the “With Estimation Noise” paragraph on page 7.
***
Even though such a noise model leads to compounding errors over time (as in the case of a real robot), we acknowledge that this simple noise model may not perfectly match noise in the real world.
***


(2) We will make italic the following existing text in the “Without Estimation Noise:” paragraph on page 7. 
***
Note that this setting is not very realistic as there is always observation error in an agent’s estimate of its location.
***


(3) We will add the following text to the Appendix C.7:
***
Details of noise generation for experiments with estimation noise in Section 4.2: 
1. Without loss of generality, we initialize the agent at the origin, that is $x(0) = \mathbf{0}$.
2. The agent takes an action a(t).  We add truncated Gaussian noise to the action primitive(e.g., move forward 0.25m) to get the estimated pose x(t+1),  i.e., x(t+1)=x(t)+ (a(t) with noise)  where x(t) is the estimated pose in time step t.
3.  Iterate the second step until the maximum number of steps is reached.
Thus, in this noise model, the agent estimates its new pose based on the estimated pose from the last time step and the executed action. Thus, we don’t use oracle odometry in the noise experiments. This noise model leads to compounding errors over time (as in the case of a real robot), though we acknowledge that this simple noise model may not perfectly match noise in the real world.
***

(4) We have fixed the typo (“exiting a room”). 


We very much appreciate your understanding and kindly request you to keep the original rating. We will be happy to rephrase these changes and add further clarifications if you think they will be necessary. Please let us know. 

Thanks.


I reviewed the spirited discussion between R1 and the authors. I continue to think that the paper provides a fine and informative addition to the literature. It's above the bar for ICLR and I vote for acceptance.

The discussion did bring up many interesting and relevant references to prior (pre-deep) work. These were brought up both by R1 and by the authors. I strongly encourage the authors to incorporate these into the paper. I think this will be useful to the community. These references should be in the paper, not just on the discussion board.

I am lowering my rating a bit from 7 to 6 because the authors did not address my request from the original review in the revision, even though they could. I see no reason not to: that's what the ICLR revision period is for.



**Comparison with other learning methods**:
We would like to remind R1 that we added an experiment where the RL agent only gets collision avoidance reward in Figure C.4(a) in the Appendix C4. It shows that the agent does not learn anything meaningful (which is not surprising as even a stay-in-place policy will get a perfect reward). Sadeghi and Levine 2017 and other works on collision avoidance, either explicitly additionally use rewards for moving forward, or appropriately engineer the action space by forcing the agent to move forward in each time step. We did not pick the action space ourselves but used whatever came with House3D, making direct comparisons to such approaches infeasible. Also, the policy in Sadeghi and Levine 2017 shows some exploration behaviors in narrow hallways as the only way to keep agent moving and not colliding the walls, in this specific case, is to move forward. However, it would fail to show exploration behavior in large open space such as living rooms (as in our experiments), because the agent can simply keep turning in a circle to stay away from any wall. Again, we argue that the major purpose of Sadeghi and Levine 2017 is to learn a collision-free policy that keeps the agent moving without a specific intent to keep agent exploring the environments. Experiments that we included (going forward and randomly turning at collision) uses *ground truth collision checking*, and thus already has an advantage over a policy that uses a learned model for collision checking.

As for the comparison to Pathak et al. GSP method in ‘Zero-shot visual imitation’. We would like to emphasize to AC and R1 that we have personally communicated with one of the authors of Pathak et al. before formulating this reply: GSP tackles a completely different problem that of acquiring skills using self-supervision. While, GSP can do local navigation, GSP is NOT designed for long-horizon navigation tasks. When GSP is applied to the navigation task, first, it requires the **goal** positions to be within **20-30 steps** from the agent’s current position and the agent will see the target observation within the first 5-10 steps. But in our experiments, our agent is exploring whole houses in House3D, which easily takes thousands of steps and the agent rarely ever sees the target observation within the first 5-10 (or for that matter even 100s of) steps. Second, GSP requires **an expert** to provide a sequence of **landmark images** to guide the agent to move to a far target location while our agent explores the house environment efficiently on its own without the need of experts. Thus, ZSVI does not attempt to solve long-term navigation problem by itself and requires an expert to break long-term navigation into several short-term navigation problems. If we attempt to use ZSVI to solve long-term navigation (without “expert waypoints" as in our experiments) it would fail.

Zhu et al. 2017 proposed a target-driven navigation policy that can find the object given an image. However, the training and testing environment are the same in Zhu et al.’s case. Their goal is to learn a policy to find the object in the same room which requires millions of interaction in the training/testing environment. Our work focuses on learning an exploration policy that generalizes to new environments in a zero-shot manner. During our experiments, we confirmed the same: the policy learned in Zhu et al. fails to generalize to new environments without re-training for millions of iterations as mentioned in their paper.


References:
Dornhege, Christian, and Alexander Kleiner. "A frontier-void-based approach for autonomous exploration in 3d." Advanced Robotics 27.6 (2013): 459-468.

Wang, Yiheng, Alei Liang, and Haibing Guan. "Frontier-based multi-robot map exploration using particle swarm optimization." Swarm Intelligence (SIS), 2011 IEEE Symposium on. IEEE, 2011.

Fraundorfer, Friedrich, et al. "Vision-based autonomous mapping and exploration using a quadrotor MAV." Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on. IEEE, 2012.

Mannucci, Anna, Simone Nardi, and Lucia Pallottino. "Autonomous 3D exploration of large areas: a cooperative frontier-based approach." International Conference on Modelling and Simulation for Autonomous Systems. Springer, Cham, 2017.

Campos, Francisco M., et al. "A complete frontier-based exploration method for Pose-SLAM." Autonomous Robot Systems and Competitions (ICARSC), 2017 IEEE International Conference on. IEEE, 2017.

Mahdoui, Nesrine, Vincent Frémont, and Enrico Natalizio. "Cooperative Frontier-Based Exploration Strategy for Multi-Robot System." 2018 13th Annual Conference on System of Systems Engineering (SoSE). IEEE, 2018.



**Frontier-based exploration**:
We want to clarify that the frontier-based exploration we have implemented is an improved version of the original algorithm. The original algorithm commands the agent to go along with the frontier grids in 2D. However, in our version of the frontier exploration, we sampled a target point which, in most cases, are far away from the agent’s current locations. Then the agent uses a shortest path planning algorithm to go to that target position. Since we are using a vision-based RGBD sensor, we don’t have to go through the frontier grids one by one as the robot can see many frontier grids in one time, which greatly improves the efficiency. This is a fairly efficient algorithm if pose estimates are accurate. In fact, in our implementation, we need to sample less than 10 target points to cover majority of the area in most houses. Our implemented frontier-based exploration method, in some sense, is more similar to the frontier-based exploration in 3D proposed in Dornhege et al. 2013 (we will open source the code). We cited Yamauchi, 1997 because, to our knowledge, this paper is the earliest paper that proposed the frontier-based exploration method. Also, we would like to clarify that frontier-based exploration is not an outdated technology. It’s still being used in the robotics community. Just to give a few examples, Wang et al. 2011, Fraundorfer et al. 2012, Mannucci et al. 2017, Campos et al. 2017, Mahdoui et al. 2018 all use frontier-based exploration method.

**Related work**:
Thanks for the suggestion on the related work. We are happy to add these relevant works in the final version of the paper as R1 requires. However, our work is different from these works. The main focus on Xu et al. 2017 is on generating smooth movement path for high-quality camera scan. Bai et al. 2016 proposed an information-theoretic exploration method using Gaussian process regression. This is computationally inefficient when the kernel matrix becomes large. Thus, Bai et al. 2016 only show experiments on simplistic map environments. GPs are computationally expensive when maps are complicated, which is the case in our experiments. Kollar et al. 2008, assume access to the ground-truth map and learn an optimized trajectory that maximizes the accuracy of the SLAM-derived map. In contrast, our learning policy directly tells the action that the agent should take next and estimates the map on the fly. 



We thank R1 for reading through our paper more carefully. Unfortunately, we still believe R1’s understanding of the paper is incorrect. This is clearly highlighted by the following statement made by R1: 

“While the proposed method also uses human demonstration, authors argue that SPTM requires a human to demonstrate the environment, which is impractical in real-world scenarios. This is a contradicting statement.”

SPTM requires the expert demonstration trajectories even at test time. Our work only uses human demonstration for imitation learning during training time ONLY. Again we emphasize, unlike SPTM, we *do not* require human demonstrations at test time. 

We individually address *ALL* the other points raised by R1 below.

Improving performance of down-stream navigation task is listed as one of the main contributions of the paper (second paragraph of page2). However, proper empirical comparison with a state-of-the-art navigation method is not conducted. While it is mentioned in the rebuttal that they could combine their approach with a state of the art navigation method (such as SPTM) they refused to conduct such comparison. As also mentioned in my original review, an empirical comparison between “proposed method + a state-of-the-art navigation method (e.g. SPTM)” versus “a state-of-the-art navigation method (e.g. SPTM)” is required to understand if the proposed exploration policy is actually needed to improve navigation or not or how much the exploration step can help improving navigation. 

While the proposed method also uses human demonstration, authors argue that SPTM requires a human to demonstrate the environment, which is impractical in real-world scenarios. This is a contradicting statement. 

Other prior works of navigation such as Pathak et al’s “Zero-shot visual imitation” or “Zhue et al 2017” could also be used as a baseline for navigation as both propose a goal-driven navigation method where the image of goal is taken as input. For comparing with Pathak et al’s “Zero-shot visual imitation” no modification would be required as it works on a similar navigation task setup at conducted in section 4.3 (taking image of the goal as input). The code of Pathak et al’s “Zero-shot visual imitation” is available in github.

Based on the above points, this paper lacks proper comparison with state-of-the-art prior works and many relevant prior works are ignored. Technical novelty is incremental, known learning techniques and architectures are used. In addition, the paper is also not offering a novel application or a novel problem and experiments are not conveying interesting empirical results (prior works are ignored and not cited). Also the claims of the paper for its contribution are not backed up with analytical or experimental evaluations. Based on these, my vote is for rejection of the paper.


As upon to the authors request, I read the paper one more time. Before responding to the argues made by the authors, I would like to invite them to read the reviews thoroughly with more attention and consider relevant state-of-the-art research work before arguing that the reviewer has *misunderstood* or *missed* the paper. The rebuttal repeatedly has pointed to R1 for not understanding the paper or missing things while no convincing answers is provided for the major points that I had mentioned in my (R1) review. The revised version is a better version than the submission, thanks to authors for revision. However, I still do not see this paper to provide interesting technical novelty or compelling results for the major audience of the ICLR conference and I keep my initial vote for rejection of this paper.
 
Having this in mind that one of major goals of peer-reviewing is to provide constructive feedback I respond to the argues brought by authors in their rebuttal hoping that they don’t argue about these facts with incomplete rewordings or pretending misinterpretation:

Here are the responses to some of the points in the rebuttal:

I had not missed the comparison with “fronitier-based” method in the manuscript.
I'd want to make it clear that “frontier-based exploration” method of Yamauchi , 1997 is *not* a classic SLAM method. “Frontier-based exploration” is a very old exploration heuristic proposed in 1997. Therefore, I do not consider it as a “compelling comparison point” or even a strong baseline. 

I'd want to highlight that the following sentence from the second paragraph of the introduction in the main manuscript is incorrect and is particularly ignoring many years of active research on good exploration policies for map construction. I have pointed to a few of such prior works bellow.  Yamauchi, 1997 cannot be considered as a state-of-the-art method and is not a proper point of comparison for exploration policies. 
Incorrect sentence in the manuscript: “How does one build a map? How should we explore the environment to build this map? Current approaches either use a human operator to control the robot for building the map (e.g. Thrun et al. (1999)), or use heuristics such as frontier-based exploration (Yamauchi, 1997)”


I had requested comparison with a SLAM-based method that constructs the map and then does navigation on that map (Please read at my original review thoroughly). Since, it seems that the authors are not aware of state-of-the-art works in SLAM and map reconstruction as well as state-of-the-art autonomous exploration policies I point them to a few recent works (out of numerous works conduced in this area of research in the past few years):

[a] Xu, K., Zheng, L., Yan, Z., Yan, G., Zhang, E., Niessner, M., ... & Huang, H. Autonomous reconstruction of unknown indoor scenes guided by time-varying tensor fields. ACM Transactions on Graphics (TOG), 2017.

[b] Shi Bai, Jinkun Wang, Fanfei Chen, and Brendan Englot. Information-theoretic exploration with Bayesian optimization. In Intelligent Robots and Systems (IROS), 2016 IEEE/RSJ International Conference on . IEEE, 2016.

[c] Thomas Kollar and Nicholas Roy. Trajectory optimization using reinforcement learning for map exploration. Int. J. Robotics Research, 2008.


It is strongly recommended that these relevant citations be added to any final version of this manuscript.

Specially, in [a] an autonomous exploration policy is proposed and is shown to be robust to noise and can work on a real robot and in a real environment rather than just simulation. 


The collision avoidance policy baseline is not conducted properly. It is not acceptable to coin a heuristic method of “moving straight” and then “random turn” agent and call it a “sophisticated version” of collision avoidance policy. Such statement is not correct. A state-of-the-art learning-based collision avoidance method should be used as a point of comparison. For example, look at Sadeghi and Levine, 2017 for a learning based collision avoidance policy that does not fall into degenerate solution of staying in place and also have a rich action space. 


We thank R1 for their comments. R1’s primary concerns are about novelty and missing empirical comparison. These perhaps stem from some misunderstandings about our paper as some requested comparisons are either irrelevant or stronger comparisons are already presented in the paper.  Therefore, we urge the reviewer to take a second look at the paper in light of the rebuttal.

1. Novelty: In this paper, we learn policies for exploring novel 3D environments (Section 3 through Section 4.2), and show that exploration data, gathered by executing our learned exploration policies, improves performance at downstream navigation tasks (Section 4.3). To the best of our knowledge, this is the first work that studies learned exploration policies for navigation, systematically compares them to classical and learning-based baselines, and shows the effectiveness of exploration data for downstream tasks. In doing so, we adopt existing learning techniques (imitation learning + reinforcement learning), and map building techniques. Our novelties are orthogonal to these aspects:
      (a) Problem formulation: Framing exploration as a learning problem, and showing the utility of exploration data for downstream tasks.
      (b) Map based policy architectures and reward functions. Classical SLAM based approaches indeed produce maps but: (a) it still needs a policy for exploration during the map-building phase; (b) does not solve navigation rather uses geometric analysis for path planning. Our approach focuses on (a) and unlike heuristic approaches used in SLAM, we use a learning-based approach.
      (c) We also show maps can also be used for learning effective policies, and for computing reward signals.
      (d) Use of IL + RL to optimize our policy, as opposed to pure RL that is typically used.

2. Comparison with other exploration approaches: 
a) Simple Greedy Baseline: We experimented with the suggested one-step greedy policy. Here we virtually simulate all possible actions that the agent can take, and compute the gain in coverage. We then execute the action that results in the maximum gain in coverage. At 1000 steps such a policy only covers 40m^2, as opposed to our policies that cover up to 125 m^2. This is not surprising as the policy gets stuck inside local regions of full coverage. No action leads to any increase in coverage and the agents move back and forth. The full performance plot is provided in Fig C3(a) in the updated PDF. 

Note, in the paper, we have provided a more compelling comparison point to classical exploration approaches: frontier-based method. Reviewer seems to have missed this comparison as R1 still asks for comparisons to classical approaches.

b) Collision Avoiding Policy: A policy that purely avoids collisions has a degenerate solution of the agent staying in-place, resulting in negligible coverage (Fig C4(a)). We also tried a more sophisticated version, where the agent moves straight unless a collision happens (Fig C3(a)), at which point it randomly rotates (by angle between 0 and 2pi), and continues to move straight. To help the policy further, we used ground truth collision-checking. This policy covers 75m^2, still much lower than our performance (125m^2).

3. Comparison with other learning-based navigation works: First, we do not study a specific navigation task, but instead our contribution is a task-independent exploration policy. We do however show that exploration helps in downstream navigation tasks (Section 4.3). We use well-established Classical Path Planning, the simplest navigation algorithm for doing these experiments. This was a conscious choice so as to not-conflate quality of learned navigation policy with the quality of our learned exploration policy. Our contribution is orthogonal to navigation task itself and therefore our approach can be used in conjunction with any navigation approach. For example, the exploration data by running our policy can be used ‘as is’ with Savinov et al’s state-of-the-art SPTM approach [A]. SPTM otherwise requires a human to demonstrate the environment, which is impractical in real-world scenarios. 

R1 suggests we should compare to Pathak et al’s “Zero-shot visual imitation” (ZSVI) as it uses “exploration strategies” and “imitation learning” for navigation. While they indeed use both terms (exploration and imitation), the context and usage is completely different.
      (a) *Exploration for Imitation (ZSVI) vs. Imitation for Exploration (Ours)*
           In ZSVI, exploration is used in training to collect trajectories and imitation is used in testing to follow a path. On the other hand, ours is completely the opposite. We use imitation in training to learn how to explore at test time. Again we emphasize: ZSVI does not run any explicit exploration policy during testing.
      (b) This leads to completely different behavior of two algorithms. The time/distance range in ZSVI is much smaller as compared to ours. Either the goal is in the same room or they need a lot of waypoint images to solve the navigation task.

In order to show a comparison to “RL with a good exploration … without explicit exploration“, we have implemented navigation on top of Curiosity Driven Exploration using Self-Supervision. As shown in Appendix C.6 (will be added to Sec 4.3),  the comparison is in our favor. 

4. More Experimental Details: We have added additional details in Appendix C. We have included:
a) Stats and floor-plans of houses used for training and testing (Appendix C1).
b) Coverage plots for when we run the agent for 2000 steps (Appendix C4, Fig C3). Conclusions are the same as for the original 1000 steps plots as presented in the paper.
c) Agent details. Step size is 0.25m forward motion, 9 degree rotations (already provided in the paper). Real world performance depends on how fast a robot is. A turtlebot-2 can move at a peak speed of 0.65 m/s, if that’s what you were looking for.

5. More Technical Details: 
a) We have added details about map construction in Appendix C2. Yes, we can use known-loop closure techniques in SLAM, though there may still be error and we wanted to show that learning is robust to it (Fig 2 (center), video on website).
b) Imitation learning details are in Appendix C5.
c) 3D Information: Yes, you are right depth images only give 2.5D information, however, we integrate information from different views, to obtain a more complete sense of the environment than given by a single depth image. 3D information can also be extracted from RGB images, see [B] and numerous others for example.

We will incorporate your suggestions on presentation in the final version.

[A] Semi-parametric Topological Memory for Navigation Nikolay Savinov, Alexey Dosovitskiy, Vladlen Koltun. ICLR 2018.
[B] Factoring Shape, Pose, and Layout from the 2D Image of a 3D Scene Shubham Tulsiani, Saurabh Gupta, David Fouhey, Alexei A. Efros, Jitendra Malik. CVPR 2018.


Thank you for your comments and suggestions. We acknowledge that most of our evaluation is in the perfect odometry setting which is unrealistic. We experimented with a reasonable noise model that compounds over time within the episode, but we admit it may not be very realistic. We will prominently note both these points in the final version of the paper upon acceptance.



Thanks for your comments and suggestions. We address your specific concerns below:

1. Explicit mapping is hand-engineering. We acknowledge (and will explicitly state in the paper) that using occupancy map as the policy input is based on domain/task knowledge. Using the occupancy map gives the agent a better representation of long-horizon memory and show great improvement compared to the policy without the map as input. We do agree ego-motion estimation in real-world might be noisy. To handle that we performed experiments with noise and show that our model seems robust (See video on the website, Fig 4b in the paper).  

With regard to end-to-end approaches, approaches like Zhang et al. (2017) uses a differentiable map structure to mimic the SLAM techniques. These works are orthogonal to our effort on exploration. Indeed, our exploration policy can benefit from their learned maps instead of only using reconstructed occupancy map. We also believe our current approach provides a strong baseline for future end-to-end versions.

2. Explicit environment rewards for exploration: We agree that the use of reward yielding objects throughout the environment will lead to a very similar outcome as our approach. The key distinction is that our approach instruments the agent (with a depth sensor) as opposed to instrumenting the environment. This makes our proposed formulation more amenable to being trained and deployed in the real world: all we need is an RGB-D sensor. This is a big advantage over spreading reward yielding objects that disappear as the agents arrive at those locations, which is almost impractical in the real world. With this key distinction being said, we did do several experiments where our policy is trained with external rewards. The performance is shown in Fig C4(c) in Appendix C4. The results show that our coverage map reward is much more effective than external rewards generated by reward-yielding objects. Our method covers 125m^2 on average while even 4 reward yielding objects per square meter is 91m^2.

3. Role of collision avoidance penalty: We added the performance of the agent trained with our policy but with only coverage reward (no collision penalty) in Fig C4(b) in Appendix C4. We observe that adding collision penalty indeed helps improve performance slightly (125m^2 with penalty as opposed to 120m^2 without penalty). Thus, our policy explores well even without explicit collision avoidance penalty.

We will add more references to the related work and improve the writing as you suggested in the final version of the paper.


We thank the reviewers for their comments and suggestions. We are glad that the reviewers found:
        (a) our paper to tackle an important and clearly motivated problem (R1, R3)
        (b) our approach to be a great idea (R2), a good addition to the literature (R3) and not-complicated (R1).
        (c) our paper to be “well-executed”, with “various ablations”, and “comparisons to … commendably a classical SLAM baseline” (R2)
        (d) our paper to be well-written (R1, R3), and well-explained (R2).
We have answered *ALL* questions that the reviewers posed by providing additional experimental comparisons, pointing to relevant existing experiments and providing clarifications. Hopefully, this clarifies some of the misunderstandings that R1 has about our paper. Additional experiments have been added to Appendix C of the updated PDF. We will incorporate these experiments and other suggestions in camera-ready upon acceptance.



This paper proposes a method for learning how to explore environments. The paper mentions that the “exploration task” that is defined in this paper can be used for improving the well-known navigation tasks. For solving this task, a reward function a network architecture that uses RGBD images + reconstructed map + imitation learning + PPO is designed.

<<Pros>>

-The paper is well-written (except for a few typos).
-The overall approach is simple and does not have much complications. 
-The underling idea and motivation is clearly narrated in the intro and abstract and the paper has a easy-to-understand flow.  

<<Cons>>

**The technical novelty is not significant**

-This paper does not provide significant technical novelty. It is a combination of known prior methods: imitation learning + ppo (prior RL work). The presented exploration task is not properly justified as to how it could be useful for the navigation task. The reconstruction of maps for solving the navigation problem is a well-explored problem in prior SLAM and 3D reconstruction methods. Overall the novelty of the approach and the proposed problem is incremental. 

**The paper has major short comings in the experimental section. The presented experiments do not support the main claim of the paper which is improving the performance in the well-known navigation task. Major baselines are missing. Also, the provided results are not convincing in doing the right comparison with the baselines. **

-Experimental details are missing. The major experimental evaluations (Fig. 2 and Fig. 3) are based on the m^2 coverage after k steps and the plots are cut at 1000 steps. What are the statistical properties of the 3D houses used for training and testing? E.g what is their area in m^2? How big is each step in meters?  Why are the graphs cut at 1000 steps? How would different methods converge after more than 1000 steps, e.g. 2000 steps? I would like to see how would the different methods converge after larger number of steps? How long would each step take in terms of time? How could these numbers convey the significance of the proposed method in a real would problem settings? 

-The experiments do not convey if learning has significantly resulted in improved exploration. Consider a simple baseline that follows a similar approach as explained in the paper for constructing the occupancy map using the depth sensor. A non-learning agent could use this map at each step to make a greedy choice about its next action which greedily maximizes the coverage gain based on its current belief of the map. While the performance of random policy is shown in Fig.2 the performance of this greedy baseline is a better representative of the lower bound of the performance on the proposed task and problem setup.

-What is the performance of a learning-based method that only performs collision avoidance? Collision avoidance methods tend to implicitly learn to do a good map coverage. This simple baseline can show a tangible lower bound of a learning-based approach that does not rely on map.

-The major promise of the paper is that the proposed exploration task can improve navigation. However, the navigation experiment does not compare the proposed method with any of prior works in navigation. There is a huge list of prior methods for navigation some of which are cited in the “learning for Navigation” section of the related works and the comparison provided in Fig. 4 is incomplete compared to the state-of-the-arts in navigation. For example, while the curiosity driven approach is compared for the exploration, the more related curiosity based navigation method which uses both “exploration strategy” and “imitation learning” : “Pathak, Deepak, et al. "Zero-shot visual imitation." International Conference on Learning Representations. 2018.
“ is missed in navigation comparison. The aforementioned paper is also missed in the references.  

-Algorithmic-wise, it would make the argument of the paper clearer if results were conducted by running different exploration strategies for navigation to see if running RL with a good exploration strategy could solve the exploration challenge of the navigation problem without needing an explicit exploration stage (similar to the proposed method) which first explores and constructs the map and then does navigation by planning.

-The navigation problem as explained in section is solved based on planning approach that uses a reconstructed map. This is a fairly conventional approach that SLAM based methods use. Therefore, comparison with a SLAM method that constructs the map and then does navigation would be necessary. 


** Technical details are missing or not explained clearly**

- Section 3.1 does not clearly explain the map construction. It seems that the constructed map is just a 2D reconstruction of the space (and not 3D) using the depth sensor which does not need transformation of the 3D point cloud. What is the exact 3D transformation that you have done using the intrinsic camera parameters? This section mentions that there can be error in such map reconstruction because of robot noise but alignment is not needed because the proposed learning method provides robustness against miss-alignment. How is this justified? Why not using the known loop closure techniques in SLAM? 

-The technical details about the incorporated imitation learning method are missing. What imitation learning method is used? How is the policy trained during the imitation learning phase? 

-Last paragraph of intro mentions that the proposed method uses 3D information efficiently for doing exploration. The point of this sentence is unclear. What 3D information is used efficiently in the paper? Isn’t it only 2.5D (information obtained by depth sensor) used in the proposed method?

**Presentation can be improved**

-The left and right plots of the Figure 3 contains lots of repetitions which brings in confusion in comparing the performance of runs with different settings. These two plots should be presented in a single plot. 

- Interpretation of “green vs white vs black” in the reconstructed maps is left to the reader in Fig. 1. 

- Last line in page 5: there is no need for reiteration. It is already clear.

**Missing references**

-Since the paper is about learning to explore, discussion about “exploration techniques in RL” is recommended to be added in at least the related work section. 

-A big list papers for 3D map reconstruction is missing. Since the proposed method relies on a map reconstruction, those papers are relevant to this work and can potentially be used for comparison (as explained above). It is highly recommended that relevant prior 3D map reconstruction papers be added to the related work sections. 




This is a well explained and well executed paper on using classical SLAM-like 2D maps for helping a standard Deep RL navigation agent (convnet + LSTM) explore efficiently an environment and without the need for extrinsinc rewards. The agent relies on 3 convnets, one processing RGB images, one the image of a coarse map in egocentric referential, and one of the image of a fine-grained map in egocentric referential (using pre-trained ResNet-18 convnets). Features produced by the convnets are fed into a recurrent policy trained using PPO. Two rewards are used: the increase in the map's coverage and an obstacle avoidance penalty. The agent is further bootstrapped through imitation learning in a goal-driven task executed by a human controlling the agent. The authors analyze the behavior of the navigation algorithm by various ablations, a baseline consisting of Pathak's (2017) Intrinsic Curiosity Module-based navigation and, commendably, a classical SLAM baseline with path planning to empty, unexplored spaces.

Using an explicit map is a great idea but the authors need to acknowledge how hand-engineered all this is, when comparing it to actual end-to-end methods. First, the map reconstruction is done by back-projections of a depth image (using known projective geometry parameters) onto a 3D point cloud, then by slicing it to get a 2D map, accumulated over time using nearly perfect odometry. SLAM was an extremely hard problem to start with, and it took decades and particle filters to get to the quality of the images shown in this paper as obvious. Normally, there is drift and catastrophic map errors, whereas the videos show a nearly perfect map reconstruction. Is the motion model of the agent unrealistic? Would this ever work out of the box on a robot in a real world? The authors brush off the need for bundle adjustment, saying that the convnet can handle noisy local maps. Second, how do you get and maintain such nice ego-centric maps? Compared to other end-to-end work on learning how to map (see Wayne et al. or Zhang et al. or Parisotto et al., referred to later in the paper), it looks like the authors took a giant shortcut. All this SLAM apparatus should be learned!

One crucial baseline that is missing is that of explicit extrinsic rewards encouraging exploration. These rewards merely scatter reward-yielding objects throughout the environment; over the course of an episode, an object reward that is picked does not re-appear until the next exploration episode, meaning that the agent needs to cover the whole space to forage for rewards. Examples of such rewards have been published in Mnih et al. (2016) "Asynchronous methods for deep reinforcement learning" and are implemented in DeepMind Lab (Beatie et al., 2016). Such an extrinsic reward would be directly related to the increase of coverage.

A second point of discussion that is missing is that of the collision avoidance penalty: roboticists working on SLAM know well that they need to keep their robot away from plain-texture walls, otherwise the image processing cannot pick useful features for visual odometry, image matching or ICP. What happens if that penalty is dropped in this navigation agent?

Finally, the authors mention the Neural Map paper but do not discuss Zhang et al. (2017) "Neural SLAM" or Wayne et al. (2018) "Unsupervised Predictive Memory in a Goal-Directed Agent", where a differentiable memory is used to store map information over the course of an episode and can store information relative to the agent's position and objects' / obstacles' positions as well.

Minor remark: the word "finally" is repeated twice at the end of the introduction.

Meta-review: The authors have proposed an approach for directly learning a spatial exploration policy which is effective in unseen environments. Rather than use external task rewards, the proposed approach uses an internally computed coverage reward derived from on-board sensors. The authors use imitation learning to bootstrap the training and then fine-tune using the intrinsic coverage reward. Multiple experiments and ablations are given to support and understand the approach. The paper is well-written and interesting. The experiments are appropriate, although further evaluations in real-world settings really ought to be done to fully explore the significance of the approach. The reviewers were divided, with one reviewer finding fault with the paper in terms of the claims made, the positioning against prior art, and the chosen baselines. The other two reviewers supported publication even after considering the opposition of R1, noting that they believe that the baselines are sufficient, and the contribution is novel. After reviewing the long exchange and discussion, the AC sides with accepting the paper. Although R1 raises some valid concerns, the authors defend themselves convincingly and the arguments do not, in any case, detract substantially from what is a solid submission.

Decision: accepted-poster-papers