Rapid Exploration for Open-World Navigation with Latent Goal Models
Authors: Dhruv Shah
Benjamin Eysenbach
Nicholas Rhinehart
Sergey Levine
Venue: CoRL2021 Oral (2021)
Keywords: robot learning
navigation
exploration

Abstract: We describe a robotic learning system for autonomous exploration and navigation in diverse, open-world environments. At the core of our method is a learned latent variable model of distances and actions, along with a non-parametric topological memory of images. We use an information bottleneck to regularize the learned policy, giving us (i) a compact visual representation of goals, (ii) improved generalization capabilities, and (iii) a mechanism for sampling feasible goals for exploration. Trained on a large offline dataset of prior experience, the model acquires a representation of visual goals that is robust to task-irrelevant distractors. We demonstrate our method on a mobile ground robot in open-world exploration scenarios. Given an image of a goal that is up to 80 meters away, our method leverages its representation to explore and discover the goal in under 20 minutes, even amidst previously-unseen obstacles and weather conditions. Please check out the project website for videos of our experiments and information about the real-world dataset used at https://sites.google.com/view/recon-robot.

Rapid Exploration for Open-World Navigation with Latent Goal Models
Dhruy Shah', Benjamin Eysenbach", Nicholas Rhinehart', Sergey Levine!
UCC Berkeley Carnegie Mellon University
sites. gooele. hot////////eew/eeecorroootttt
Abstract: We describe robotic learning system for autonomous exploration and navigation in diverse, open- world environments. At the core of our method is a learned latent variable model of distances and actions, along with a non-parametric topological memory of images. We use an information bottleneck to regularize the learned policy, giving us (i) a compact visual representation of goals, (ii) improved generalization capabilities, and (iii) a mechanism for sampling feasible goals for exploration. Trained on large offline dataset of prior experience, the model ac- quires representation of visual goals that is robust to inse-intelevantt distractors. We demonstrate our method on a mobile ground robot in open-world exploration scenarios. Given an image of goal that is up to 80 meters away, our method leverages its representation to explore and discover the goal in under 20 minutes, even amidst previously-unsed obstacles and weather conditions.
1 Introduction
Robustness is a key challenge in learning to navigate diverse, real-world environments. A robotic learning system must be robust to the difference between an offline training dataset and the real world (i.e., it must generalize), be robust to non-stationary changes in the real world (i.c., it must ignore visual distractors), and be equipped with mechanisms to actively explore to gather information about traversability. Different environments may exhibit similar physical structures, and these similarities can be used to accelerate exploration of new environments. Learning-bor based methods provide an appealing approach for learning a representation of this shared structure using prior experience.
In this work, we consider the problem of navigating to user- specified goal in previously unseen environment. The robot has access to a large and diverse dataset of experience from other en- vironments, which it can use to learn general navigational affor- dances. Our approach to this problem uses an information bot- tleneck architecture to learn compact representation of goals. Learned from prior data, this latent goal model encodes prior knowledge about perception, navigational affordances, and short- horizon control. We use a non por-mmamttr memory to incorporate experience from the new environment. Combined, these compo- nents enable our system to learn to navigate to goals in a new envi- ronment after only few minutes of exploration.
Figure 1:
We demonstrate RECON on Clcarpath Jackal.
The primary contribution of this work is a method for exploring novel environments to discover user- specified goals. Our method operates directly on stream of image observations, without relying on structured sensors or geometric maps. An important part of our method is a compressed representa- tion of goal images that simultaneously affords robustness while providing simple mechanism for exploration. Such a representation allows us, for example, to specify a goal image at one time of day, and then navigate to that same place at different time of day: despite variation in appearance, the latent goal representations must be sufficiently close that the model can produce the correct ac- tions. Robustness of this kind is critical in real-world settings, where the appearance of landmarks can change significantly with times of day and seasons of the year.
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

Exploration Path
O Start @ Goal Image
Discovered Path
Map Node
(d) Training Environments
(a)
(b)
(c) Unseen Test Environments
Figure 2: System overview: Given a goal image (a), RECON explores the environment (b) by sampling prospective latent goals and constructing topological map of images (white dots). operating only on visual observations. After finding the goal (c), RECON (red path (b)). RECON uses data collected from diverse training environments (d) learn navigational priors that enable to quickly explore and learn to reach visual goals a variety of unseen environments (e).
the map reach arbitrary goals in the environment
We demonstrate our method, Rapid Exploration Controllers for Outcome-driven Navigation (RE- CON), on mobile ground robot (Fig. 1) and evaluate against competitive baselines spanning over 100 hours of real-world experiments in distinct open-world environments (Fig. 2). Our method can discover user-specified goals up to 80m away after just 20 minutes of interaction in a new environ- ment. We also demonstrate robustness in the presence of visual distractors and novel obstacles. We make this dataset publicly available as a source of real-world interaction data for future resesarch.
2 Related Work
Exploring a new environment is often framed as the problem of efficient mapping, posed in terms of information maximization guide the robot to uncertain regions of the environment. Some prior exploration methods use local strategies for generating control actions for the robots [1-4], while others use use global strategies based on the frontier method [5-7]. However, building high-fidelity geometric maps can be hard without reliable depth information. Further, such maps do not encode semantic aspects of traversability, c.g., tall grass is traversable but wire fence is not.
Inspired by prior work [8-12], we construct a topological map by learning a distance function and low-level policy. We estimate distances via supervised regression and learn local control policy via goal-conditioned behavior cloning [13, 14]. However, these prior methods do not describe how to learn navigate in new, unseen environments. We equip RECON with explicit mechanism for exploring new environments and transferring knowledge across environments.
Well-studied methods for exploration in reinforcement learning (RL) utilize novelty- -baed bonus, computed from a predictive model [15-21], information gain [22, 23], or methods based on counts, densities, distance from previously-visited states [24-26]. However, these methods learn to reason about the novelty of a state only after visiting it. Recent works [27, 28] improve upon this by predicting explorable areas for interesting parts of the environment to accelerate visual exploration. While these methods can yield state-of-the-arr results in simulated domains [29, 30], they come at the cost of high sample complexity (over 1M samples) and infeasible train in open- world environments without a simulated counterpart. Instead, our method enables the robot to explore an environment from scratch in just 20 minutes, using prior experience from other environments.
The problem of reusing experience across tasks studied in the context of meta-lcarning [31-33] and transfer learning [34-38]. Our method uses information bottleneck [39], which serves dual purpose: first, it provides a representation that can aid the generalization capabilities of RL algorithms [40, 41], and second, it serves as a measure of task-relevant uncertainty [42], allowing us to incorporate prior information for proposing goals that are functionally-reelvanr for learning control policies in the new environment.
The problem of learning gial-directed behavior has been studied extensively using RL [43-46] and imitation learning (II.) [13, 14, 47-50J. Our method builds upon prior goal-conditioned IL methods to solve slightly different problem: how to reach goals in a new environment. Once placed in new environment, method explores by carefully choosing which goals to visit, inspired by prior

---

work [51-55]. Unlike these prior methods, however, our method makes use of previous experience
in different environments to accelerate learning in the current environment.
3 Problem Statement and System Overview
We consider the problem of goal-directed exploration for visual navigation in novel environments: robot is tasked with navigating goal location G, given image observation Og taken at G. Broadly, this consists of three separate stages: (I) learning from offline data, (2) building a map in new environment, and (3) navigating to goals in the new environment. We model the task of navigation as a Markov decision process with time-indexed states 8, E S and actions d, E A. We do not assume the robot has access to spatial localization or map of the environment, or access to the system dynamics. We use videos of robot trajectories in a variety of environments to learn general navigational skills and build a compressed representation of the perceptual inputs, which can be used to guide the exploration of novel environments. We make no assumption on the nature of the trajectories: they may be obtained by human teleoperation, self-exploration, or as a result of preset policy. These trajectories need not exhibit intelligent behavior. Since the robot only observes the world from a single on- on-board camera and does not run any state estimation, our system operates in partially observed setting. Our system commands continuous linear and angular velocities.
3.1 Mobile Robot Platform
We implement RECON on a Clearpath Jackal UGV platform (see Fig. 1). The default sensor suite consists of 6-DoF IMU, GPS unit for approximate global position estimates, and wheel encoders to estimate local odometry. In addition, we added forward- 170 sold-of-view RGB camera and an RPLIDAR 2D laser scanner. Inside the Jackal is an NVIDIA Jetson TX2 computer. The GPS and laser scanner can become unreliable in some environments [56], so we use them solely safety controllers during data collection. Our method operates only using images taken from the onboard RGB camera, without other sensors ground-trudh localization.
3.2 Sele-Supervised Data Collection Labeling
Our aim is to leverage data collected in wide range of different environments to enable the robot to discover and learn to navigate to novel goals in novel environments. We curate dataset of self-supervised trajectories collected by time-correlated random walk in diverse real-world envi- ronments (see Fig. (d,e)). This data was collected over a span of 18 months and exhibits significant variation in appearance due seasonal and lighting changes. We make this dataset publicly avail- able' and provide further details in Appendix A.
4 RECON: A Method for Goal-Directed Exploration
Our objective is to design a robotic system that uses visual observations to efficiently discover and re- liably reach a target image in previously unseen environment. RECON consists two components that enable it to explore new environments. The first is an uncertainty- representation of goals that can quickly adapt to novel scenes. The second component is a topo- logical map, where nodes represent egocentric observations and edges are the predicted distance between them, constructed incrementally from Froseier-based exploration, maintaining a compact memory of the target environment.
context-conditionel
4.1 Learning Represent Goals
Our method learns compact representation of goal images that is robust to task-irrelevan factors of variation. We learn this representation using variant of the information bottleneck architec- ture [42, 57]. We use a context-conditione representation of goals to lear control policy in the target environment (Fig. 3 describes the graphical model). Letting I(-:-) denote mutual information, the objective in Eq. encourages the model to compress the incoming goal image 0g into a repre-
Available for download sites. cectees.gogle.cooo ptt.g//...o/////ww//eeootttttttt

---

sentation of conditioned on the current observation or that is predictive of the best action of and the temporal distance de to the goal (upper-case denotes random variables):
I((A), - (
(1)
Following [42], we approximate the intractable objective in Eq. with a variational posterior and decoder (an upper bound), resulting in the maximization objective:
L
Σ (o
[log go (al: df zl. o.)) - 3KL (pol. og. ())l)(())
(D]
where define the prior r(z)) & I) and D is a dataset trajectories characterized by (os, Og. a. de) quadruples. The first term measures the model's ability to predict actions and distances from the encoded rep- resentation, and the second term measures the model's compression of incoming goal images.
Decola (71g.17)
Z?
A
O₂
D!
As the encoder P& and decoder qe conditioned on ot. the representation z only encodes information about rel- ative location of the goal from the context this allows the model to represent feasible goals. If, instead, we had typical VAE (in which the input images autoencoded), the samples from the prior over these representations would not necessarily represent goals that are reachable from the current state. This distinction is crucial when exploring new environments, where most states from the training envi- ronments are not valid goals.
Figure 3: Graphical model of actions and distances
4.2 Goal-Directed Exploration with Topological Memory
The second component of our system is a topological memory constructed incrementally as the robot explores a new environment. map that the robot can use to later navigate to arbitrary goals. To build this memory, the robot uses the model from the previous section propose subgoals for data collection. Note that this is done in the exploration phase and have a latent goal model pre-trained on the offline dataset. Given a subgoal, our algorithm (Alg. 1) proceeds by executing actions towards the subgoal for fixed number of timesteps (Alg. L12). The data collected during subgoal navigation expands the topological memory (Alg. L.1) and is used to fine-tune the model (Alg. L15). Thus, the task of efficient exploration is reduced to the task of choosing subgoals.
provides an estimate of the exploration frontier well as
Algorithm RECON for Exploration: RECON takes as input an encoder Pas a decoder qe, prior r, the current observation and goal observation 0g. 51, 62, e, E 14; H, E N are hyperparameters.
1: function RECON (q6. Ps, ot, Ogi 51, 82, 3,7, H)
2: + 0, - O
Initialize graph and data Continue while goal
3:
while not reached goal (d] < 61] do
4: On + L-aast ELeastExplordNNeigbbbrrr (9 02)
5:
27 ~ ps(s ot, 0g)
D Encode relative goal
6:
if goal is feasible [r(=]) > e] then
7:
21 += =
Will go to the goal
8:
else robot at frontier (dp < 6 then
9:
z²- ~r(z)
Will explore from frontier
10:
else
2/th "xpp(2> oc, on) Dm. Oz -- SubgoalNavigate(?"" H)
Will go to frontier
12:
D DUD
14:
ExpandGraph ot)
Step L(ò, 0: D. 9) for epochs
> Eq. 2
16:
return networks pa go and graph g

---

Subgoals are represented by latent variables in our model, which may either come from the posterior pa(z|ot, 0g), or from the prior
Algorithm 2 RECON for Goal-Reaching: After exploration, RECON uses the topological graph quickly navigate towards the goal Og.
r(z). Given a subgoal and observation ot, the model decodes it into an action and dis- tance pair q(a;, |i |z, or); the action is used to control the robot towards the goal, and the dis- tance is used to construct edges in the topolog- ical graph. The choice of intermediate subgoal
1: procedure GoalNavigate(G, or, og; H)
2:
Vt f- Associate To VesootiateoVVrtee((... 0t)
3:
Ug f Associate Vertex o₂)
4:
(Ur...., Ug) Shortest Path U2, Un))
5:
for U E (Vt..... 22) Myydd do
to navigate toward at any step is based on the robot's estimate of the goal reachability and its proximity to the frontier. To determine the fron- 7:
6:
Po 0g v.O) Dw. 0s t- SubgoalNarigate(22 H)
tier of the graph, we track the number of times each node in the graph was selected as the navigation goal; nodes with low counts are considered to be on the frontier. In the following,
z denote the mean of the encoder (o(= (= 0g) and
de to denote the distance component of the mean of the decoder qo(as, de 2. 0x) (i.e., the predicted number of time steps from o, to a,). The choice of subgoal at each step is made as follows:
The robot believes it can reach the goal and adopts the representation of the goal image as the subgoal (Alg. 1.7). The robot's confidence in reaching the goal based on the probability of the current goal embedding of under the prior r(z). Large r(=;)) implies the relationship between the observation on and the goal 0g is in-distribution, suggesting that the model's estimates of the distances is reliable intuitively, this means that the model is confident about the distance to Og and can reach it.
(i) Feasible Goal:
The robot is at the "least-explored node" (frontier) On and explores by sampling a random conditional subgoal latent 2 from the prior (Alg. L9). The robot determines whether it is at the frontier based on the distance (estimated by querying the model) to its "least explored neighbor" di the node in the graph within a distance threshold (62) of the current obser- vation that has the lowest visitation count. If the distance to this node di is low (threshold 81), then the robot is at the frontier.
(ii) Explore at Frontier:
(iii) Go to Frontier: The robot adopts its "Teast-explored neighbor" on as a subgoal (Alg. I LII).
The SubgoalNavigate function rolls out the learned policy for fixed time horizon H to navigate to the desired subgoal latent 2/", by querying the decoder qe (4,, de|z,", 07) with a fixed subgoal latent. The endpoint of such rollout is used to update the visitation counts in the graph g. At the end of each trajectory, the ExpandGraph subroutine is used to update the edge and node sets {E, V} of the graph to update the representation of the environment. We provide the pseudocode for these subroutines in Appendix B. We also share broader implementation details including choice of hyperparameters, model architectures and training details in Appendix B.2.
4.3 System Summary
RECON uses the latent goal model and topological graph to quickly explore new environments and discovers user-specified goals. Our complete system consists of three stages:
A) Prior Experience: The goal-conditioned distance and action model (Sec. 4.1) is trained using experience from previously visited environments. Supervision for training our model is obtained by using time steps as a proxy for distances and a relabeling scheme (Appendix A).
B) Exploring Novel Environment: When placed in a new environment, RECON uses combina- tion of frontier-based exploration and latent goal-sampling with the learned model. The learned model is also fine-tuned to this environment. These steps are summarized in Alg. and Sec. 4.2.
C) Navigating an Explored Environment: Given an explored environment (represented by a topo- logical graph g) and the model, RECON uses G to navigate to goal image by planning path of subgoals through the graph. This process is summarized in Alg. 2.
5 Experimental Evaluation
We designed our experiments answer four questions:
5

---

Method
Expl. Time (mm:ss)
Nav. Time (mm:ss)
SCT [58] f
PPO RND [18]
21:18
00:47
0.22
InfoBot [41]
23:36
00:48
0.21
Active Neural SLAM (ANS) [21]
21:00
00:45
0.33
ViNG [11]
19:48
00:34
0.60
Ours + Episodic Curiosity (ECR) [20]
14:54
00:31
0.73
RECON (Ours)
09:54
00:26
0.92
Table 1:
Exploration and goal reaching performance: Exploring real-world environments, RE-
CON reaches the goal 50% faster than the best baseline (ECR). ANS takes up to 2x longer to find the goal and NTS [27] fails to find the goal in every environment. On subsequent traversals, RECON navigates to the goal 20 85% faster than other baselines, and exhibits > 30% higher weighted success.
Q1. How does RECON compare to prior work for visual goal discovery in novel environments?
Q2. After exploration, can RECON leverage its experience navigate to the goal efficiently?
Q3. What is the range of perturbations and non-stationary elements to which RECON robust?
Q4. How important are the various components of RECON, such as sampling from an information bottleneck and con-parametric memory, its performance?
5.1 Goal-Directed Exploration in Novel Environments
Aerial soal
Trajectory
(Irgge)
spal
IntoBot (41]
RECON-ECO ECR [20)
Time
N27
RND [18]
Figure Visualizing goal-reaching behavior of the system: (left) Example trajectories to goals discovered by RECON in previously unseen environments. (right) Policies learned by the different methods in one such environment. Only RECON and ECR reach the goal successfully, and RECON takes the shorter route.
We perform evaluation in diverse variety of outdoor environments (examples in Fig. 2), in- cluding parking lots, suburban housing, sidewalks, and cafeterias. We train our self-supervised navigation model using an offline navigation dataset (Sec.3.2) collected distinct set of training environments, and evaluate our system's 's ability to discover user-specified goals in previously unseen environments. We compare RECON to five baselines, each trained the same 20 hours of offline data as our method, and finetuned in the target environment with online interaction.
1. PPO + RND: Random Network Distillation (RND) widely used prediction bonus-based exploration strategy in RL [18], which use with PPO [59, 60). This comparison represen- tative of a frequently used approach for exploration in RI. using novelty bonus.
2. InfoBot: An offline variant of InfoBot [41], which uses goal-conditioned information bottle-
neck, analogous to our method, but does not use the nan-parametric memory.
3. Active Neural SLAM (ANS): A popular indoor navigation approach based on metric spatial maps proposed for coverage- maxeerag--mximizinggg exploration [21]. We adapt it to the gial-dieected task by using the distance function from RECON to detect when the goal is nearby.
4. Visual Navigation with Goals (ViNG): A method that uses random action sequences to explore and incrementally build a topological graph without reasoning about visitation counts [11].

---

5. Episodic Curiasity (ECR): A method that executes random action sequences at the frontier of topological graph for exploration [20]. We implement this as an ablation of our method that samples random action sequences, rather than rollouts to sampled goals (Alg. Line 7).
We evaluate the ability of RECON to discover visually- indicated goals in 8 unseen environments and navigate to them repeatedly. For each trial, we provide an RGB image of the desired target (one per environment) to the robot and report the time taken by each method to (i) discover the desired goal (Q1), and (ii) reliably navigate to the discovered goal second time using prior exploration (Q2). Additionally, we quantify navigation performance using Success weighed by Completion Time (SCT), success metric that takes into account the agent's 's dynamics [58]. We show quantita- tive results in Table 1, and visualize sample trajectories of RECON and the baselines in Fig. 4.
RECON outperforms all the baselines, discovering goals that are up to 80m away in under 20 min- utes, including instances where no other baseline can reach the goal successfully. RECON+ECR and ViNG discover the goal in only the easier environ- ments, and take up to 80% more time to discover the goal in those environments. RND, InfoBot and ANS are able to discover goals that are up to 25m away but fails to discover more distant goals, likely because using reinforcement Icarning for fine-tuning data-inefficient. We exclude reporting metrics on NTS, which fails to successfully explore any envi- ronment, likely due overfitting to the offline tra- jectories. Indeed, the simulation experiments re- ported in each of these online algorithms require upwards of 1M timesteps to adapt to new environ- ments [21, 27, 41J. We attribute RECON's success to the context-conditioned sampling strategy (de- scribed in Sec. 4.1), which proposes goals that can accelerate the exploration of new environments.
(e) Image
(e)
(4)
Car
Novel Obstacles
Orcrced
Twilight
Varying Lighting Conditions
Figure Exploring non-stationary environ- ments: The Iearned representation and topolog- ical graph is robust to visual distractors, enabling reliable navigation to the goal under novel ohsta- cles (c-e) and appearance changes (f-h).
We then study RECON's ability to quickly reach goals after initial discovery. Table shows that RE- CON variants are able to quickly recall feasible path to the goal. These methods create a compact topological map from experience in the target environment, allowing them to quickly reach previously-seen states. The other baselines are unsuc- cessful at recalling previously seen goals for all but the simplest environments. Fig. shows an aerial view of the paths recalled by various methods in one of the environments. Only the RECON variants are successfully able to navigate to the checkerboard goal; all other baselines result in collisions in the environment. Further, RECON discovers shorter path to the goal and takes 30% less time to navigate to it than ECR ablation.
5.2 Exploring Non-Stationary Environments
Outdoor environments exhibit non-stationarity due to dynamic obstacles, such as automobiles and people, as well as changes in appearance due to seasons and time of day. Successful exploration and navigation in such environments requires learning a representation that is invariant to such distrac- tors. This capability is central interest when using a non-parametric memory: for the topological map remain valid when such distractors are presented, must ensure the invariance of the learned representation to such factors (Q3).
To test the robustness of RECONto unseen obstacles and appearance changes, we first had RE- CON explore new "junkyard" to learn reach a goal image containing blue dumpster (Fig. 5- a). Then, without any more exploration, we evaluated the learned goal-reaching policy when pre- sented with previously unseen obstacles (trash cans, traffie cones, and a car) and and weather condi- tions (sunny, overcast, and twilight). Fig. 5 shows trajectories taken by the robot as it successfully navigates to the goal in scenarios with varying obstacles and lighting conditions. These results suggest that the learned representations are invariant to visual distractors that do not affect robot's
7

---

Start
Exlumiton Fronierr
Expl.
Nav.
SCT
Method
Time
Time
[58]
(mm:ss)
(mm:ss)
Reactive
11:54
00:37.4
0.63
Random Actions
14:54
00:31.4 .4
0.73
Vanilla Sampling
14:06
00:28.7
0.83
Ours
09:56
00:25,8
0.92
Figure 6: Exploration via sampling from our context-condiionne prior (right) allows the robot to explore times faster than using random actions, c.g. in ECR [20] (left).
Table 2: Allation experiments confirm the importance of using an information bottleneck and a non- parmpeaacttric memory.
decisions reach a goal (e.g., changes in lighting conditions do not affect the trajectory goal, and hence, are discarded by the bottleneck).
5.3 Dissecting RECON
RECON explores by sampling goals from the prior distribution over state-goal representations. To quantify the importance of this exploration strategy (Q4), deploy RECON to perform undirected exploration in a novel target environment without building graph of the environment. We compare the coverage of trajectories of the robot over 5 minutes of exploration when: (a) it executes random action sequences [20], and (b) it performs rollouts towards sampled goals. We see that performing rollouts to sampled goals results in 5x faster exploration in novel environments (see Fig. 6).
We also evaluate several variants of RECON that ablate its goal sampling and non- non-parametric mem- ory on the end-to-end task of visual goal discovery in novel environments:
Reactive: our method deployed without the topological graph for memory.
Random Actions: a variant our method that executes random action sequences at the frontier rather than rollouts to sampled goals. This is identical to the ECR baseline described in Sec. 5.1.
Vanilla Sampling: a variant of our method which learns goal-conditioned policy and distances without information bottleneck to obtain compressed representations.
We deploy these variants in subset of the unseen test environments and summarize their perfor- mance in Table 2. These results corroborate the observations in Fig. 6: learning a compressed goal representation is key to the performance of RECON. "Vanilla Sampling", despite sampling from a joint prior, performs poorly and is unable to discover distant goals. We hypothesize that our method is more robust because the information bottleneck helps learn representation that ignores task- irrelevantt information. We also observe that "Reactive" experiences smaller degradation in exploration performance, suggesting that goal-sampling can help with the exploration problem even without the graph. However, we find a massive degradation in its ability to recall previously discov- ered goals, suggesting that the memory is key to the navigation performance of RECON.
Discussion
We proposed system for efficiently learning goal-directed policies in new open-world environ- ments. The key idea behind method is to use a learned goal-conditioned distance model with a latent variable model representing visual goals for rapid goal-directed exploration. The problem setup studied in this paper, using past experience to accelerate learning in a environment, is reflective real-world robotics scenarios: collecting new experience at deployment time is costly, but experience from prior environments provide useful guidance solve new tasks.
In future work, we aim to provide theoretical guarantees for when and where we can expect stochas- tic policies and the information bottleneck to provide efficient exploration. One limitation of the current method that does not explicitly account for the value of information. Accounting for such states generate a better goal-ieaching policy.
8

---

Acknowledgments
This research was supported by ARL DCIST CRA W911NF-17-2-0181, DARPA Assured Auton- omy, and the Office of Naval Research. The authors would like to thank Suraj Nair and Brian Ichter for useful discussions, and Gregory Kahn for setting up the infrastructure used for autonomous collection of real-world data.
References
[1] B. Kuipers and Y.-T. Byun, "A robot exploration and mapping strategy based on a semantic hierarchy of spatial representations"" Robotics and Autonomous Systems, 1991, special Issue Toward Learning Robots.
[2] F. Bourgault, A. A. Makarenko, S. B. Williams, B. Grocholsky, and H. F. Durrant- Whyte, "Information based adaptive robotic exploration." in IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), vol. 1, 2002, pp. 540-545 vol.1.
[3] Kollar and N. Roy, "Efficient optimization of information- exploration in slam," in Proceedings of the AAAI Conference on Artificial Intelligence (AAAT), 2008.
[4] W. Tabib, K. Goel, J. Yao, M. Dabhi, C. Boirum, and N. Michael, "Real-time information- theoretic exploration with gaussian mixture model maps." in Robotics: Science and Systems, 2019.
[5] B. Yamauchi, "A frontier-base approach for autonomous exploration," IEEE International Symposium Computational Intelligence in Robotics and Automation (CIRA), 1997.
[6] B. Charrow, S. Liu, V. Kumar, and N. Michael, "Informatioonthhoreti mapping using cauchy- schwarz quadratic mutual information," in IEEE International Conference on Robotics and Automation (ICRA), 2015.
[7] D. Holz, N. Basilico, F. Amigoni, and S. Behnke, "A comparative evaluation of exploration strategies and heuristics to improve them," 2011.
[8] N. Savinov, A. Dosovitskiy, and V. Koltun, Deemm-Paramttric Topological Memory for Navi- gation," in International Conference on Learning Representations, 2018.
[9] A. Faust, K. Oslund, O. Ramirez, A. Francis, L. Tapia, M. Fiser, and Davidson, "Prm- rl: Long-range robotic navigation tasks by combining reinforcement learning and sampling- based planning." " in 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2018. pp. 5113-5120.
[10] B. Eysenbach, R. R. Salakhutdinov, and S. Levine, "Search on the Replay Buffer: Bridging Planning and RL." in Advances in Neural Information Processing Systems (NeurIPS), 2019.
[11] D. Shah, B. Eysenbach, G. Kahn, N. Rhinehart, and S. Levine, "ViNG: Learning Open- World Navigation with Visual Goals," in IEEE International Conference on Robotics and Automation (ICRA), 2021.
[12] X. Meng, N. Ratliff, Y. Xiang, and D. Fox, "Scaling Local Control to Large-Scale Topological Navigation," " in IEEE International Conference on Robotics and Automation (ICRA), 2020.
[13] D. Ghosh, A. Gupta, Fu, Reddy, C. Devin, B. Eysenbach, and S. Levine, "Learning to reach goals without reinforcement learning," arXiv preprint arXiv:1912.06688 2019.
[14) C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet, "Learning latent plans from play." in Conference
Robot Learning. PMLR, 2020, pp. 1113-1132.
(15) B. C. Stadie, S. Levine, and P. Abbeel, "Incentivizing exploration in reinforcement learning with deep predictive models," arXiv preprint arXiv: /507.00814. 2015.
[16] D. Pathak, P. Agrawal, A. A. Efros, and Darrell, "Curiosity-driven exploration by self- supervised prediction," in International Conference on Machine Learning. PMLR, 2017, pp.
[17] R. Bajesy, Y. Aloimonos, and J. K. Tsotsos, "Revisiting active perception." Autonomous Robots, vol. 42, no. 2, pp. 177-196, 2018.
[18] Y. Burda, H. Edwards, Storkey, and O. Klimov, "Exploration by random network distilla- tion." arXiv preprint arXiv: 18X.::810..1894 2018.

---

[19] Y. Satsangi. S. Lim, S. Whiteson. F. Olichoek, and M. White, "Maximizing information gain in partially observable environments via prediction reward." arXiv preprint (/Xiv:2005004912, 2020.
[20] N. Savinov, A. Raichuk, R. Marinier, D. Vincent, M. Pollefeys, T. Lillicrap, and S. Gelly, "Episodic curiosity through reachability," International Conference on Learning Representa- rions (ICLR, 2019.
[21] D. Chaplot, D. Gandhi, Gupta, A. Gupta, and R. Salakhutdinov, "Learning to Explore us- ing Active Neural SLAM," in International Conference on Learning Representations (ICLR), 2020.
[22] R. Houthooft, X. Chen, Y. Duan, J. Schulman, F. De Turck, and P. Abbeel, "Vime: Variational information maximizing exploration." arXiv preprint arXiv:1605.0966 2016.
[23] A. Mirchev, B. Kayalibay, M. Soelch, P. van der Smagt, and J. Bayer, "Approximate bayesian inference in spatial environments," arXiv preprint
2018.
[24] M. G. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos, "Unifying count-based exploration and intrinsic motivation," " arXiv preprint arXiv: 1606.01868, 2016.
[25] E. Hazan, S. Kakade, K. Singh, and A. Van Soest, "Provably efficient maximum entropy ex- ploration," " in International Conference on Machine Learning. PMLR, 2019, pp. 2681-2691.
[26] L. Lee, B. Eysenbach, E. Parisotto, E. Xing, S. Levine, and R. Salakhutdinov, "Efficient explo- ration via state marginal matching," arXiv preprini 1rXiv:1906.00274, 2019.
[27] D. Singh Chaplot, R. Salakhutdinov, A. Gupta, and S. Gupta, "Neural topological slam for visual navigation," " in IEEF/CVF Conference on Computer Vision and Patlern Recognition (CVPR), 2020.
[28] D. S. Chaplot, H. Jiang, S. Gupta, and A. Gupta, "Semantic curiosity for active visual learning." in ECCV, 2020.
[29] Manolis Savva", Abhishek Kadian*, Oleksandr Maksymets*, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun, J. Malik, D. Parikh, and D. Batra, "Habitat: A Platform for Embod- ied AI Research," in IEEE/CVE International Conference on Computer Vision (ICCV), 2019.
[30] E. Kolve, R. Mottaghi, W. Han, E. VanderBilt, L. Weihs, A. Herrasti, D. Gordon, Y. Zhu, A. Gupta, and A. Farhadi, "AI2-THOR: An Interactive 3D Environment for Visual AI," ArXiv, vol. ab5/1712.054744 2019.
[31] Y. Duan, J. Schulman, X. Chen, P. L. Bartlett, I. Sutskever, and P. Abbeel, "RI2: Fast reinforce- ment learning via slow reinforcement learning," arXiv preprint arXiv: /6/ 1.02779, 2016.
[32] S. Sæmundsson, K. Hofmann, and M. P. Deisenroth, "Meta reinforcement learning with latent variable gaussian processes," arXiv preprint /003.0883.0555 2018.
[33] A. Nagabandi, I. Clavera, S. Liu, R. S. Fearing, P. Abbeel, S. Levine, and C. Finn, "Learning to adapt in dynamic, woaldworld environments through retf-reiiforreement learning," " arXiv preprint 1803. 2018
[34] M. E. Taylor and P. Stone, "Cross- transfer for reinforcement learning." in Proceedings of the 24th international conference on Machine learning, 2007, pp. 879-886.
[35] A. Lazaric, "Transfer in reinforcement learning: framework and survey," in Reinforcement Learning. Springer, 2012, pp. 143-173.
[36] E. Parisotto, J. L. Ba, and R. Salakhutdinov, "Actor-mimic: Deep multitask and transfer rein- forcement learning," " arXiv preprint 0rXiv:1511.063: 2015.
[37] A. Gupta, C. Devin, Y. Liu, P. Abbeel, and S. Levine, "Learning invariant feature spaces to transfer skills with reinforcement learning." arXiv preprint arXiv:1703.02949) 2017.
[38] S. Gamrian and Y. Goldberg, "Transfer learning for related reinforcement learning tasks via image-to-image translation," in International Conference on Machine Learning. PMLR, 2019, pp. 2073-2072.
[39] N. Tishby, F. C. Pereira, and W. Bialek, "The information bottleneck method," arXiv preprint physics/0004057,
2000. 2000.
10

---

[40] M. Igl. K. Ciosek, Y. Li, S. Tschiatschek, C. Zhang, S. Devlin, and K. Hofmann, "Generaliza- tion reinforcement learning with selective noise injection and information bottleneck," arXiv preprint arXiv: 1910. ,,
[41] A. Goyal, R. Islam, D. Strouse, Z. Ahmed, M. Botvinick, H. Larochelle, Y. Bengio, and
S. Levine, "Infobot: Transfer and exploration via the information bottleneck," arXiv preprini arXiv: 10902, 2019.
[42] A. A. Alemi, I. Fischer, V. Dillon, and K. Murphy, "Deep variational information bottleneck,"
arviv preprini arXiv: 1612000110 2016.
[43] L. P. Kaelbling, "Learning to achieve goals." in IJCAI. Citeseer, 1993, pp. 1094-1099.
[44] Schaul, Horgan, K. Gregor, and D. Silver, "Universal Value Function Approximators." in International Conference on Machine Learning (ICML), 2015.
[45] V. Pong. S. Gu, M. Dalal, and S. Levine, "Temporal difference models: Model-free deep rl for model-basee control," arXiv preprint arXiv: arXiiv:/822.0008/ 2018.
[46] B. Eysenbach, R. Salakhutdinov. and S. Levine, "C-learning: Learning to achieve goals via recursive classification," " arXiv preprint arXiv:2011.08909 2020.
[47] Y. Ding, C. Florensa, M. Phielipp, and P. Abbeel, "Goal-conditioned imitation learning," arXiv preprint arXiv: 90,00538, 2019.
[48] H. Sun, Z. Li, Liu, D. Lin, and B. Zhou, "Policy continuation with hindsight inverse dy- namics," arXiv preprint arXiv: /910. 14055, 2019.
[49] R. K. Srivastava, P. Shyam, F. Mutz, W. Jaškowski, and Schmidhuber, "Training agents using upside-down reinforcement learning," arXiv preprint arXiv: 2019.
[50] N. Rhinehart, R. McAllister, and S. Levine. "Deep imitative models for flexible inference, planning, and control," in International Conference on Learning Representations, 2020.
[51] V. H. Pong, M. Dalal, S. Lin, A. Nair, S. Bahl, and S. Levine, "Skew-fit: State-covering self- supervised reinforcement learning," arXiv preprint arXiv: 1903.0333333 2019.
[52] C. Colas, P. Fournier, M. Chetouani, O. Sigaud, and P.-Y. Oudeyer, "Curious: intrinsically motivated modular multi-goal reinforcement learning," in International conference on machine learning. PMLR, 2019, pp. 1331-1340.
[53] I.. Zhang, G. Yang, and B. C. Stadie, "World model graph: Learning latent landmarks for planning." arXiv preprint vrXiv:2011.12499 2020.
[54] Pitis, H. Chan, S. Zhao, B. Stadie, and Ba, "Maximum entropy gain exploration for long horizon multi-gon goal reinforcement learning," in International Conference on Machine Learning. PMLR, 2020, pp. 7750-7761.
[55) K. Liu, T. Kurutach, C. Tung. P. Abbeel, and A. Tamar, "Hallucinative topological memory for zero-shot visual planning," " in International Conference on Machine Learning. PMLR, 2020,
pp. 6259-6270.
[56) G. Kahn, P. Abbeel, and S. Levine, "BADGR: An Autonomous Self-Supervised Learning- Based Navigation System," " 2020.
[57] A. Achille and S. Soatto, "Emergence of invariance and disentanglement in deep representa- tions," The Journal of Machine Learning Research, vol. 19, no. 1, pp. 1947-1980, 2018.
[58] N. Yokoyama, Ha, and D. Batra, "Success weighted by completion time: dynamics-awan evaluation criteria for embodied navigation," 2021.
[59] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, "Proximal Policy Optimiza- tion Algorithms," 2017.
[60] E. Wijmans, A. Kadian, A. Morcos, S. Lee, I. Essa, D. Parikh, M. Savva, and D. Batra, "DD- PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames," in International Conference on Learning Representations (ICIR), 2020.
[61] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang. T. Weyand, M. Andreetto, and
H. Adam, "MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applica- tions." " 2017.
[62] D. P. Kingma and M. Welling, "Auto-encoding variational bayes," t arXiv preprint 1:Xiv:1312.6114, 2013.
11


Reviews:
[{"review_id": "sJZBYvwn3qd", "reviewer": "Reviewer_gi9D", "summary": "Full disclosure; I was made aware of this paper by [name removed] tweeting about it in April: [link removed] and I’m also aware that this work was presented at an ICLR workshop this year: [link removed]. So at best, this is a single-blind review.\n\nThe paper presents a novel method for goal-driven visual outdoor navigation. The method is pretrained on a large dataset to propose goals for navigation and the online algorithm creates a graph of the environment and learns compressed visual embeddings that are robust to weather, light conditions, and distractor objects.\n\nThe work is overall very impressive and I respect the sheer amount of field experiments that were carried out for this work. However, I think that the work currently suffers from some clarity issues to the point of being impossible to reproduce without contacting the authors.\n\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- The content that made it into the paper is well-written and well-presented. The text itself is clear, the Figures are illustrative, the pseudocode blocks are mostly great, and the experimental section is excellent.\n- As mentioned above, the experimental time commitment in this work is staggering. The authors mentioned over 100 hours of real-world experiments and there was probably also robot maintenance and intermittent tuning of the method, so I have to commend the authors for that sheer effort!\n- I really appreciated that the authors clearly stated their research questions at the beginning of the experiments section.\n- The choice of recent baselines and amount thereof is great.\n\n### Weaknesses\n\n- Practicality: The authors mentioned that this was only possible by having a large offline dataset of the robot and also all the data is collected on the same robot platform. That raises many questions that weren't addressed in the paper:\n  1. You emphasized that the training and testing environments weren't the same. But how disjoint/out-of-distribution is the data? What's the closest distance between any possible sample from the train and test dataset. I feel like if they were recorded in the same neighborhoods, the test environment is very much in-distribution and some of the claims of generality need to be toned down a bit.\n  2. I'm unclear what the importance of the perspective is in this method - can the data be recorded from a different perspective (height above ground) or with a different visual sensor? What about a different FOV?\n  3. Part of your motivation is that this method uses \"only\" real-world data and doesn't need to be pretrained in simulation (see related works section line 66 and following) but also your \"...method enables the robot to explore an environment from scratch in just 20 minutes, using prior experience from other environments\". What about pretraining your method in simulation? Have you looked into that? Would any of this work if the dataset was simulated? Because I feel like a large dataset for pretraining would be easier to come by if it were simulated.\n  4. You also mention in the main body of the paper that you're making the dataset publicly available (in the appendix?) (line 109) but I see no evidence of that. The dataset could have been uploaded to some anonymous google drive folder or cloud storage to be given to the reviewers for anonymous review. \n- Reproducibility: The present text raises many questions about how to reimplement this:\n\t5. You mention in the main body of the paper that the data is collected over 18 months through random walk but in the appendix, you further specify that the data is not random but obtained from a heuristic that resets after collision and also something about bumpiness and self-labeling. The reader is left utterly lacking any understanding of this dataset. (a) How many data points are there? (b) What is the distribution of the data, i.e. are there any over- or underexplored areas, what is the distribution of \"bumpinesses\" and collisions in your data? (c) did you preprocess the data in any way? (d) Did you filter the data to exclude certain episodes or events?\n\t6. How exactly is the pretraining done? (a) I.e. what parts are pretrained and for how long? (b) How is the prior $r(\\cdot)$ calculated?\n\t7. In Algorithm 1, there are a few unclear things - some of which are explained in the appendix. (a) the function LeastExploredNeighbor is taking a hyperparameter $\\delta_2$ and the purpose of this HP is unclear. This is explained in the appendix but from my best understanding, the appendix is there to provide additional examples, and additional clarification but the main body of the paper should be able to be clear and self-contained. (b) Same for the function SubgoalNavigate and the hyperparameter $H$. (c) What is $o_n$ in line 11? (d) If $r(\\cdot)$ is the prior over goals that returns the value of a normal distribution (i.e. a scalar, line 128), then how is a goal sampled from that in line 9? \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I think this paper has a lot of potential and I would like to see this accepted but to recommend acceptance, I first need some crucial details clarified (see weaknesses section above).", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "See section \"Weaknesses\" above. I numbered all my issues with this paper so the authors can respond in turn. Some additional questions, mostly out of interest:\n\n8. What's the formula for SCT in table 1?\n9. You mentioned moving objects (people) but I didn't see any in your Figures and video. How does your model work when there's foot traffic, qualitatively?"}}, {"review_id": "Y_uyoGecnlS", "reviewer": "Reviewer_fB76", "summary": "The paper proposes an approach to learn a latent representation of relative goals to be able to produce actions and distance estimates from a current image and a goal image. This model, trained on offline data, is used to build a topological map of a new environment, explore rapidly this new environment, and later navigate in this environment. Performances evaluations and comparisons with a large set of baselines are provided on a real robot in several outdoors environments, showing improved exploration and navigation speed. \n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is very well written, the motivation and relation to the state of the art are clear, the technical description is sufficient in the paper and well complemented in the appendix and supplementary video. At a high level, the proposed approach illustrates the well-known fact that goal-based exploration is much more efficient than random action sampling, but its implementation as a vision-based latent goal model seems an interesting contribution and its evaluation on real robots in challenging outdoor environments is quite impressive.\n\nI don't see any major problem with the paper, I have only a few questions that could be clarified :\n- What is the reward function used in the PPO baseline? Only sparse at the goal or using a more informative reward?\n- Could you illustrate the interest of fine-tuning the model? Is it important in the performances? And does the model specialize in the environment or integrates knowledge and remains valid for other environments (i.e. no forgetting)?\n- Line 195: you mention a relabeling scheme that is not described in sec 3.2 nor in the appendix.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper proposes an interesting new approach, with a solid experimental evaluation.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "See the minor weaknesses mentioned above.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "Eu3ExAvXjXg", "reviewer": "Reviewer_nTAe", "summary": "The paper addresses the problem of finding a goal image specified by a user in a possibly unknown environment using a single ground mobile robot. The paper proposes a method that uses learned goal-conditioned distance model with a latent variable model representing visual goals, trained on a variety of environments, to explore the environment, and uses a topological graph to navigate towards the found goal. The paper shows experimental results with a real ground robot in known and unknown environments to validate the proposed method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper presented an interesting learning system which improves the state of the art in image-goal navigation by handling new environments and at the same time using prior experience in other environments. The paper is generally well written, with provided motivations behind the choices made. The paper is technically sound with a rather complete experimental evaluation with a real robot that includes comparisons with other 6 methods in different settings and an ablation study. It is quite impressive the dataset that was collected over a span of 18 months.\n\nThe main comments about the paper revolve around the actual practicality of the proposed method and the lack of comparison with alternative non-learning-based approaches. Some of the questions include: in outdoor, what are some of the actual uses of the proposed method which does not use any other sensor other than camera, as GPS is typically reliable, especially considering the types of environments considered in the paper? While different environments have been considered, the environments share similar characteristics, i.e., some human-made structures with large passages; how does the method generalize to totally different environments, e.g., in forest? What happens when the goal is much further away than 80m? What is the motivation to use a pure learning-based approach for the full pipeline, when for example visual SLAM could be used to create a map; the robot could follow a coverage pattern; and eventually find the goal avoiding to have paths that look suboptimal? Discussions on these aspects would improve the quality and the impact of the paper.\n\nA minor comment regards the dataset used, which could be better described in terms of how the 20 hours of offline data.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "While the paper could discuss the practicality and compare with more traditional exploration systems, the paper presents an interesting method for image-goal navigation in unseen environments with a rather complete evaluation that includes a real robot.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "The revised paper could discuss the questions raised in the \"Strengths and Weaknesses\" box.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "DiECuCmA1G", "reviewer": "Reviewer_7dLW", "summary": "This paper presents a robot learning system for goal-directed exploration in novel environments where the robot is tasked to navigate to a goal region in the environment while provided with a goal image taken at the goal region. The system, called Rapid Exploration Controllers for Outcome-driven Navigation (RECON) consists of two parts: An autoencoder that serves as the policy and distance estimator, given the goal image and the current image of the robot, and a topological map that is constructed online to guide the robot towards exploring the environment efficiently. The system is evaluated and compared to 6 baseline methods on a set of previously unseen real-world scenarios.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, the method is interesting, albeit somewhat incremental, and the motivations behind the two parts of the system (i.e. the autoencoder and the topological map) are clear. The paper does a good job in specifying the problem it attempts to address. A big plus of the system is the fact that it doesn't require any geometric map of the environment, which is often unavailable or hard to obtain for complex outdoor scenarios. The experimental results indicate that the system is more efficient in exploring previously unseen environments and reaching a goal within the environment compared to the baseline methods. I also appreciate the fact that the system was tested on a physical robot, which indicates that the system is effective in real-world scenarios.\n\nThe main problem I have with the paper is that some concepts and ideas are not very well explained which makes the paper hard to follow at times, particularly in Section 4.2: \n\nFirst, the graph (i.e. the \"topological memory\") and what it represents needs to be explained, idealy as early as Section 3 instead of handwavy introducing it in Section 4.2. Additionally, I was initially under the impression that the nodes in the graph represent states and the edges represent their connectivity, but after examining the algorithms, it seems that the nodes represent images and the edges represent distances. This needs to be mentioned.\n\nSecond, in Section 4.2 (i) (Feasible Goal), it is not clear to me how a large probability of the goal embedding z_g^t under the prior implies that the model believes it can reach the goal?\n\nThird, in Section Section 4.2 (ii) the paper mentions that the robot determines if it is at the frontier of the graph by checking it's distance to the least explored node within a distance threshold. How does this work? Line 8 in Algorithm 1 seems to suggest that the mean of the decoder is used to measure the distance between o_t (the current observation) and the observation associated to a node (I suppose this is what o_n denotes in Algorithm 1?). However, I'm just guessing here because this is not explained in the paper. \n\nFourth, how and when exactly is the graph constructed? Appendix B1 mentions that at the end of each trajectory, the subroutine ExpandGraph is called to do this. So I guess this routine should be called in the for loop in Algorithm 3?\n\nAdditionally, the paper should mention the structure of the trajectory dataset that was used to train the model in Section 4.1. Equation 2 reveals that a datapoint consists of a observation-goal observation-action-distance quadruple, but this is nowhere specified in the main text. \n\nUPDATE: \nBased on the author's feedback and the revisions that were carried out, I'm changing my recommendation from \"Weak Accept\" to \"Strong Accept\".", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper proposes a system for for visual-guided goal discovery that seems to work well in practice. However, some revisions are neccessary to improve the clarity of the paper.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "The points above regarding unclear explanations and missing notations need to be addressed. ", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, a robotic learning system for goal-oriented exploration in a novel environment is proposed. A compact representation of goal images is learned and a topological memory is constructed online incrementally during the exploration. All reviewers recognize the field experiments conducted. The concerns are mostly focused on the practicality of the proposed system and some more detailed explanation are lacking. 

In the rebuttal session, the authors have responded to the reviewers' concerns and more details are provided. And the reviewers have reached a consensus of accepting this paper. 
Decision: Accept (Oral)