Last-Mile Embodied Visual Navigation
Authors: Justin Wasserman
Karmesh Yadav
Girish Chowdhary
Abhinav Gupta
Unnat Jain
Venue: CoRL 2022 Poster (2022)
Keywords: Visual Navigation
Embodied AI
Image-Goal Navigation
Perspective-n-Point
AI Habitat
Sim-to-Real

Abstract: Realistic long-horizon tasks like image-goal navigation involve exploratory and exploitative phases. Assigned with an image of the goal, an embodied agent must explore to discover the goal, i.e., search efficiently using learned priors. Once the goal is discovered, the agent must accurately calibrate the last-mile of navigation to the goal. As with any robust system, switches between exploratory goal discovery and exploitative last-mile navigation enable better recovery from errors. Following these intuitive guide rails, we propose SLING to improve the performance of existing image-goal navigation systems. Entirely complementing prior methods, we focus on last-mile navigation and leverage the underlying geometric structure of the problem with neural descriptors. With simple but effective switches, we can easily connect SLING with heuristic, reinforcement learning, and neural modular policies. On a standardized image-goal navigation benchmark (Hahn et al. 2021), we improve performance across policies, scenes, and episode complexity, raising the state-of-the-art from 45% to 55% success rate. Beyond photorealistic simulation, we conduct real-robot experiments in three physical scenes and find these improvements to transfer well to real environments.

Last-Mile Embodied Visual Navigation
Justin Wasserman "1, KKrmmshh Yadav?, Girish Chowdhary"!, Abhinav Gupta', Unnat Jain "University of Illinois Urbana- Champaign, Meta AI Research, Carnegie Mellon University
Abstract: Realistic long-horizon tasks like image-goal navigation involve ex- ploratory and exploitative phases. Assigned with an image of the goal, an embod- ied agent must explore to discover the goal, i.e., search efficiently using learned priors. Once the goal is discovered, the agent must accurately calibrate the last- mile of navigation to the goal. As with any robust system, switches between ploratory goal discovery and exploitative last-mile navigation enable better re- covery from errors. Following these intuitive guide rails, propose SLING to improve the performance of existing image-goal navigation systems. Entirely complementing prior methods, focus on last-mile navigation and leverage the underlying geometric structure of the problem with neural descriptors. With sim- ple but effective switches, we can easily connect SLING with heuristic, reinforce- ment learning, and neural modular policies. On a standardized image-goal naviga- tion benchmark [1], we improve performance across policies, scenes, and episode complexity, raising the statc-of-the-ar from 45% to 55% success rate. Beyond photorealistic simulation, we conduct real-robot experiments in three physical scenes and find these improvements to transfer well to real environments. Code .it.eu..ittt....... wwtpoottoooo//s/11..
and results:
Keywords: Embodied AI, Robot Learning, Visual Navigation, Perspective-n- Point, AI Habitat, Sim-to-Real.
Introduction
Imagine you are at a friend's home and you want to find the couch you have seen in your friend's photo. At first, you use semantic priors i.e. priors about the semantic structure of the world, to navigate to the living room (a likely place for the couch). But as soon as you get the first glimpse of the couch, you implicitly estimate the relative position of the couch, use intuitive geometry, and navigate towards We term the latter problem, of navigating to a visible object or region, as last- mile navigation.
The field of visual navigation has a rich history. Early approaches used hand-designed features with geometry for mapping followed by standard planning algorithms. But such an approach fails to capture the necessary semantic priors that could be learned from data. Therefore, recent years, we have seen more efforts and significant advances in capturing these priors for semantic navigation tasks such as image-gor goal [2, 3, 4, 1, 5, 6] and object-goal navigation [7, 8, 9, 10]. The core idea is to train navigation policy using reinforcement or imitation learning and capture semantics. But an effort to capture the semantic priors, these approaches almost entirely bypass the underlying geometric structure of the problem, specifically when the object or view of interest has already been discovered.
One can argue that last-mile navigation can indeed be learned from data itself. We agree that, in principle, can be. However, we argue and demonstrate that an unstructured local policy for last- mile navigation is either (a) sample inefficient (billions of frames in an RL framework [11]) or (b) biased and generalize poorly when learned from offline demonstrations (due to distributional shift [12, 13]). Therefore, our solution is to revisit the basics! We propose Switchable Last-Mile
"qqual technical contribution; ccorrsspodingg authors
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Discovery
Navigation
Environment (Vartaal Pipsican)
(explare)
(explinit)
Obervaaions
Goal Disoovery
Reached goal
Steuight NINNS,
OVRL, Ornele
Previous works
GoaL discovered
Switch
SLING
Switch
Starting location
Last-Mile
Navigation
(a) Phases Image-Gort Goal Navigation
(b) Policy Overview
Figure I: Switchable Last-Mile Image-Goal Navigation. (a) Long-horizon semantic tasks such as image-goal navigation involves exploratory discovery of goals and exploitative last-mile navigation, (b) An overview of SLING that allows for switching between policies from prior work and our last-mile navigation system.
Image-Goal Navigation (SLING) a simple yet very effective geometric navigation system and associated switches. Our approach can be combined with any off-the-shelf learned policy that uses semantic priors to explore the scene. As soon the object view of interest is detected, the SLING switches to the geometric navigation system. We observe that SLING provides significant performance gains across baselines, simulation datasets, episode difficulty, and real-world scenes.
Our key contributions are: (1) A general-purpose last-mile navigation system and switches, that we connect with five diverse goal discovery methods, leading to improvements across the board. (2) new state-of the-art of 54.8% success i.e. a huge jump of 21.8% vs. published work [5] and 9. 9.2% concurrent preprint [6], on the most widely-tested fold (Gibson-curved) of the AI Habitat image-goal navigation benchmark []]: (3) Extensive robot experiments of image-goal navigation in challenging settings with improved performance over neural, modular policy [1] trained on real-world data [14].
Related Work
Prior work visual navigation and geometric 3D vision is pertinent to SLING.
Embodied navigation. Anderson el al. [15] formalized different goal definitions and metrics for the evaluation of embodied agents. point-goal navigation, relative coordinates of the goal are avail- able (either at all steps [16, 11, 17, 18, 19] or just at the start of an episode [9, 20, 21]). Successful navigation to a point-goal could be done without semantic scene understanding, as seen by compet- itive depth-only agents [16, 11. Semantic navigation entails identifying the goal through an image (image-goal [1, 2, 22]), acoustic cues (audio-gon [23, 24)). category label (object-goal goal [8, 9]). Several extensions of navigation include senguugg-condiiiionn
navigation following [25, 26, 27, 28), social navigation [29, 30, 31, 32, 33], and multi-agent tasks [34, 35, 36, 37, 38, 39]. However, each of these build-off single-agent navigation and benefit from associated advancements. For more em- bodied tasks and paradigms, we refer the reader to a recent survey [40]. this work, we focus on image-goal navigation in visually rich environments.
Image-goal navigation. Chaplot er al. [3] introduced modular and hierarchical method for nav- igating to an image-goal that utilizes a topological map memory. Kwon et al. [41] introduced memory representation based on image similarity, which turn is learned in an unsupervised fash- ion from unlabeled data and the agent's 'ss observed images. Following up on [3], NRNS [1] improves -ooggiicllggahh----e the topological graph-basee based architecture and open-sourced public dataset and IL. and RL base- lines [11, 3] within AI Habitat. This dataset has been adopted for standardized evaluation [5, 6]. ZER [5] focuses on transferring an image-goal navigation policy to other navigation tasks. a con- current preprint, Yadav el al. [6] utilize sulf-super pretraining [42] to improve an end-to-end visual RL policy [11] ]] for the image-goal navigation benchmark. Our contributions are orthogonal to the above and can be easily combined with them, as we demonstrate in Sec. 4.

---

Beyond simulation, SLING finds relevance to the rich literature of navigating to an image-god on physical robots. Meng et al. [4] utilize a neural reachability estimator and local controller based on a Riemannian Motion Policy framework to navigate to image-goals. Hirose et al. [43] train a deep model predictive control policy to follow a trajectory given by sequence of images while being robust to variations in the environment. Even in outdoor settings, meticulous studies have shown great promise, based on negative mining, graph pruning, and waypoint prediction [44] and utilizing geographic hints for kilometer-long navigations [45]. Complementing this body of work, SLING tackles image-goal navigation in challenging indoor settings, without needing any prior data of the test environment (similar to [1, 6, 3]) i.e. during evaluation access to information (trajectories, GPS, or top-down maps) in the test scenes is assumed.
Last-mile navigation. The works included above focus primarily on goal discovery. In contrast, recent works have also identified 'last-mile" errors that occur when the goal is sight of or close to the agent. For multi-object navigation, Wani et al. [46, 47] observed a two-fold improvement when allowing an error budget for the final 'found' or "stop" actions. Chattopadhyay er al. [48] found the last step of navigation to be brittle i.e. small perturbations lead severe failures. Ye et al. [10] identified last-mile errors as a prominent error mode (10% of the failures) in object-goal navigation. However, none of these works address the problem with the last-mile of navigation. From study inspired by [46], we infer that better (or more tolerant to error) last-mile navigation can indeed lead to better performance in the image-goal navigation task (details in Appendix H).
Connections 3D vision. The objective of our last-mile navigation system predict the relative camera pose between two images iee agent's view and image-goal To this end, pose estimation of a calibrated camera from 3D-to-21) point correspondences connects our embodied navigation task to geometric 3D computer vision. The ciospective-nPooin (PnP) formulation, with extensive research and efficient solvers [49, 50, 51], fits this use case perfectly. To find an accurate PnP solution, locating correspondences between the local features of the two images is critical. We utilize SuperGlue [52] which is based on correspondences learned via attention graph neural nets and partial assignments. We defer details of PnP and finding correspondences to Sec. 3.3, to make the approach self-sufficient. Notably, different from related works in 3D vision [53, 54, 55], we apply SLING to sequential decision-make take policies to the real world, we utilized robust SLAM methods [56, 57] for local odometry and pose estimation, which has also been found reliable by prior works in sim-to-real [58, 59, 60).
in embodied settings, particularly, image- goreeggallll navigation. To
3 SLING
In this section, we begin with an overview of the task and the entire pipeline of SLING. We then discuss the implementations for goal discovery, our proposed system for last-mile navigation, and switches to easily combine it with prior works. While we explain key design choices in the main paper, a supplementary description and a list of hyperparameters, for effective reproducibility, is deferred to Appendix A.
3.1 Overview
We follow the image-goal navigation task benchmark by Hahn et al. [1] (similar to the prior formu- lations [2, 3]). The agent observes an RGB image I,, a depth map Da. and the image- mmage-goal Ig. The agent sample actions from A = (move forward, turn right, turn left, stop). The stop action terminates the episode.
As shown in Fig. 1a, we divide image-goal navigation into a goal discovery and a last-mile nav- igation phase. In the goal discovery phase, the agent responsible for discovering the goal i.e. navigating close enough for the goal to occupy a large portion of the egocentric observation ('goal discovered' image). Fig. lb shows how the control flows between our system. If the explore-exploit switch isn't it triggered, learning-basee based exploration will continue. Otherwise, if the explore-exploit switch triggers, the agent's observations now overlap with the image-goal and the control flows to the last-mile navigation system. We find that one-sided flow (as attempted in [1, 3]) from explore exploit is too optimistic. Therefore, we introduce symmetric switches, including one that flows control back to goal discovery.
3

---

Lifting 20.330 (P.)
GNN-baxed Motcher
PerspectivennPPoiit
Feature Extraction
Erzor(e)
Agent lanuge (1.)
X F
X,
moze forward burn night surn left stop
Switch to excplore?
No
Local Policy
Goal Inaae (1))
Camera
Intrinsios [K)
Estimate rotation RJ and trsaslation(t)
Goal Disravery systtam
Depth mask (D,)
Figure 2: Last-Mile Navigation system. Neural keypoint feature descriptors extracted and matched to obtain correspondences between the agent's view and the image-goal. The geometric problem of estimating the relative pose between the agent and goal view is solved using efficient perspective-n-ooit. A exploit-explore switch, if triggered, flows control back to the goal discovery phase. Else, the estimations are fed into a local policy head decide the agent's actions.
3.2 Goal Discovery
We can combine our versatile last-mile navigation system and switching mechanism with any prior method. These prior methods are previously suggested solutions image-goal navigation. We demonstrate this with five diverse goal discovery (GD) implementations.
Straight [61]. A simple, heuristic exploration where the agent moves forward and unblocks itself, stuck, by turning right (similar to an effective exploration baseline in [61]).
Distance Prediction Network (NRNS-GD) [1]. Exploratory navigation is done by proposing way- points navigable areas (determined utilizing the agent's depth mask), history is maintained using a topological map, and processed using graph neural nets. The minimum cost waypoint is chosen utilizing outputs from a distance prediction network. More details are given in Appendix and [1]-
Decentralized Distributed PPO (DDPPO-GD) [11]. An implementation of PPO [62] for pho- torealistic simulators where rendering is the computational bottleneck. This has been standard end-to-end deep RL baseline in prior works, across tasks [18, 1, 5, 6, 63].
Offline Visual Representation Learning (OVRL-GD) [6]. A DDPPO network, with its visual encoder pretrained using self-supervised pretext tasks [42] on images obtained from 3D scans [64].
Environment-State Distance Prediction (Oracle-GD). To quantify the effect of errors coming from the goal discovery phase, we devise an upper bound. This is a privileged variant of NRNS-GI that accesses the ground-truth distances from the environment, exclusively for the goal discovery phase. For fine details of its construction, particularly, how we curtail this to an oracle explorer and not oracle policy, see Appendix B.
3.3 Last-Mile Navigation
The proposed last-mile navigation module transforms the agent's 's observations and image- goal into actions that take the agent closer to the goal. The steps shown in Fig. and detailed next.
Neural Feature Extractor. We first transform the agent's RGB I, to local features (Xa, Fa). where X2 E I are the positions and F, R are the visual descriptors in the agent's image. Here, na is the number of detected local features and k is the length each descriptor. Similarly, I, leads x& to features (&g. Fy), where &g E g",x x2 and F, E x with ng local features in the image- goal. Following DeTone et al. [65], we adopt an interest-point detector, pretrained on synthetic data followed by cross-domaie homography adaptation (here, k 256).
Matching Module. From extracted features (X,, F.) and (Xg. F2), we predict matched subsets X. E x2 and Xg E g=x2 The matching is optimized to have Xa and Xg correspond to the same point. We utilize and occlusions well using an optimal transport formulation, following Sarlin et al. [52]. The above neural feature extractor and GNN- based matcher help enjoy benefits of learning- based methods, par- ticularly, those pretrained on large offline visual data without needing online, end finetuning. The geometric components, relying these neural features, are described next.
attention-based graph neural net (GNN) that tackles partial matches
Lifting Points from 2D-3D. Next, the agent's s local features are lifted to 3D with respect to the agent's s coordinate frame i.e. P. E x This is done by utilizing the camera intrinsic matrix K

---

(particularly, principle point Px: Py and focal lengths fz. fa) and the corresponding depth values for each position in Xa. say da E IR.. The 1thh row of P. is calculated as
Xa (a Pe (1),
X (i() Ps da da((),da(i)
Pa (1,:)
(I)
fz
fg
where (a(i,) 1) and (a(), correspond to the r and y coordinate 1h feature in Xa. respectively. Formally, da(1) == D. (ia (i. 1), Xa,,))))) 2)).
Perspective-n-Point. The objective of the next step iee. Perspective-n- n-Point (PnP) is to find the rotation and translation between the agent and goal camera pose that minimized reprojection enror. Concretely, for given rotation matrix R R3x3 and translation vector E R&, the 3D positions Pa of local features can be reprojected from the coordinate system of the agent to that of the goal camera:
X 1
Pa 1
K
[
Reprojection error e: |X Xgl
where Xg are the reprojected positions. Minimizing the reprojection error e, via ePnP [51] and RANSAČ [49] (to handle outliers), we obtain the predicted rotation and translation. The reprojec- tion is visualized in Fig. 2. where the agent's 's amber point is lifted and reprojected in goal camera coordinates. The reprojection is different from its correspondence in the goal image.
Estimating Distance and Heading to Goal. The predicted translation can help calculate the distance م = |||ll2 from the agent to the goal. Similarly, the heading from the agent to the goal can be obtained from the dot product of the unit vectors along the optical axis (of the agent's view) and Concretely, & sgn(((1]) arceos (t .o. (///|l)/|/loo.|l2)) The sign comes from {[]] which points along the axis perpendicular to the agent's 's optical axis but parallel to the ground. The sign particularly important when calculating the heading as distinguishes between the agent turning right or left.
Local policy. Finally, the distance P and heading between the agent's current position to the estimated goal are utilized to estimate actions in the action space AA to reach the goal. Following accurate implementations [66, IJ, adopt local metric map allow the agent to heuristically avoid obstacles and move towards the goal. For further details, see Appendix A.
3.4 Switches
We define simple but effective switches between the two phases of goal discovery (explore) and last- mile navigation (exploit). The explore-exploit switch is triggered if the number of correspondences n > Au, where nu is a set threshold. This indicates that the agent's image has significant overlap with the image-goal, switch performs better than training a specific deep net to achieve the same (variations attempted in [1, 4)). For exploit-explore, if the optimization for R. (see Eq. (2)) fails if the predicted distance is greater than da (tuned to 4m), the agent returns to the goal discovery phase.
control can flow to the last-mile navigation phase. We find that this simple
Experiments
We report results for image-goal navigation both in photorealistic simulation and real-world scenes.
4.1 Data and Evaluation
We evaluate image-goal navigation policies on the benchmark introduced by Hahn et al. [1] and follow their evaluation protocol and folds. The benchmark consists of numerous folds: (Gibson [67], MP3D [68]] x (straight, curved) x (easy, medium, hard). For a direct comparison to prior work [3, 5, 6] that reports primarily on "Gibson-curved" fold, we follow the same in the main paper. Consistent performance trends are seen in 'Gibson-straight and in the MP3D folds as well. These results are deferred to Appendix C and Appendix F. Performance on image-goal navi- gation is chiefly evaluated via two metrics percentage of successful episodes (success) and success weighted by inverse path length (SPL) [15]. For top-performing baselines, we also include the av- erage distance to the goal at the end of the episode in Appendix G. The objective of the image-goal navigation task is to execute stop within Im of the goal location. The agent is allowed 500 steps.
5

---

Table Results for 'Gibson-curved" episodes Note the significant gains by adding SLING to prior works. Consistent trends are seen in "Gibson-straight (Appendix C) and MP3D-curved episodes (Appendix F).
Overall
Easy
Medium
Hard
Method
Succt
SPL
Sucet
SPL1
Suce
SPL
Sucet
SPL
BC w/ Spatial Memory [69]
1.3
1.1
3.1
2.5
0.8
0.7
0.2
0.1
2
BC w/ GRU [69, 70]
1.7
1.3
3.6
2.8
1.1
0.9
0.5
0.3
DIPPO (from [1])
15.7
12.9
22.2
16.5
20.7
18.5
4.2
3.7
NRNS [1]
21.7
8.1
31.4
10.7
22.0
8.2
11.9
ZER [5]
33.0
23.6
48.0
34.2
36.0
25.9
15.1
10.8
6
OVRL [6]
45.6
28.0
53.6
31.7
47.6
30.2
35.6
21.9
DDPPO-LMN + OVRL-GD
44.3
30.1
52.4
36.6
48.6
32.6
31.9
21.2
SLING Straight-GD
31.0
12.8
39.2
14.3
33.0
14.3
21.0
9.9
SLING DDPPO-GD
37.9
22.8
52.2
32.7
42.2
25.2
19.4
10.5
10
SLING NRNS-GD
43.5
15.1
58.7
17.4
47.0
17.4
25.0
10,5
11
SLING OVRL-GD
54.8
37.3
65.4
45.7
59.5
40.6
39.6
25.5
4.2
Methods
We compare our last-mile navigation with several standardized baselines [69, 11, 1]. Note that field-of-vieys rotation amplitude, etc. vary across baselines and we adopt the respective settings for fair comparison (implementation details of SLING are in Appendix A). Prior methods use mix of sensors including RGB, depth, and agent pose, but no dense displacement vector the goal. While we did include the most relevant baselines in Sec. 3.2, we also compare SLING to several other image-goal solvers. This includes imitation learning baselines such as Behavior Cloning (BC) w/ Spatial Memory [69] and BC w/ Gated Recurrent Unit [69, 70J. We also compare to estab- lished reinforcement learning baselines DDPPO [11] and Offline Visual Representation Learning objective. Finally, compare to related modular baselines include NRNS [1] and Zero Experience Replay (ZER) [5]. We defer detailed discussion of these baselines to Appendix B.
(OVRL) [6]. OVRL also makes use of pretraining using self-
SLING Ablations. For comprehensive empirical study, we combine SLING with Straight-GID, NRNS-GD, DDPPO-GD, OVRL-GD, and Oracle-GD (see Sec. 3.2 for details). We also introduce a neural baseline, DDPPO-LMN, a DDPPO model trained to perform last-mile navigation.
Further, we include clear ablations to show the efficacy of the components of our method and
robustness to realistic pose and depth sensor noise:
w/ MLP switch: instead of SLING's explore-xxxloit switch (that utilizes geometric structure), if
MLP' detects similarity between the agent and goal images (as in [1]).
w/o Recovery: if the exploit-explo
switch is removed í.e. one-sided flow of control.
w/o Neural Features: if the neural features [65] are replaced with traditional features [71].
w/ Pose Noise: add noise to pose that emulates real-world sensors [66, 72] (same as [3, 1J).
w/ Depth Noise: imperfect depth by adopting the Redwood Noisy Depth model [73] in Al Habitat.
access ground-truth distances to move the agent closer to the goal during exploration (see Sec. 3.2 and Appendix B).
w/ Oracle-GD, privileged baseline where NRNS-GD
w/ Oracle-LM-Poen privileged last-mile system with perfect displacement from agent goal
w/ Ora privileged baseline where local policy can teleport agent to the goal prediction
4.3 Quantitative Results
In the following, include takeaways based the results in Tab. and Tab. 2.
State-of-the-art performance. As Tab. details, SLING + OVRL-GD outperforms a suite of IL, RL, and neural modular baselines. The Gibson-curved fold is widely adopted in prior works and hence the focus the main paper. With a 54.8% overall success and 37.3 SPL we are the best- performing method the benchmark, improving success rate by 21.8% ZER and 9.2% OVRL ('overall success' column of rows 5, 6, & 11). In Appendix I, we also demonstrate state-of- the-art performance when panoramic images are used.
'trained over offline dataset of expert demonstrations, where adjacent nodes in topological graph (that they maintain) considered positives

---

Table Ablations on 'Gibson-curved" episodes. Both switches are key to SLING's performance.
SLING is resilient to sensor noise. Similar trends can be observed over ablations performed with OVRL-GD in Appendix F. The privileged last-mile navigation system establishes an upper bound for last-mile navigation. Even with Oracle-GD, performance improves if SLING is added.
Overall
Easy
Medium
Hard
Method
Sucet
SPL
Succt
SPL.1
Succt
SPL
Sucet
SPL
NRNS [1]
21.7
8.1
31.4
10.7
22.0
8.2
11.9
5.4
SLING + NRNS-GD
43.5
15.1
58.7
17.4
47.0
17.4
25.0
10.5
w/ MLP Switch
42.5
14.8
55.4
16.7
47.3
17.3
24.9
10.5
4 w/ MIP Switch w/o Recovery
31.5
11.5
45.6
14.3
32.8
12.9
16.1
7.3
w/o Neural Features
33.7
11.3
47.5
13.5
35.9
13.0
17.7
7.5
w/ Pose Noise
43.7
14.3
58.6
16.1
47.6
16.8
24.9
10.1
7
w/ Pose & Depth Noise
43.5
14.0
56.9
15.9
47.2
15.9
26.6
10.3
Privileged Last-Mile Navigation
w/ Oracle-LP
45.1
17.8
60.8
21.2
48.7
20.3
25.8
12.1
w/ Dracle-LM-Pae M-Pose
53.3
19.3
72.3
23.4
57.1
21.6
30.5
13.1
10
w/ Oracle-LM-Poone & Oracle-LP
53.7
22.4
72.6
27.7
57.6
24.7
31,0
14.9
Privileged Goal Discovery
11 NRNS Oracle-Gi GD (upper bound)
67.7
60.2
68.5
58.4
71.2
63.7
63.5
58.7
12 SLING + Oracle GD (upper bound)
86.2
74.8
85.9
72.2
88.6
77.7
84.3
74.6
SLING works across methods. Using switches, we add our last-mile navigation system to DDPPO [11], NRNS [1], and OVRL [6]. and observe gains across the board. As shown in Tab. 1, SLING improves the success rate of DDPPO by 22.2%, NRNS by 21.8%, and OVRL by 9.2% (rows 3 & 9, 4 & 10, 6 & 11, respectively). Quite surprisingly. SLING even with simple straight exploration, can outperform deep IL, RL, and modular baselines. (rows 1, 3, 4, & 8).
SLING outperforms neural policies for last-mile navigation. SLING surpasses DDPPO trained over 400M steps for last-mile navigation by 10.5% success rate (rows 7 & 11).
SLING succeeds across scene datasets. Similar improvements are also seen in MP3D scenes - adding SLING to OVRL improves success by 5.1%. Further details and results can be found in Appendix F.
SLING is resilient to sensor noise. As shown in rows 6 & 7 of Tab. 2. minor drops in perfor- mance are observed despite challenging noise in pose and depth sensors -PL successively reduces 15.1 +1..3 >14.0% (rows 2 >6 -7).
Geometric switches are better. Performance reduces if we swap out SLING's explore-exploit switch with the MLP switch of NRNS [1]. The effect is exasperated when SLING's exploit-explore switch is also removed, leading a drop of 12% (Tab. 2, rows 2 & 4). The neural features utilized in SLING are useful, as seen by comparing rows 2 and 5. Further, over a set of 6500 image pairs, evaluate the accuracy of switches. SLING's explore-exploi switch is 92.0% accurate and MLP switch [1] is only at 82.1%. Also, SLING exploit-explore switch is 84.1% 1% accurate while NRNS doesn't 't have such a recovery switch (details of this study in Appendix D).
Large potential for last-mile navigation. When Oracle-LM-Pone and Oracle-LP are used there is 10.2% overall improvement in success from 43.5 53.7% (Tab. 2, rows 2 & 10). Notably, in easy episodes, oracle performance is an ambitious upper bound with an increase in success of 13.9% (58.7-722 )58.7%72.6%) For the hard (i.e. longer) episodes, the oracle components have relatively lower impact. This is quite intuitive as goal discovery errors are a more prominent error mode in long-horizon episodes instead of last-mile navigation.
Improvements with Oracle-GD. Even if we assume a perfect variant of goal discovery system from [1], we observe that performance saturates 67.7% success (row 11, Tab. 2). Comparing rows 11 and 12, SLING can boost this asymptotic success rate by 18.5% (67.7 -8.2%%)
Analysis: Why is SLING more robust? In Fig. 3a, we visualize the frequency distribution of heading (from the agent to the target) in expert demonstrations [1] ('train GT') and that observed at inference ("test GT'). With no geometric structure, NRNS picks up the bias in training data, par- ticularly, towards the heading of (optimal trajectories entail mostly moving forward). Concretely, 72.2% of the training data is within 155,, 15%]. This drops to 39.4% at test time when the last-mile
7

---

Function (from KDE)
Train GT
Probability Density
1.0
NRNS
SLING
0.5
-10
-0.5
0.0
0.5
1.0
Heading (agent.target) in radians
(a) Bias in heading distribution
(b) Robot experiments (more visuals in Figures 5, 6, 7)
Figure 3: (a) Significant distribution shift between training and test heading from agent to goal
(Sec. 4.3), (b) Navigation policies deployed on robot in cluttered real-world scenes (Sec. 4.4).
navigation phase is reached (using the best-performing Oracle-GD). Quantified with (first) Wasser- stein distance, W GT, NRNS) 0.0134 vs. W(Test GT, SING)) 0.0034, demonstrating SLING can better match the distribution at inference.
4.4 Physical Robot Experiments
We test the navigation policies on a TerraSentia [74] wheeled robot, equipped with an Intel RealSense" TM D435i depth camera (further hardware details in Appendix E). The robot is initial- ized in an unseen indoor enviromment and provided an RGB image-goal. We ran a total of 120 trajectories, requiring 30 human hours of effort, across three scenes and two levels of difficulty. Following the previously collected simulation dataset [1], easy goals are 1.5-3m from the starting location and hard goals are 5-10m from the starting location. Particularly, we test within an office and the common areas in two department buildings, over easy and hard episodes (following defi- nitions from [1]). The physical setup (office) is shown in Fig. 3b. As in simulation, the agent is successful it executes stop action within Im of the goal. Examples of the image-goal utilized in physical robot experiments and precautions taken are included in Appendix E.
As shown in Tab. 3, for sim-to-real experiments, we base the goal discovery system on the NRNS model. We choose NRNS as the authors published an instantiation trained exclusively on real-world trajectories, particularly, RealEstate 10K [14] (house tours videos from YouTube). In preliminary experiments, we verified that this NRNS instantiation outperformed its simulation counterpart. For direct comparison, in SLING + NRNS-GD, we utilize the same goal discovery system but add our switching and last-mile navigation system (SLING) around it. With SLING, we improve performance from 40.0% success to 56.6%. The gains become more prominent as the task horizon increases, leading an improvement in success rate from 3.3% to 20.0%. The large gains hard episodes (which are exploration heavy) accounted to SLING's better explore-exxpoii switch and SLING's last-mile navigation system that is not biased to zero heading (particularly important for curved and long episodes).
Easy
Hard
Method
Sucet
SPL
Sucet
SPL1
NRNS [1]
40.0
37.7
3.3
3.3
+ SLING
56.6
53.7
20.0
19.3
Table 3: Results in real-world scenes.
Conclusion
In this work, we identify and leverage the geometric structure of last-mile navigation for the chal- lenging image-goal navigation task [1]. With analysis of data distributions, we demonstrate that learning from expert demonstrations may lead to developing a bias. Being entirely complementary to prior work, we demonstrate that adding SL.NG leads to improvements across data splits, episode complexity, and goal discovery policies, establishing the new state-of-the-arn for image-goal navi- gation [1]. We also transfer policies trained in simulation to real-world scenes and demonstrate sig- nificant gains in performance. Further improvements in the switching mechanism, neural keypoint features, visual representations from view augmentations, etc. complement our proposed approach to help improve performance in future work.
Like any method, SLING has several aspects where follow-up works can improve on. We list them explicitly: (1) Our method is limited by mistakes in matching correspondences. (2) We add addi- tional parameters that need to be tuned. (3) We make single prediction for last-mile navigation. (4) We assume access to depth and pose information. More details of these aspect as well as discussion on pose errors, depth noise, and the nuanced image-goal navigation definition in Appendix J.
8

---

Acknowledgments
We thank the reviewers for suggesting additional experiments to make the work stronger. JW and GC are supported by ONR MURI N00014-19-1-2373. We are grateful to Akihiro Higuti and Mateus Valverde for physical robot help, Dhruv Batra for helping broaden the scope, Jae Yong Lee for help with geometric vision formulation, Meera Hahn for assistance in reproducing NRNS results, and Shubham Tulsiani for helping ground the work better to 3D vision. A big thanks who gave feedback to improve the submission draft Homanga Bharadhwaj. Raunaq Bhirangi, Xiaoming Zhao, and Zhenggang Tang.
friends
References
[1] M. Hahn, D. S. Chaplot, S. Tulsiani, M. Mukadam, J. M. Rehg, and A. Gupta. No no simulation: Learning to navigate without navigating. NeurlPS, 2021.
[2] Y. Zhu, R. Mottaghi, Kolve, J. J. Lim, A. Gupta, L. Fei-Fei, and A. Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. ICRA, 2017.
[3] D. S. Chaplot, R. Salakhutdinov, A. Gupta, and S. Gupta. Neural topological slam for visual navigation. CVPR, 2020.
[4] X. Meng, N. Ratliff, Y. Xiang, and D. Fox. Scaling local control large-scale topological navigation. ICRA, 2020.
[5] Z. Al-Halah, S. K. Ramakrishnan, and K. Grauman. Zero experience required: Plug & play modular transfer learning for semantic visual navigation. CVPR, 2022.
[6] K. Yadav, R. Ramrakhya, A. Majumdar, V.P. Berges, S. Kuhar, D. Batra, A. Baevski, and
O. Maksymets. Offline visual representation learning for embodied navigation. arXiv preprint arXiv:2204. 13226, 2022.
[7] D. S. Chaplot, D. P. Gandhi, A. Gupta, and R. R. Salakhutdinov. Object goal navigation using goal-oriented semantic exploration. NeurlPS, 2020.
[8] D. Batra, A. Gokaslan, A. Kembhavi, O. Maksymets, R. Mottaghi, M. Savva, A. Toshev, and
E. Wijmans. Objectnav revisited: On evaluation of embodied agents navigating to objects. arXiv preprint 1311::,006..1171, 2020.
[9] H. Team. Habitat CVPR challenge, 2020. URL 2020/.
[10] J. Ye. D. Batra, A. Das, and E. Wijmans. Auxiliary tasks and exploration enable objectgoal navigation. ICCV, 2021.
[11] E. Wijmans, A. Kadian, A. Morcos, S. Lee, I. Essa, D. Parikh, M. Savva, and D. Batra. Dd-ppo: Learning near near-perfect pointgoal navigators from 2.5 billion frames. ICLR, 2019.
[12] S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. AISTATS, 2011.
[13] S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.
[14] T. Zhou, R. Tucker, J. Flynn, G. Fyffe, and N. Snavely. Stereo magnification: Learning view synthesis using multiplane images. SIGGRAPH, 2018.
[15] P. Anderson, A. Chang, D. S. Chaplot, A. Dosovitskiy, S. Gupta, V. Koltun, J. Kosecka, J. Ma- lik, R. Mottaghi, M. Savva, et On evaluation of embodied navigation agents. arXiv preprint 1rXiv:1807.06757, 2018.
[16] M. Savva, Kadian, O. Maksymets, Y. Zhao, E. Wijmans, B. Jain, J. Straub, Liu, V. Koltun, J. Malik, D. Parikh, and D. Batra. Habitat: A Platform for Embodied AI Research, ICCV, 2019.

---

[17] H. Team. Habitat CVPR challenge, 2019. URL http:///aihabttaa..oogccchhaaaaa 2019/.
[18] J. Ye, D. Batra, E. Wijmans, and A. Das. Auxiliary tasks speed up learning pointgoal naviga- tion. CoRL, 2020.
[19] L. Weihs, U. Jain, I.-J Liu, J. Salvador, S. Lazebnik, A. Kembhavi, and A. Schwing. Bridg- ing the imitation gap by adaptive insubordination. In NeurIPS, 2021. the first two authors contributed equally.
[20] H. Team. Habitat CVPR challenge, 2021. URL http::/aahabbtat.oorgggg 2021/.
[21] S. Datta, O. Maksymets, J. Hoffman, S. Lee, D. Batra, and D. Parikh. Integrating egocentric localization for more realistic point-goal navigation agents. CoRL, 2020.
[22] L. Weihs, J. Salvador, K. Kotar, U. Jain, K.-H. Zeng, R. Mottaghi, and A. Kembhavi. Allenact: framework for embodied research. arXiv preprint arXiv:2008. 12760, 2020.
[23] C. Chen, U. Jain, C. Schissler, S. V. A. Gari, Z. Al-Halah, V. K. Ithapu, P. Robinson, and
K. Grauman. Soundspaces: Audio-visual navigation in 3d environments. ECCV, 2020.
[24] C. Gan, J. Schwartz, S, Alter, D. Mrowca, M. Schrimpf, J. Traer, J. De Freitas, J. Kubilius, A. Bhandwaldar, N. Haber, al. Threedworld: A platform for interactive multi-modal physical simulation. NeurIPS Datasets and Benchmarks Track, 2021.
[25] P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, I. Reid, S. Gould, and A. van den Hengel. Vision-and-language navigation: Interpreting visually-grounder naviga- tion instructions in real environments. In CVPR, 2018.
[26] A. Suhr, C. Yan, Schluger, Yu, H. Khader, M. Mouallem, I. Zhang, and Y. Artzi, Executing instructions in situated collaborative interactions. In EMNLP, 2019.
[27] M. Hahn, Krantz, D. Batra, Parikh, J. M. Rehg, S. Lee, and P. Anderson. Where are you? localization from embodied dialog. In EMNLP, 2020.
[28] S. Raychaudhuri, S. Wani, S. Patel, U. Jain, and A. Chang. Language-aligned waypoint (law) supervision for vision-and-language navigation in continuous environments. In EMNLP, 2021.
[29] S. Srivastava, C. Li. M. Lingelbach, R. MarrtnnMarrtn, F. Xia, K. E. Vainio, Z. Lian, C. Gok- men, S. Buch, K. Liu, et al. Behavior: Benchmark for everyday houschold activitics in virtual, interactive, and ecological environments. In CoRL., 2022.
[30] A. Padmakumar, Thomason, Shrivastava, P. Lange, A. Narayan-Cherl S. Gella, R. ramithu, G. Tur, and D. Hakkani-Tne -ur. TEACh: Task-driven embodied agents that chat. arXiv, 2021.
[31] X. Puig, T. Shu, S. Li, Z. Wang, Y.-H. Liao, J. B. Tenenbaum, S. Fidler, and A. Torralba. Watch-and-help: challenge for social perception and human-(ai) collaboration. In ICLR, 2021.
[32] Xia, W. B. Shen, C. Li, Kasimbeg, M. Tchapmi, A. Toshev, R. Martin-Martín, and S. Savarese. Interactive gibson benchmark: A benchmark for interactive navigation in clut- tered environments. IEEE Robotics and Automation Letters, 2020.
[33] C. Pérez-D' Pérez-D'Arpino, C. Liu, P. Goebel, R. Martin-Martin, and S. Savarese. Robot navigation in constrained pedestrian environments using reinforcement learning. In ICRA, 2021.
[34] U. Jain, L. Weihs, E. Kolve, M. Rastegari, S. Lazebnik, A. Farhadi, A. G. Schwing, and
A. Kembhavi. Two body problem: Collaborative visual task completion. In CVPR, 2019.
[35] U. Jain, I. Weihs, Kolve, A. Farhadi, S. Lazebnik, A. Kembhavi, and A. Schwing. A cordial syne: Going beyond marginal policies for multi-agent embodied tasks. In ECCV, 2020.
10

---

[36] J. Thomason, M. Murray, M. Cakmak, and L. Zettlemoyer. Vision-and ieioo-and--ialos didloo navigation. In CoRL, 2020.
[37] U. Jain, I.J. Liu, S. Lazebnik, A. Kembhavi, L. Weihs, and A. G. Schwing. Gridtopix: Training embodied agents with minimal supervision. In ICCV, 2021.
[38] I.-J Liu, Z. Ren, A. Yeh, and Schwing. Semantic tracklets: An object-centric repre- sentation for visual multi-agent reinforcement learning. In IROS, 2021.
[39] I.JJ. Liu, U. Jain, R. A. Yeh, and A. Schwing. Cooperative exploration for multi-agent deep reinforcement learning. In ICML, 2021.
[40] M. Deitke, D. Batra, Y. Bisk, T. Campari, A. X. Chang, D. Chaplot, C. Chen, C. P. D' Arpino,
K. Ehsani, A. Farhadi, et al. Retrospectives on the embodied ai workshop. arXiv preprint arXiv:2210.06849, 2022.
[41] O. Kwon, N. Kim, Y. Choi, H. Yoo, J. Park, and Oh. Visual graph memory with unsupervised representation for visual navigation. ICCV, 2021.
[42] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. CVPR, 2021.
[43] N. Hirose. F. Xia, R. Martín-Martin, A. Sadeghian, and S. Savarese. Deep visual mpc-policy learning for navigation. RA-L, 2019.
[44] D. Shah, B. Eysenbach, G. Kahn, N. Rhinehart, and S. Levine. ViNG: Learning Open- Navigation with Visual Goals. ICRA, 2020.
Open-World World
[45] D. Shah and Levine. ViKiNG: Vision-Based Kilometer- Navigation with Geographic Hints. RSS, 2022.
[46] S. Wani, S. Patel, U. Jain, A. Chang, and M. Savva. Multion: Benchmarking semantic map memory using multi-object navigation. NeurlPS, 2020.
[47] S. Patel, S. Wani, U. Jain, A. Schwing, S. Lazebnik, M. Savva, and A. Chang. Interpretation of emergent communication in heterogencous collaborative embodied agents. ICCV, 2021.
[48] P. Chattopadhyay, J. Hoffman, R. Mottaghi, and A. Kembhavi. Robustnav: Towards bench- marking robustness in embodied navigation. ICCV, 2021.
[49] M. A. Fischler and R. C. Bolles. Random sample consensus: paradigm for model fitting with applications image analysis and automated cartography. Communications of the ACM, 1981.
[50] C.-P. Lu, G. D. Hager, and E. Mjolsness. Fast and globally convergent pose estimation from video images. TPAMI, 2000.
[51] V. Lepetit, Moreno-Noguer, and P. Fua. Epnр: An accurate o (n) solution to the pup problem. IJCV2009
[52] P.-E. Sarlin, D. DeTone, T. Malisiewicz, and A. Rabinovich. Superglue: Learning feature matching with graph neural networks. CVPR, 2020.
[53] S. Agarwala, L. Jin, C. Rockwell, and D. F. Fouhey. Planeformers: From sparse view planes to 3d reconstruction. ECCV, 2022.
[54] C. Rockwell, J. Johnson, and D. F. Fouhey. The 8-point algorithm as an inductive bias for relative pose prediction by vits. 3DV, 2022.
[55] M. El Banani, L. Gao, and J. Johnson. UnsupervisedR&&R Unsupervised Point Cloud Regis- tration via Differentiable Rendering. CVPR, 2021.
[56] R. Mur-Artal and J. D. Tardós. ORB-SLAM2: an open-source SI.AM system for monocular, stereo and RGB-D cameras. IEEE Transactions Robotics, 2017.
11

---

[57] M. Labbé and F. Michaud. Rtab-map as an open-souree lidar and visual simultaneous local- ization and mapping library for large-scale and long-term online operation. JFR, 2019.
[58] P. Anderson, A. Shrivastava, J. Truong, A. Majumdar, D. Parikh, D. Batra, and S. Lee, Sim- to-real transfer for vision-and-language navigation. CoRL, 2020.
[59] Truong, S. Chernova, and D. Batra. Bi-directional domain adaptation for sim2real transfer of embodied navigation agents. RA-L, 2021.
[60] A. Kadian, J. Truong, A. Gokaslan, A. Clegg, E. Wijmans, S. Lee, M. Savva, S. Chernova, and D. Batra. Sim2real predictivity: Does evaluation in simulation predict real-world performance? RA-L, 2020.
[61] T. Chen, S. Gupta, and A. Gupta. Learning exploration policies for navigation. ICLR, 2019.
[62] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint orXiv:/777.06:: 2017.
[63] A. Khandelwal, L. Weihs, R. Mottaghi, and A. Kembhavi. Simple but effective: Clip embed- dings for embodied ai. CVPR, 2022.
[64] A. Eftekhar, A. Sax, J. Malik, and A. Zamir. Omnidata: A scalable pipeline for making multi- task mid-level vision datasets from 3d scans. ICCV, 2021.
[65] D. DeTone, T. Malisiewicz, and A. Rabinovich. Superpoint: Self-supervised interest point detection and description. Proceedings of the IEEE conference on computer vision and patlern recognition workshops, 2018
[66] D. S. Chaplot, S. Gupta, A. Gupta, and R. Salakhutdinov. Learning to explore using active neural mapping. ICLR, 2020.
[67] Xia, A. R. Zamir, Z. He, A. Sax, J. Malik, and S. Savarese. Gibson env: Real-world percep- tion for embodied agents. CVPR, 2018.
[68] A. Chang, A. Dai, Funkhouser, M. Halber, M. Niessner, M. Savva, Song, A. Zeng, and
Y. Zhang. Matterport3d: Learning from rgb-d data in indoor environments. 3DV, 2017.
[69] M. Bain and C. Sammut, A framework for behavioural cloning. Machine Intelligence, 1995.
[70] K. Cho, A. C. Courville, and Y. Bengio. Describing multimedia content using attention-based encoder-decoder networks. IEEE Transactions on Multimedia, 2015.
[71] D. Lowe. Object recogmition from local scale-invariant features. ICCV, 1999.
[72] A. Murali, T. Chen, K. Alwala, D. Gandhi, L. Pinto, S. Gupta, and A. Gupta. Py- robot: An open-source robotics framework for research and benchmarking. arXiv preprint 1rXiv:1906.088236.22 2019.
[73] S. Choi, Q.-Y. Zhou, and V. Koltun. Robust reconstruction of indoor scenes. CVPR, 2015.
[74] V. A. Higuti, A. E. Velasquez, D. V. Magalhaes, M. Becker, and G. Chowdhary, Under canopy light detection and ranging-basee autonomous navigation. JFR, 2019.
[75] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. CVPR, 2016.
[76] S. Gupta, J. Davidson, S. Levine, R. Sukthankar, and J. Malik. Cognitive Mapping and Plan- ning for Visual Navigation. CVPR, 2017.
[77] S. Gupta, D. Fouhey, S. Levine, and J. Malik. Unifying map and landmark based representa- tions for visual navigation. arXiv preprint arXiv:/712.08/22 2017.
[78] Y. Zhu, D. Gordon, F. Kolve, D. Fox, I.. Fei-Fei, A. Gupta, R. Mottaghi, and Farhadi. Visual Semantic Planning using Deep Successor Representations, ICCV, 2017.
12

---

[79] L. Mezghani, S. Sukhbaatar, T. Lavril, O. Maksymets, D. Batra, P. Bojanowski, and K. Ala- hari.
reinforcement learning for image-goal navigation. arXiv preprint arXiv:2101.00 181, 2021.
[80] M. V. Gasparino, A. N. Sivakumar, Y. Liu, A. E. B. Velasquez, V. A. H. Higuti, J. Rogers, H. Tran, and G. Chowdhary. Wayfast: Traversability predictive navigation for field robots. IROS, 2022.
[81] M. Camplani and L. Salgado. Efficient spatio-temporal hole filling strategy for kinect depth maps. Three-dimensional image processing (3DIP) and applications Ii, 2012.
[82] J. Shen and S.-C. S. Cheung. Layer depth denoising and completion for structured-light rgb-d cameras. CVPR, 2013.
13


Reviews:
[{"review_id": "ld5xwWiD2Vz", "reviewer": "Reviewer_FqFC", "summary": "This paper tackles the problem of last-mile embodied visual navigation, in which the agent has to navigate to the goal after the goal image is visible in the current view. Once the image is visible, the correspondences between the current and the goal image is used to recover the relative rotation and translation of the two images. This, combined with the depth, is used to estimate the heading and distance to the goal, which is then used with a local metric map to avoid obstacles to head to the given goal. The paper also proposes ways to switch between goal discovery and last mile navigation phases, enabling the proposed method to easily be combined with existing goal discovery methods. The method is evaluated thoroughly on Gibson and MP3D datasets with numerous state of the art baselines establishing that the proposed method performs much better than the baselines and gets consistent improvement. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. The paper is well motivated. It picks the task of last mile navigation which empirically causes a non-trivial loss in performance in the proposed end-to-end navigation methods. \n2. The paper is well written, has illustrative figures, and is easy to follow.\n3. The experiments and ablations are quite thoroughly done on both Gibson, MP3D and real-world scenes. The baselines look relevant and include state-of-the-art methods including end-to-end RL, Imitation Learning, and Neural Modular Policies. The improvements convincingly show that the proposed method SLING adds non-trivial gains when combined with the existing goal discovery methods.  \n4. A limitation of the proposed method is that it has several error-prone steps and hyperparameters which make it very complicated to implement and deploy. It would be good to simplify some of the steps by subsuming them in a learned policy, as discussed below in the suggested baseline.\n5. Another limitation of the paper is that the paper compares with no alternative baseline methods for last mile navigation itself. One baseline which would be good to have is to learn a policy to navigate to the goal when initialized with the last mile navigation task, i.e., goal in sight. \n6. Another baseline that would be instructive to have is Neural Topological SLAM [3], which also seems to use the geometric information which should implicitly lead to good last mile navigation performance. \n7. In L273-274, the authors say that the gains become sharper for the hard tasks. For completeness, could the authors describe what is the difference in these tasks compared to the easier benchmark in the real world? If the task just entails more distant goals, then I would assume that this is the job of goal discovery, and not last mile navigation, and hence would require an explanation on why the performance of the proposed method is significantly better in this regime.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper focuses on the problem of last-mile navigation and gives a compelling empirical case to show that the proposed method SLING can lead to consistent improvement across the board when combined with existing goal discovery methods. I'm leaning towards an acceptance because the paper has strong empirical evidence to support the importance of separately solving the last mile navigation task, and then combining it with existing goal discovery methods. ", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "From the comments above:\n\n1. Lacks an alternative baseline method comparison for last mile navigation itself. One baseline which would be good to have is to learn a policy to navigate to the goal when initialized with the last mile navigation task, i.e., goal in sight. \n2. Another baseline that would be instructive to have is Neural Topological SLAM [3], which also seems to use the geometric information which should implicitly lead to good last mile navigation performance. \n3. In L273-274, the authors say that the gains become sharper for the hard tasks. For completeness, could the authors describe what is the difference in these tasks compared to the easier benchmark in the real world? If the task just entails more distant goals, then I would assume that this is the job of goal discovery, and not last mile navigation, and hence would require an explanation on why the performance of the proposed method is significantly better in this regime.\n"}}, {"review_id": "e5tmzzMDbYQ", "reviewer": "Reviewer_D7Jh", "summary": "The paper focuses on the problem of last-mile navigation on image-goal navigation, which aims at reaching the goal after discovering it. It is also the key problem of many visual navigation methods but have barely been researched. The proposed method SLING is proved to raise the SOTA on success rate. Physical experiments are carried to prove the effectiveness on real environment.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1)\tThe problem of last-mile navigation have barely been researched, despite it is proved to be the reason of many failures.\n2)\tThis paper proposes a simple and interpretable module that can be used with many baselines. The experimental results are rich and impressive.\n3)\tPhysical experiments are carried to demonstrate the effectiveness of transferring to real environment.\n\nWeaknesses:\n1)\tThere is no experiment that compares or combine SLING with VGM [1] which is the SOTA ImageNav method.\n2)\tIt would be better to discuss some failure cases caused by adding SLING as a plug-in module, so it is more clear how SLING affects the baselines it is added to.\n\nReferences:\n[1] O. Kwon, N. Kim, Y. Choi, H. Yoo, J. Park, and S. Oh. Visual graph memory with unsupervised representation for visual navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15890–15899, 2021.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper raises an interesting problem for deeper research, and proposes an effective method to tackle the problem with adequate experiments.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Add experiments on VGM.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "2TBmBfYFUzH", "reviewer": "Reviewer_wKSA", "summary": "This paper addresses the challenge of last-mile goal-reaching in visual navigation tasks, i.e. the last stage of navigating to an image-goal in an unknown environment, when the goal is in the line-of-sight of the agent. The authors propose a simple controller that extracts visual keypoints in the agent’s observations and establishes correspondence to the goal image — converting the last-mile image-goal problem into a point-goal problem. Extensive quantitative analysis suggests that this indeed improves performance on benchmark tasks, and the learned policies generalize to real-world indoor environments.\n", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n---\n- The paper is very well-written and the observation of converting the last mile to image-goal navigation tasks to a point-goal navigation task, albeit simple, seems well motivated. I also found the method explanation to be easy to follow.\n- The authors do a good job at statistically significant evaluations (both in sim and real) of the tasks and baselines presented. The experiments compare against SOTA visual navigation techniques and include useful ablations that make it very easy to understand which design decisions were most relevant.\n\n*** \nWeaknesses\n---\n- [Brittle Components] While the proposed method makes complete sense on paper, I am very skeptical of the {feature extraction + correspondence matching + lifting + SFM + geometric estimation} pipeline being broadly usable beyond simulated Habitat tasks and simple indoor settings. The biggest motivation for learning-based approaches to navigation is to reduce the “hard” reliance of the geometric pipelines on factors such as camera intrinsic parameters, local odometry and pose estimation, etc. While re-combining learning and geometry is generally a good idea, the specific mechanism proposed in the paper relies a bit too heavily on the brittle parts of geometric method and it seems unlikely to broadly useful.\n- [Interpreting Results: Upper Bound] I had some trouble interpreting the results at different environment complexities and what the contribution of the proposed method (SLING) really is — having some sort of an oracular upper bound baseline would be really useful here. The improvements in many cases seem absurdly high, and it is not clear why such large improvements are caused by SLING (esp. for real-world experiments); the paper does not discuss this too well. Two baselines I strongly recommend adding, that shed light on different aspects of the system, can be:\n\t-  An oracle baseline for the exploit-only phase would be a useful addition to have (the paper currently contains Oracle-GD, not in exploit case). This could serve as an _upper bound_ to evaluate how good the detection and PointGoal performance really is.\n\t- An evaluation with increased tolerance (for any baseline: NRNS/OVRL) that directly gets at the hypothesis presented to motivate the two-stage process (Line 73).\n\t- Another important metric to include (in Table 1) would be the “distance of trajectory endpoint to goal” for the non-SLING baselines. This would really help understand the failure modes better — are the failures happening mostly due to long-horizon issues in the last mile, sooner, or other reasons.\n- [Understanding Oracle-GD Improvements] I don’t follow the argument about result improvements with Oracle-GD (Line 245). The *NRNS+Oracle-GD* baseline has access to the ground-truth environment state, then why does adding SLING to this baseline lead to such a massive jump in performance? Particularly, the goal-discovery phases of the two methods are identical, and post discovery, SLING only adds information about the estimated goal location but this is already available (noise-free) to the method — why then does the performance change at all? I may be misinterpreting how this baseline works, but this needs to be clarified in the text better.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "In my review, I raised a major concern regarding the significance of the findings (owing to brittle components), and suggestions regarding presentation of experimental results. I cannot recommend acceptance of the paper in its current form, but I am very willing to engage with the authors in the discussion phase and reconsider my recommendation.\n\n---\n**Update:** After the round of discussions with the authors, my concerns have been mostly addressed, as long as the authors actually incorporate these (and other changes/additional expts requested by other reviewers) changes in the final version. I'm on the borderline between a WA/WR and cannot enthusiastically recommend acceptance, but I have no major actionable concerns with the paper after the recent discussion.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": "- [Points Raised Above] I raised some concerns in the above section and some clarification in response to those points would really help me re-evaluate the submission score.\n\t- Adding new baselines for upper bound and a better metric\n\t- Understanding Oracle-GD baseline\n"}}]

Author rebuttal:

Decision: Accept (poster)