End-to-End (Instance)-Image Goal Navigation through Correspondence as an Emergent Phenomenon
Authors: Guillaume Bono
Leonid Antsfeld
Boris Chidlovskii
Philippe Weinzaepfel
Christian Wolf
Venue: ICLR 2024 poster (2024)
Keywords: Navigation
Embodied AI
Perception
Primary area: applications to robotics, autonomy, planning
TL;DR: In an ImageGoal navigation context, we propose two pre-text tasks which let correspondence emerge as a solution and train a dual visual encoder based on a binocular transforme

Abstract: Most recent work in goal oriented visual navigation resorts to large-scale machine learning in simulated environments. The main challenge lies in learning compact representations generalizable to unseen environments and in learning high-capacity perception modules capable of reasoning on high-dimensional input. The latter is particularly difficult when the goal is not given as a category ("ObjectNav") but as an exemplar image ("ImageNav"), as the perception module needs to learn a comparison strategy requiring to solve an underlying visual correspondence problem. This has been shown to be difficult from reward alone or with standard auxiliary tasks. We address this problem through a sequence of two pretext tasks, which serve as a prior for what we argue is one of the main bottleneck in perception, extremely wide-baseline relative pose estimation and visibility prediction in complex scenes. The first pretext task, cross-view completion is a proxy for the underlying visual correspondence problem, while the second task addresses goal detection and finding directly. We propose a new dual encoder with a large-capacity binocular ViT  model and show that correspondence solutions naturally emerge from the training signals. Experiments show significant improvements and SOTA performance on the two benchmarks, ImageNav and the Instance-ImageNav variant, where camera intrinsics and height differ between observation and goal.

Published as a conference paper at ICLR 2024
END-TO-END (INSTANCE) IMAGE GOAL NAVIGATION THROUGH CORRESPONDENCE AS AN EMERGENT PHE- NOMENON
Guillaume Bono, Leonid Antsfeld, Boris Chidlovskii, Philippe Weinzaepfel Christian Wolf Naver Labs Europe Meylan, France [firstname. iastname anaerllaseeee..commmmm
ABSTRACT
Most recent work in goal oriented visual navigation resorts to large-scale ma- chine learning in simulated environments. The main challenge lies in learning compact representations generalizable to unseen environments and in learning high-capacity perception modules capable of reasoning
high-dimensional in- put. The latter is particularly difficult when the goal is not given as a category ("OhjectNav") but exemplar image ("ImageNav"), as the perception module needs to learn a comparison strategy requiring to solve an underlying visual cor- respondence problem. This has been shown with standard auxiliary tasks. We address this problem through sequence of two pretext tasks, which serve as prior for what we argue is one of the main bottle- neck in perception, extremely wide- bade-baseline relative pose estimation and visibility prediction complex scenes. The first pretext task, cross-vier view completion is a proxy for the underlying visual correspondence problem, while the second task addresses goal detection and finding directly. We propose new dual encoder with large-capacity binocular ViT model and show that correspondence solu- tions naturally emerge from the training signals. Experiments show significant improvements and SOTA performance the two benchmarks, ImageNav and the Instance- Mattance-ImagNaav variant, where camera intrinsics and height differ between ob- servation and goal.
difficult from reward alone
INTRODUCTION
Goal oriented visual navigation usually addressed through large-scale training in simulation, followed by sim2real transfer. While decision taking has not yet been solved either, recent research provides evidence that perception is a major bottleneck with several chal- lenges: learning representations required for planning: extracting 3D information, difficult when depth is not available not reliable; and, generalizing to unseen environments, which is challenging given the limited number of existing training environments.
Goal
Observation
Action
The perception module of an agent needs to ad- dress several skills, which include detecting naviga- ble space and obstacles, detecting exits necessary for long horizon planning, detecting goals and estimating the agent's 'ss relative pose with respect to them, see Fig- ure 1. The detection visual goals given by exem- plars requires to solve partial matching task, which in essence a wide-baseline visual correspondence problem. They classical in computer vision and relative pose estimation (Humenberger et al., 2022; Revaud et al., 2019; Sarlin et al., 2020). We argue that in navigation, however, they did not get the attention they deserve.
Visual encoders
Policy
Navigation skills include de- tecting navigable space, exits, and the agent's 's relative pose wrt. the goal. The correspondence solutions required by pose emerge from training with pretext tasks.
Figure 1:
heart of methods in visual localization and

---

Published as a conference paper at ICLR 2024
Instead, robot perception is addressed through scene reconstruction, for instance with SLAM (Chap- lot et al., 2020b; Lluvia et al., 2021; Thrun et al., 2005) or by putting the full burden of perception on visual encoder trained end-to-end from objectives like RL (Jaderberg et al., 2017; Zhu et al., 2017) or imitation learning (Ding et al., 2019). The former does not address goal detection, which needs to be outsourced to external component. The latter, when trained on tasks like ImageNav, attempts to solve the problem implicitly without direct supervision through weak learning signals.
This has been shown to be difficult, witnessed by the wide usage of more complex sensors for the ImageNav task compared to tasks like ObjectNav, where the goal specified through its category. For several years, state-of-the art methods for ImageNav used panoramic images consisting of 4 observed images taken at angles of 90" (see Figure 2), which facilitates learning the underlying pose estimation task from weak learning signals, but is restrictive in terms of robotic applications. Only recently the field switched to mono-view input.
Panoramic
Mono-view
Figure Panoramic vs. mono-view input.
We propose a new method for goal oriented visual navigation, which introduces sequence of pretext tasks of directional learning and visual correspondence. We take advantage of recent advances in unsupervised model pre-training for low-level scene understanding and build the work of Weinzaepfel et al. (2022), which proposes type of multi-view pretext task, namely cross-view completion, a 3D variant of masked image modeling. We show that the underlying correspondence problem solved by this model is particularly relevant to the ImageNav problem and adapt this model through a new dual-encoder model.
In recent work (Krantz al., 2023), the same problem is addressed with explicit feature matching combined with a modular map-based method. In contrast, our method does not rely on explicit cor- respondence calculations, the entire agent differentiable and the perception module is comprised of a combination of a monocular and a binocular encoder. We do, however, show that correspon- dence solutions emerge from pre-training through the cross-attention behavior, see Figure 4 in the experimental section.
We present the following contributions: (i) We define a pretext task of extremely wide-baseling relative pose estimation highly correlated with navigation and positioning cues from visual input. We also introduce a new dataset tailored for navigation; (ii) We couple relative pose estimation with the estimation of visibility, which we link to the specific capacity to decide whether to explore or to exploit; (iii) We additionally perform self-supervised pre-training for cross-view completion ("CroCo") (Weinzaepfel et al., 2022) and show its impact; (iv) We show that correspondence solu- tions emerge from pre-training with these tasks. (v) We propose a dual visual-encoder architecture based vision transformers and cross- attenttt,n,, which integrate into end-to-end agent and, as proof-of conceet,, into a modular architecture; (vi) We obtain SOTA performance on two standard benchmarks, ImageNav and Instance- Imageeav.
RELATED WORK
Visual navigation navigation has been classically solved in robotics using mapping and plan- ning (Burgard et al., 1998; Macenski et al., 2020; Marder-Eppstein et al., 2010), which requires solutions for mapping and localization (Bresson et al., 2017; Labbé & Michaud, 2019; Thrun et al., 2005), for planning (Konolige, 2000; Sethian, 1996) and for low-level control (Fox et al., 1997: Rösmann et al., 2015). These methods depend on accurate sensor models, filtering, dynam- ical models and optimization. End-to-end trained models directly map input to actions and are typically trained with RL (Jaderberg et al., 2017; Mirowski et al., 2017: Zhu al., 2017; Bono et al., 2024) or imitation learning (Ding et al., 2019). They learn representations, either flat re- current states or occupancy maps (Chaplot et al., 2020b), semantic maps (Chaplot et al., 2020a), latent metric maps (Beeching et al., 2020b; Henriques & Vedaldi, 2018; Parisotto & Salakhutdi- nov, 2018); topological maps (Beeching et al., 2020a; Chaplot et al., 2020c; Shah & Levine, 2022), self-attention (Chen et al., 2022a; Du et al., 2021; Fang et al., 2019; Reed et al., 2022) or implicit representations (Marza et al., 2023). Our method is end-to-end trained but adds pretext tasks for perception.
2

---

Published as a conference paper at ICLR 2024
Goal-oriented navigation - In the easier ObjectNav setting, the goal is provided as category and a detector can encode object shapes in model parameters, trained explicitly for detection, e.g. with semantic maps (Chaplot et al., 2020a), map-less object detectors (Savva et al., 2019) or image seg- menters (Maksymets et al., 2021), or end-to-end through the navigation loss. ImageNav provides the goal as exemplar image and is significantly harder task, requiring the perception model to learn a matching strategy itself. Most work are based on end-to-end training (Zhu et al., 2017; Mezghani al., 2022; Majumdar al., 2023), potentially supported through self-supervised losses (Majumdar et al., 2022b). Modular approaches have also been proposed (Das et al., 2018a; Wu et al., 2022). The very recently introduced Instance- -inuunee- task requires to handle different camera intrinsics and heights between observation and goal, which prior work does with explicit feature matching (Krantz et al., 2023). We make it possible to address the ImageNav and Maaaannee-magNaav with end-to-end methods in the challenging mono-view setting through new pretext tasks.
Pretext tasks in CV and navigation- - -widely used in NLP and CV, pretext tasks aim at learning representations followed by fine-tuning for particular tasks (Devlin et al., 2019; Yuan et al., 2021). In navigation or robotics, known forms are depth prediction (Das et al., 20122018800 18;;;; Mirowski et al,, 2017), contrastive self-supervised learning (SSL) (Majumdar et al., 2022b) or privileged informa- tion from the simulator like object categories (Pashevich et al., 2021), goal directions (Marza et al,, 2022), exploration (Ye al., 2021) or visual correspondences in visuomotor policy (Florence al., 2020; Sax et al., 2019; Hong et al., 2023). Supervised learning requires in-domain data collection that makes extension beyond the training environment and task difficult. Alternatives come from SSL (Wang et al., 2022; Xie et al., 2022). Recent pre-trained visual encoders, like DINO (Caron et al., 2021) and masked autoencoders (MAE) (He al., 2022). have been used in (Yadav et al., 2022) and (Yadav et al., 2023), respectively. Once pre-trained, the encoder is often frozen before passing into a policy learning module. To be effective across a range of real-world robotic tasks, Ra- dosavovic et al. (2022) diversified the image sources when pre-training. Mezghani et al. (2022) favor nearby frames to have similar visual representations.
Pose Relative Pose Estimation (RPE) has been intensively studied in CV (Kendall et al., 2015; Kim & Ko, 2022; Xu et al., 2022). It evolved from feature matching and correspondences (Mur- Artal et al., 2015; Tang et al., 2023) to end-to-end training (Li et al., 2021; Mousavian et al., 2017) or transfer from large-scale classification (Melekhov et al., 2017) and finetuning after pre-training on geometric tasks (Weinzaepfel et al., 2022). Existing solutions revise various components of the regression pipeline (Jin et al., 2021), discretize the distribution over poses (Chen et al., 2021), etc.
Conventional RPE was developed for rather small camera displacements and assumes high visual overlap between images. Wide-baseline RPE refers to a challenging scenario of large view-point changes and occlusions, on which not all conventional methods work well (Jin et al., 2021). Even more challenging, navigation requires what we call Extremely wide-baseline RPE and Visibility, as an agent may be located in different or cluttered places and therefore may have small or no visual overlap at all. It also requires detecting overlap/ / visibility, which we perform in this work.
3 LEARNING PERCEPTION FOR GOAL ORIENTED VISUAL NAVIGATION
We target image-goal navigation in 3D environments (ImageNav and mustance-Imageli agent is asked to navigate from a starting location to a visual goal (Figure 1). The agent receives at each timestep a single image observation x, <x and a goal image x xB= xxxxxXW both of size 112x112. The agent can select one action from the action set A =MMOV FORMARD O. 25m TURN LEFT 10" TURN RIGHT 10", and STOP). Navigation is considered successful if the STOP action is selected when the agent is within Im of the goal position in terms of geodesic distance. For Instance- Naauun,ve-mmgeVa, two additional actions are LOOK UP and LOOK
where an
DOWN.
Our objective is to learn a perception module which predicts a latent representation given an obser- vation and goal. We conjecture that this requires the following three perception skills:
S1 - Low-level geometric perception of the 3D structure of the scene, which includes the de- tection of navigable space, obstacles, walls and exits, key elements for planning.
$2 - Perception of semantic categories is not only required in tasks where object categories are given as goals, which is not the case in the more challenging tasks we target, but is also an additional powerful intermediate cue for other required skills, like geometric perception. Detecting navigable space, for instance, is highly correlated with categories like Floor, Wall.
3

---

Published as a conference paper at ICLR 2024
Pretext task 1: Cross-view Completion
Pretext task 2:
Downstream task:
Relative pose estimation-visibllity
Visual navigation
8.
bw Rw8
Obs
7
Goal
Adapters
h a h
VIT
VIT
Pretext, then frozen
D
D
adapted with RI
Pre-training (CroCo)
Finetuning (RPEV)
Frozen adapted
From scratch with RL
(RL, ImageGoal)
Figure 3: We address correspondence problems in image goal navigation, which we address through two pretext tasks: ① cross-view completion (Weinzaepfel et al., 2022), which reconstructs masked image from a reference image, and @2 relative pose and visibility estimation. They are learned by binocular VIT b and combined with a monocular encoder taking only observations, forming "DEBiT" Outputs are provided to a recurrent policy, maintaining memory h, and predicting actions ar. Encoder m and policy are trained with RL @ the high-capacity model is frozen but adapted.
S3 - Specific object detection and relative pose estimation under large viewpoint changes ("ex- tremely wide baseline") is more difficult than the detection of known object classes and re- quires to solve a visual correspondence problem, potentially helped by semantic cues.
In end-to-end approaches, these skills have been traditionally learned directly from reward with II., potentially supported by additional tasks like monocular depth prediction (Das et al., 20;8;;;; Mirowski et al., 2017), contrastive self-supervised learning (Majumdar et al., 2022b) or privileged information from the simulator. We argue for a more holistic approach and propose dual visual encoder combined with a multi-step pre-training strategy. Dubbed "DEBiT" Dual Encoder Binoc- ular Transformer, it consists of a binocular model ((x,x"), which targets skill S3, goal detection and goal pose estimation, and a monocular model mxx)). which targets skills S1 and S2 not related to the goal x see Figure 1. The two encoders produce embeddings ef and e!", respectively, which are integrated into a recurrent policy,
ep g(b(xx,xx))
1/ skil1 53 goal direction
e m(xx)
// skilla 31, 52
(1)
f(h ei (((=-1)) p(a.) r(h,),
h,
// recuzrent state updaze (agent men))
// policy
where is a fully connected layer, / is an embedding function, and f is the update function of a GRU (Cho et al., 2014); for clarity we have omitted the equations of gating functions. The monocular encoder ma (xx) takes over the navigation skills not related to the goal, we therefore kept it reasonably small and propose a half-width ResNet-18 architecture (He et al., 2016), which trained from scratch and from reward.
The binocular visual encoder b(x, x*) decomposes into Siamese monocular encoder & applied to each image individually and a binocular decoder D combining both encoder & outputs, expressed as
e
g(b(xx,, x")) ((D(E(x)),
(2)
DEBiT implemented ViT with self-attention layers in both & and D and with cross-attention layers in the decoder DD. 'D can naturally represent the correspondence problems between image patches through the attention distribution, as we will experimentally show Section For more details we refer to (Weinzaepfel et al., 2022) and to Appendix A.
Training the large-capacity binocular encoder entirely from scratch through reward in navigation is difficult. The underlying geometric correspondence problem is complex and can't be handled by the weak learning signal, in particular since the navigation policy needs to jointly learn multiple perception skills, plus some form of internal mapping as well as planning. Training perception separately through losses highly correlated to the perception skills we identified above, in particular Figure 3: pre-train the binocular model b with the CroCo pretext task requiring reasoning on low-level geometry (Section 3.1) and then finetune on novel pretext task dedicated to ImageNav (Section 3.2).
S3, proved to be key design choice
4

---

Published as a conference paper at ICLR 2024
3.1 CROSS- COMPLETION
Recently, Weinzaepfel et al. (2022) introduced Cross-Vier Completion (CroCo), potent pre- training task trained from a large amount of heterogeneous data and which captures the ability to perceive low-level geometric cues highly relevant to vision downstream tasks. It is extension of masked image modeling (He et al., 2022) processing pairs of images (x, x'), which correspond to two different views of the same scene with important overlap. The images split into sets of non-overlapping patches p = Ii...NN and p' {p/]i=.....N respectively. The first input image is partially masked, and the set of non-masked patches is denoted p. The pretext task requires the reconstruction of the masked content p.p from the visible content in the second image, therefore during pre-tratie gre-training we replace the final FC layer g by a patch- wise reconstruction layer denoted r.
p r(((, p')) = ((D((()).E())))
Training minimizes the MSE loss:
1 Σ Ip. 10--P pill?.
L (xxx')
(4)
Ip PiEp'P
CroCo is applicable to monocular and binocular downstream problems, competitive performance was shown for monocular depth estimation, optical flow and RPE (Weinzaepfel et al., 2022).
Model and Data We use the publicly available' code from Weinzaepfel et al. (2022). We re- trained the model ourselves and also explored smaller, more robot-friendly variants. Pre-training data consists of 1.8 million image pairs rendered with the Habitat simulator.
3.2 RELATIVE POSE AND VISIBILITY ESTIMATION IN NAVIGATION
Once the binocular encoder b is pre-trained with CroCo, we finetune it on second pretext task, relative pose estimation and visibility (RPEV) for navigation settings. While for navigation purposes only 2D vector t is relevant, which encodes the direction and distance from the agent to the goal, we train the prediction of the full classical relative pose estimation (RPE) problem, which also includes a 3 x 3 matrix R representing the relative rotation of the camera capturing the goal image w.r.t. the current agent orientation. While not useful for navigation, it can potentially add useful learning signals.
Visibility - Classically, accurate pose estimation assumes that two images (observation and target) share a sufficiently large part of the visual content, with the overlap providing cues sufficient to estimate the translation and rotation components from one image to the other. This assumption was also satisfied in (Weinzaepfel et al., 2022), but it is, by far, not valid assumption in navigation. The agent is initially placed far from the goal location and is required to explore the scene, in which case the RPE task cannot be solved through geometry and correspondence, as no scene points are shared between the two images. Recent work has shown that regularities in scene layouts can be exploited to predict distributions over unseen object positions with some success (Ramakrishnan al., 2022), but this has been reported for object categories and it is unsure whether similar results can be achieved for image exemplars.
For this reason, we added visibility measure to training data, which addresses two issues: (i) it ensures feasibility of RPE and excludes image pairs with insufficient correspondence from training the translation and rotation pose components. We do, however, train the pose components even for low amounts of overlap and treat low visibility as an extreme case ("extremely wide haseline")) (ii provides an additional feature to the agent, as visibility is a strong prior in both positive and negative cases. High visibility indicates closeness to the goal, which can be exploited directly through directional information provided by the same model, captured in the embedding of. Low visibility suggests to explore the scene and rather move away from the current position.
Compared to alternatives like frustrum overlap (Baintas et al., 2018), we define visibility vE(0, as the proportion of patches P of the goal image x' which are visible in the observed image x. Note, that this definition is not symmetric, and exchanging the two images alters the visibility value.
The RPEV model we predict the two RPE components, translation tER3 and rotation matrix 3x33 as well as visibility v, from an additional head h (which is actually composed of three
httpa: http:///hhiiii.ccm///eeeerrrre/t
5

---

Published as a conference paper at ICLR 2024
individual heads) attached to the binocular encoder, taking the embedding ea input. (5)
(t, R. v) = h(c²) ) = i(b(x,, x x)),
where x is the observed image and x" is the goal image. To ensure that R is a valid rotation matrix, we use orthogonal Procrustes normalization from the Roma library (Brégier, 2021).
After CroCo pre-training, we finetune the model with the following loss:
CRPEV Σ, [,[ of 1,;>,[1t, t RR R
where: i indexes image pairs, t,, Rī, oo denote ground truth values, 1. is the binary indicator func- tion, |-1 denotes the La loss and 7 is threshold which switches off RPE supervision in the case of insufficient visibility.
Dataset- We collect dataset tailored perception in ImageNav by sampling random views from scenes in the Gibson (Xia et al., 2018), MP3D (Chang et al., 2018) and HM3D (Ramakrishnan et al., 2021) datasets. We respect the standard train/val scenes split of each dataset. We sample two points uniformly on the navigable area and query the simulator for the shortest path from one to the other. To balance the difficulty of the dataset, we split this path into parts corresponding to increasing thresholds on geodesic distance ("in reach" S 1m, "very close" I 1. 1.5m, "close" < 2m, "approaching" S 4m, and "far" > 4m), sample 10 intermediate positions and orientations along the path in each part, from which images are captured. We compute the fractions of pixels from the goal image visible from any of the ones captured along the path using depth frames which are then discarded. This process is repeated until 100 trajectories per scene are sampled, yielding total of near 68. 8M image pairs with position, orientation and visibility labels, representing 140GB of data.
Training for navigation We train the parameters of the recurrent policy (f, πT) and the monocular encoder m. jointly from scratch with PPO (Schulman et al., 2017) with a reward definition in the lines of the one proposed by Chattopadhyay et al. (2021) for PointGoal, rr K- success A Geo -X, where 1=10, A Geo is the increase in geodesic distance to the goal, and slack cost A=0.01 encourages efficiency. The binocular encoder trained in two different variants:
Frozen we freeze the parameters of the binocular encoder after the two step pre-training phases (CroCo + RPEV), and then only finetune the FC layer g in equation (1). Faster to train, we will use this configuration for most ablations and analyses in the experimental section. Adapted - we freeze as above, but add adapter layers as in AdaptFormers (Chen et al., 2022b), which are trained with RI. jointly with the policy (J, r) and m. In the next section we will show that this leads to significant performance improvements.
4 EXPERIMENTAL RESULTS
Experimental setup We evaluate on both ImageNav, where the goal is a random view taken by the camera of the agent, and the more recent Instance-ImugeNav (Krantz et al., 2022), where the goal depicts specific object viewed from a different camera. The major parts of the experiments, ablations and analyses are performed on ImageNav in the classical setting, as (Majumdar et al., 2022b; Mezghani et al., 2022). Unless stated otherwise, an A100 GPU, As in prior work, for ImageNav we report performance on the 14 Gibson-val scenes and thus use it test set, using the unseen episodes provided by Mezghani et al. (2022). For Instance-ImageNav we follow the protocol in (Krantz et al., 2023).
trained the models for 200M steps
Metrics RPE is evaluated over the pairs with visibility over in the percentage of correct poses for given thresholds on distance and angle, e.g. I meter and 10 Visibility is evaluated over all pairs by its accuracy at +-0.05, i.e., the percentage of prediction within a 0.05 margin of the ground-truth value. Navigation performance is evaluated by success rate (SR), i.c.. fraction of episodes terminated within a distance of <Im the goal by the agent calling the STOP action, and SPL (Anderson al., 2018), i.e., SR weighted by the optimality of the path, SPL = マ Σ S ( where S; be a binary success indicator in episode i, L, is the agent path length and E; the GT path length.
Baselines - we compare with the state-of-ne oattee-f-thh-aee art methods on this task, including several variants of Siamese Encoders. which encode the x, and x" separately, used by trained policy, typically current one. First introduced by Zhu et al. (2017), they were updated by including augmented mem- ory (Mezghani et al., 2022) and powerful VIT based architectures and self- self-supervised pre-training

---

Published as a conference paper at ICLR 2024
Table 1: Image-Nav: impact of model capacity of the binocular encoder on RPEV and nav. perf. (CroCo+RPEV, 200M steps of RL, frozen, no adapters). L-layers, = d=embedd.di
Variant
Eneader
Decoder
#params
Monoc
correct poses
Vis-aee
Nav, perf.
H
L.
H
(binoc)
Im&10
2m&20
(%)
(%) SPL
DEBiT-L ("Largs"), no adapters 12 12 768
8 16 512 120M
hwRN18
97.5
98.9
94.0
82.0
59.6
DEBiT-B ("Base"), no adapters 12 6 384
8
16 512
55M
IwRNIS
92.5
96.8
89.3
83,0
55.6
DEBiT-S ("Small"), no adapters
384
2
256
24M
hwRNI8
82.7
93.5
81.6
79.6
52.1
DEBiT-T ("Tiny)) "), no adapters
6 384
2 8 256 17M
hwRNI8
80.3
92.4
80.6
79.3
50.0
Table 2: ImageNav: impact of pre-training strategies: we ablate CroCo and RPEV pre-training. All results on 100M steps of RI. only, frozen, no adapters. Left: training performance curves (SR) for DEBiT-B, best viewed in color. Right: ablated test results.
Variant
Pre-train % corr. poses Vis-acc Nav. perf.
611.
CRoco-FREy
CroCo RPEV Im& 10
2
(%) SR
SPL
DEBiT-L, no adapters
X
X
n/a
n/a
n/a
7.0 4.4
GROODONLY
DEBIT-L, no adapters /
X
n/a
n/a
n/a
60,2
33.1
DEBIT-L, adapters
X
V
40.1
66.7
58.3 11.8 9.9
DEBiT-L, adapters
/
V
97.5
98.9
94.0 82.0 54.8
DEBiT-B, B, no adapters
X
X
n/a
n/a
n/a 6,8 4.0
DEBIT B. no adapters /
X
n/a
n/a
n/a 65.7 37.3
0.5
DEBIT-B. B. adapters X
39.7
66.4
58.8 23.6 17.4
DEBIT-B, adapters
V
92.5
96.8
89.3 81.2 53.0
(Majumdar et al., 2022b; 2023; Yadav et al., 2023). We also compare to the feature-matching based method presented in (Krantz et al., 2023), which holds the current SOTA Instance-ImageNaa
Impact of model capacity - we explore variations in model capacity distributed over the encoder & and the decoder 'D of the binocular visual encoder b (the monocular part m is unchanged) and introduce four different model sizes in Table 1: DEBiT-L ("Large"), DEBiT-B ("Base"), DEBiT-S ("Small") and DEBiT-T ("Tiny"), where DEBiT-L corresponds the architecture in (Weinzaepfel et al., 2022). Performance generally improves with more model capacity.
Impact of pre-training strategies -onees---o Table 2 gives results comparing different pre-training strate- gies for the two largest variants, DEBiT-L and DEBIT-B. Directly training the binocular encoder b from scratch did not lead to exploitable results, reward as a learning signal is too weak. CroCo pre- training is essential, directly training on RPEV led to low performance. CroCo pre-training alone is not optimal, RPEV adds significant boost to the gain provided by self-supervised objective alone. The curves in Table 2 (left) shows the evolution of navigation performance (SR) during training, indicating the significant gain and head start the two pretext tasks provide.
Aligning architecture design choices with learning signals - visual encoders for end-to-end trained solutions in the literature for ImageNav are typically based on Siamese networks, where the inputs x, and x" encoded separately, the respective embeddings are passed to current policies. This late fusion approach allows to train the models from weak reward signals, as the individual encoders learn high-level representations which are compared later in the pipeline. We claim that image comparisons of higher quality can be obtained through early fusion, where images are com- pared close to input on patch-level. We argue that this leads to a finer visual perception, where correspondence information is encoded in the representation in a more direct way, and provides a more useful signal the policy. Our experiments shown in Table 3 corroborate this claim: we compare with a widely used Siamese architecture based on half-width ResNet- 18 visual encoders taken from (Zhu al., 2017) and reused in (Mezghani et al., 2022). DEBiT outperforms them when pre-trained with both pretext tasks, as CroCo pre-training allows correspondence on patch level to emerge (see further below), which leads to accurate pose estimates. Training DEBiT from reward alone is difficult. On the other hand, adding RPEV pre-training to the Siamese architecture is not helpful, the architecture based on late embedding- fusion cannot exploit this signal.
In additional experiment we verified whether this difference is explained by the presence of cross-attention layer easing the computation of correspondences. We designed a hybrid architecture, dubbed (c) in Table 3, which combines convolutional Siamese encoders, implemented as shared heResNet18, with a Tiny cross-attentios (CA) module with 2 layers, 4 heads and 256 dimensions. Performance is lukewarm, it did not manage capture the cues provided by the pretext tasks.
7

---

Published as a conference paper at ICLR 2024
Table 3: ImageNav: aligning architecture design choices with learning signals: when both are trained from scratch on navigation reward alone, the Siamese visual encoder (Mezghani et al., 2022; Zhu et al., 2017) performs better than our DEBiT architecture. However, DEBÍT shines with self- supervised pre-training and fine-tuning, and learning signals which enable learning the correspon- dence problem solved by the encoder-decoder structure of the binocular stream. RPEV pre-trained models have been added monocular encoder. Frozen, adapters.
Visual encoder
Pre-train
#parms
SR
SPL
(a) Siamese hwRN18
No
4.1M
10.1
9.6
(b) Siamese hwRN18 +m
RPEV
8.3M
8.0
7.7
(a) Stamese
(b)
Siamese
(C) 1w.RNi8+Croos-AUL
No
10M
7.4
4.7
(C) hwRN I C+-rros-Aut-rn
RPEV
10M
7.4
7.2
(d) DEBiT-B (Ours), adapters
No
60M
6.8
4.0
SA
SA
SA
(d) DEBiT-B (Ours), no adapters CroCo+RPEV
60M
83.0
55.6
SA
Cat
Baseline in (Mezghani et ol, 2022), inspired by (Zin et ol, 2017)
(c) TNN-TInyT (d) DEBIT (Ours)
Table 4: ImageNav:
comparisons with prior work: gain +12p in SR, +12p in SPL by using
RL-trained adapters of the DEBiT encoder. ANS models+ are from (Chaplot et al., 2020b).
Method
#steps
SPL(%)
Pretrained weights
Siam. hwRN18
180M
10.1
9.6
None, from scratch
Siam. hwRN18
500M
8.0'
None, from scratch
Mem. Aug. (Mezghani et al., 2022)
500M
9.0
Finetuned
ZSEL (Al-Halah et al., 2022)
500M 29.2
21.6
Obs.&policy frozen, goal from scratch
7SON (Majumdar al., 2022a)
500M 36.9
28.0
Obs. finetuned, goal frozen (CIJI)
VC1-ViT-L. (Majumdar ct al., 2023)
500M 81.6²
Finetuned
OVRL (Yadav et al., 2022)
500M 54.2"
27.0'
Finetuned
OVRL-v2 (Yadav et al., 2023)
500M
82.0 58.7
Finetuned
ANS (Chaplot et al,, 2020b) DEBiT-L
32.0
15.0
Modular architecture our frozen encoder
Ours (DEBiT-B),
adapters
200M
83.0
55.6
Frozen
Ours (DEBiT-L), no adapters
200M
82.0
59.6
Frozen
Ours (DEBiT-L) + adapters
200M
94.0
71.7
Frozen + adapted
Perf. from orig, papers; MMonovvew ablation of baseline in Table of (Mezghani et al., 2022):
3 Retrained in mono-view settings, see Table of (Al-Halah et al., 2022)
ImageNav, comparison with prior work Table 4 compares the proposed model with prior work. DEBiT largely outperforms the competing methods, including the memory augmented model (Mezghani et al., 2022), but also models on large-capacity ViTs like the "Visual Cortex" model VC1 (Majumdar et al., 2023) and OVRL2 (Yadav et al., 2023). Both have been pre-traines with masked image encoding, but in monocular Trame-by-frame basis and perform late fusion of observation and goal features, which we argue does not ease learning geometric comparisons.
Adapters adding adapters to DEBIT gains additional 12p of success rate and 12p of SPI., can be seen in Table 4. For ImageNav, it is unlikely that this is explained by improvement of the pose estimation performance through RL finetuning, We conjecture, that the adapters allow pass richer information through the embedding et from the DEBiT to the policy.
The Mistance-ImageNaa task lak--nn In Ta- the-art in the Instance-ImageNav task, where the goal can be taken with ar- bitrary camera intrinsics (in particular FOV) and from any camera height, not necessarily the height it is installed on the agent. We trained the agent for to- tal of 200M steps, 100M of which were done one the ImageNav task followed by 100M on Instance- Instance-ImageNaas As CroCo and RPEV pre-training have been done in ImageNav set- tings (equal intrinsics), adapting DEBiT to this OOD situation was key design choice, and with-
Table 5: Instance- Instance-ImageNayy ImageNav: adapters enable speci- fying goal images with different camera intrinsics and heights compared to the obs. Performance reported on val. max/avg over the last 5 checkpoints.
ble 5 we compare with the state-of-
Method
#steps SR (%)
SPL (%)
max
avg
max avg
(Krantz et al., 2022)
3500M 5.5
n/a
2.3
n/a
(Krantz et al., 2023)
n/a 56.1
n/a
23.3
n/a
Ouss(EEBTTL)
200M
61.1
59.3
33.5
32.4
8

---

Published as a conference paper at ICLR 2024
Figure 4: Emergence of correspondence from pre-training: visualization of the decoder cross- attention of the finetuned DEBiT-L model for example image pairs. We show attention of the last layer averaged over heads. Top: different scenes and poses. Bottom: image pairs taken from single trajectory, varying distance to the goal, showcasing robustness to scale changes.
Figure 5: RPEV performance, DEBiT-L during expert episodes: pink star = goal; segments point the predicted goal, color encodes visibility prediction and GT v": true negatives (TN) (o'<T,U<> TP (e' >T, (2T) (U"<T,u>T), FN (v">T, - rare, not seen. Pose and visibility (and thresholds!) are used for visualization only, the policy receives a latent embedding.
out adapters performance was actually unexploitable. Non-Siamese adapters (different for obs and goal) gained around 1p of SR compared to Siamese ones. We outperform the current SOTA method (Krantz et al., 2023) and show that this task can also be addressed without feature matching.
proof of concept, we integrate DEBÍT into the modular exploration method ANS (Chaplot ct al., 2020b), which is composed of a high-level policy predicting waypoints, and low-level navigation policy. We adapted it to ImageNav by adding the encoder b as perception module switching between (1) nav. towards the predicted goal with the local policy or (2) exploration, otherwise, see Appendix A.5. Table 4 gives non-comparable number, as we did not retrain ANS and took publicly available parameters. The lower performance is also explained by the direct usage of pose and visibility estimates, whereas the e2e trained models benefit from the richer latent embeddings from the visual encoders. This is confirmed by an ablation study where the e2e trained agent only receives pose and visibility, reaching only 20% SR in training.
Integration into modular architecture
Visualization of attention - in Figure 4 we visualize averaged attention of the last cross-attention layer of a DEBiT-L model. Correspondence solutions naturally emerge without explicit supervision of correspondence solutions. We show a variety of different pairs and poses in the top row, and single trajectory varying goal distances in the bottom row, indicating robustness to scale changes.
Visualization of RPEV performance Figure 5 illustrates pose and visibility estimation perfor- mance on several expert trajectories- - DEBiT reliably detects the goal and provides orientations toward it. Let's recall that this information is passed to the policy indirectly through latent embed- dings, the RPEV head is discarded after pre-training.
5 CONCLUSION
We have introduced pretext tasks and dual visual encoder for ImageNav and Instunce-ImageNNar navigation, which provide rich geometric information and we show that this makes solutions of correspondence problems emerge without explicit supervision. We integrate the method into an end-to-end trained agent, which outperforms competing methods and obtains SOTA performance on both benchmarks. We also showease the integration into modular navigation pipeline. Future work will use the encoder for visual odometry, extend pre-traine rre-tra to pairs with different camera intrinsics and/or different backgrounds, and integrate the method into real robotics platform.

---

Published as a conference paper at ICLR 2024
Reproducibility for the sake of reproducibility, in the of acceptance, will provide the source code for training and evaluation based on the public downloading final trained model weights (CroCo RPEV PPO). For training, will provide in- structions for setting up the codebase, including installing external dependencies, pre-trained models and pre-selected hyperparameter configuration. For the evaluation, the code will include evaluation metrics directly comparable to the paper's 's results.
codebase and links for
Ethics statement we welcome the potentially high interest for society in having autonomous agents capable of various tasks like guiding customers in shopping centers, museums, hospitals and offices, or delivering parcels.
Our introduced perception modules and pretext tasks are new step into this general direction, as they allow to improve the navigation capabilities of autonomous agents. Of course, like most scientific work in the STEM sector, this might lead to some negative societal impacts, which our work shares with most robotics applications: military robots (used in wrong hands), surveillance etc. Our direct work itself was carried out in simulation and as such is unlikely to have produced unethical results, except the impact of large-scale training on CO2 output.
REFERENCES
Ziad Al-Halah, Santhosh Kumar Ramakrishnan, and Kristen Grauman. Zero experience required: Plug & play modular transfer learning for semantic visual navigation. In CVPR, 2022.
Peter Anderson, Angel X. Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta,
Vladlen Koltun, Jana Kosecka, Jitendra Malik. Roozbeh Mottaghi. Manolis Savva, and Amir Roshan Zamir. On evaluation of embodied navigation agents, arXiv preprint, 2018. 6
Vassileios Baintas, Shuda Li, and Victor Prisacariu. RelocNet: Continuous Metric Learning Relo-
calisation Using Neural Nets. In ECCV, 2018. 5
E. Beeching, J. Dibangoye, O. Simonin, and C. Wolf. Learning reason uncertain topological maps. In ECCV, 2020a. 2
Edward Beeching, Jilles Dibangoye, Olivier Simonin, and Christian Wolf. Egomap: Projective mapping and structured egocentric memory for deep RL. In ECML-PKDD, 2020b. 2
G. Bono, H. Poirier, L. Antsfeld, G. Monaci, B. Chidlovskii, and C. Wolf. Learning navigate efficiently and precisely in real environments. In CVPR, 2024. 2
Romain Brégier. Deep regression on manifolds: 3D rotation case study. In Intern. Conf. 3D Vision (3DV), 2021.
Guillaume Bresson, Zayed Alsayed, Li Yu, and Sébastien Glaser. Simultaneous localization and mapping: A survey of current trends in autonomous driving. IEEE Transactions on Intelligent Vehicles, 2017. 2
Wolfram Burgard, Armin B Cremers, Dieter Fox, Dirk Hähnel, Gerhard Lakemeyer, Dirk Schulz, Walter Steiner, and Sebastian Thrun. The interactive museum tour-guide robot. In Aaai/imai, pp- 11-18, 1998.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021. 3
Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niebner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. In 3DV, 2018. 6
Devendra Singh Chaplot, Dhiraj Gandhi, Abhinav Gupta, and Ruslan Salakhutdinov. Object goal navigation using goal-oriented semantic exploration. In NeurlPS, 2020a. 2,
Devendra Singh Chaplot, Dhiraj Gandhi, Saurabh Gupta, Abhinav Gupta, and Ruslan Salakhutdinov. Learning explore using active neural slam. In ICLR, 2020b. 2, 8, 16
10

---

Published as a conference paper at ICLR 2024
Devendra Singh Chaplot, Ruslan Salakhutdinov, Abhinav Gupta, and Saurabh Gupta. Neural topo- logical slam for visual navigation. In CVPR, 2020c.
Prithvijit Chattopadhyay, Judy Hoffman, Roozbeh Mottagii, and Aniruddha Kembhavi. Robustnav:
Towards benchmarking robustness in embodied navigation. CoRR, 2106.04531, 2021.
Kefan Chen, Noah Snavely, and Ameesh Makadia. Wide-baseline relative camera pose estimation with directional learning. In CVPR, 2021. 3
Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Languwe Navigation. In CVPR, 2022a. 2
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang. and Ping Luo. AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition. In NeurIPS, 2022b.
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Conference on Empirical Methods in Natural Language Processing, 2014.
A. Das, G. Gkioxari, S. Lee, D. Parikh, and D. Batra. Neural modular control for embodied question answering. In CORI,, 2018a. 3,4
Abhishek Das, Samyak Datta, Georgia Gkioxari. Stefan Lee, Devi Parikh, and Dhruv Batra. Em- bodied Question Answering. In CVPR, 2018b. 3,4
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, pp. 4171-4186, 2019. 3
Yiming Ding, Carlos Florensa, Pieter Abbeel, and Mariano Phielipp. Goal-conditioned imitation learning. NeurlPS, 2019.
Heming Du, Xin Yu, and Liang Zheng. VTNet: Visual Transformer Network for Object Goal Navigation. In ICLR, 2021. 2
Kuan Fang, Alexander Toshev, Li Fei-Fei, and Silvio Savarese. Scene memory transformer for embodied agents in long-horizon tasks. In CVPR, 2019. 2
Peter R. Florence, I Lucas Manuelli, and Russ Tedrake. Self-supervised correspondence in visuomo- tor policy learning. IEEE Robotics Autom. Lett. 5(2):492-449, 2020. 3
Dieter Fox, Wolfram Burgard, and Sebastian Thrun. The dynamic window approach to collision
avoidance. IEEE Roboties & Automation ( 1997.
Habitat. Facebook Al Research, 2023. https:///aaaab:ie
org/ http:/hhhalllggegeeeeeeee 22033/ 17
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016. 4, 15
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022. 3,
João F. Henriques and Andrea Vedaldi. Mapnet: An allocentric spatial memory for mapping envi- ronments. In CVPR, 2018. 2
Yicong Hong, Yang Zhou, Ruiyi Zhang, Franck Dernoncourtl, Trung Buil, Stephen Gould, and Hao Tan. Contrastive Learning between Egocentric Views and Semantic Maps Learning Navigational Visual Representations with Semantic Map Supervision. In ICCV, 2023. 3
Martin Humenberger, Yohamn Cabon, Noé Pion, Philippe Weinzaepfel, Donghwan Lee, Nicolas Guérin, Torsten Sattler, and Gabriela Csurka. Investigating the role image retrieval for visual localization. International Journal of Computer Vision, 2022. 1

---

Published as a conference paper at ICLR 2024
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czamecki, Tom Schaul, Joel Z. Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In ICLR, 2017. 2
Yuhe Jin, Dmytro Mishkin, Anastasiia Mishchuk, Jiri Matas, Pascal Fua, Kwang Moo Yi, and Ed- uard Trulls. Image matching across wide baselines: From paper to practice. Int. J. Comput. Vis., 129(2):517-547, 2021. 3
Alex Kendall, Matthew Grimes, and Roberto Cipolla. PoseNet: a Convolutional Network for Real- Time 6-DOF Camera Relocalization. In ICCV, 2015.
Daewoon Kim and Kwanghee Ko. Camera localization with siamese neural networks using iterative relative pose estimation. J. Comput. Des. Eng., 944):1822-197, 2022.
Kurt Konolige. A gradient method for realtime robot control. In IROS, 2000.
Jacob Krantz, Stefan Lee, Jitendra Malik, Dhruv Batra, and Devendra Singh Chaplot. Instance- Specific Image Goal Navigation: Training Embodied Agents to Find Object Instances. In 2211.15876, 2022. 6,8
Jacob Krantz, Theophile Gervet, Karmesh Yadav, Austin Wang, Chris Paxton, Roozbeh Mottaghi, Dhruv Batra, Jitendra Malik, Stefan Lee, and Devendra Singh Chaplot. Navigating to objects specified by images. In ICCV, 2023. 2, 8,9
Mathieu Labbé and François Michaud. RTAB-Map as an open-source lidar and visual simultaneous localization and mapping library for large-scale and long-term online operation. Journal of Field Robotics, 36(2):416-445 2019.
Ke Li, Shijie Wang, Xiang Zhang, Yifan Xu, Weijian Xu, and Zhuowen Tu. Pose recognition with cascade transformers. In CVPR, 2021.
Iker Lluvia, Elena Lazkano, and Ander Ansuategi. Active Mapping and Robot Exploration: A Survey. Sensors, 27():244,, 2021. 2
Steve Macenski, Francisco Martín, Ruffin White, and Jonatan Ginés Clavero. The marathon 2: A navigation system. In IROS, 2020. 2
Arjun Majumdar, Gunjan Aggarwal, Bhavika Devnani, Judy Hoffman, and Dhruv Batra. ZSON:
zero-shot object-goal navigation using multimodal goal embeddings. In NeurIPS, 2022a. 8
Arjun Majumdar, Gunnar A. Sigurdsson, Robinson Piramuthu, Jesse Thomason, Dhruv Batra, and Gaurav S. Sukhatme. SSL enables learning from sparse rewards in image-goal navigation. In ICML, 2022b. 3, 4, 6, 7,
Arjun Majumdar, Karmesh Yadav, Sergio Arnaud, Yecheng Jason Ma, Claire Chen, Sneha Silwal, Aryan Jain, Vincent-Piers Berges, Pieter Abbeel, Jitendra Malik, Dhruv Batra, Yixin Lin, Olek- sandr Maksymets, Aravind Rajeswaran, and Franziska Meier. Where are we in the search for an artificial visual cortex for embodied intelligence? In arXiv:2303. 18240, 2023.
Oleksandr Maksymets, Vincent Cartillier, Aaron Gokaslan, Erik Wijmans, Wojeiech Galuba, Stefan Lee, and Dhruv Batra. THDA: treasure hunt data augmentation for semantic navigation. In ICCV, 2021. 3
Eitan Marder- Eppstein, Eric Berger, Tully Foote, Brian Gerkey, and Kurt Konolige. The office marathon: Robust navigation in indoor office environment. In ICRA, 2010. 2
Pierre Marza, Lactitia Matignon, Olivier Simonin, and Christian Wolf. Teaching agents how to map: Spatial reasoning for multi-object navigation. In IROS, 2022.
Pierre Marza, Laetitia Matignon, Olivier Simonin, and Christian Wolf. Multi-Object Navigation
with dynamically learned neural implicit representations. ICCV, 2023. 2
laroslav Melekhov, Juha Ylioinas, Juho Kannala, and Esa Rahtu. Relative camera pose estimation using convolutional neural networks. In AC/VS, 2017. 3
12

---

Published as a conference paper at ICLR 2024
L. Mezghani, S. Sukhbaatar, T. Lavril, O. Maksymets, D. Batra, P. Bojanowski, and K. Alahari. reinforcement learning for image-goal navigation. IROS, 2022. 3, 6, 7, 8,
15
Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andy Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, Dharshan Kumaran, and Raia Hadsell. Learning to navigate in complex environments. In ICLR, 2017. 2, 3,4 4
Arsalan Mousavian, Dragomir Anguelov, John Flynn, and Jana Kosecka. 3d bounding box estima- tion using deep learning and geometry. In CVPR, pp. 5632-5640, 2017. 3
Raul Mur-Artal, J. M. M. Montiel, and Juan D. Tardós. ORB-SLAM: A versatile and accurate monocular SLAM system. JEEE Trans. Robotics, 31(5):1147-1163, 2015.
Emilio Parisotto and Ruslan Salakhutdinov. Neural map: Structured memory for deep reinforcement learning. ICLR, 2018.
A. Pashevich, C. Schmid, and C. Sun. Episodic transformer for vision-and-language navigation. In ICCV, 2021. 3
Ilija Radosavovic, Tete Xiao, Stephen James, Pieter Abbeel, Jitendra Malik, and Trevor Darrell. Real-world robot learning with masked visual pre-training. In Conference on Robot Learning (CoRL), volume 205, pp. 416-426, 2022.
Santhosh Kumar Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alexan- der Clegg, John M Turner, Eric Undersander, Wojeiech Galuba, Andrew Westbury, Angel X Chang, Manolis Savva, Yili Zhao, and Dhruv Batra. Habitat-matterpon 3d dataset (HM3d): 1000 large-scale 3d environments for embodied AI. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021. 6
Santhosh Kumar Ramakrishnan, Devendra Singh Chaplot, Ziad Al-Halah, Jitendra Malik, and Kris- ten Grauman. PONI: Potential Functions for ObjectGoal Navigation with Interaction-fres Learn- ing. In CVPR, 2022. 16
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eecles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A Generalist Agent. TMLR, 2022. 2
Jerome Revaud, Cesar De Souza, Martin Humenberger, and Philippe Weinzaepfel. R2D2: Reliable and Repeatable Detector and Descriptor. NeurlPS, 2019.
Christoph Rösmann, Frank Hoffmann, and Torsten Bertram. Timed-elastic-andds for time-optimal point-to-point nonlinear model predictive control. In European Control Conference (ECC), 2015.
Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewiez, and Andrew Rabinovich. SuperGlue: Learning feature matching with graph neural networks. In CVPR, 2020.
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun. Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A platform for embodied ai research. In ICCV, 2019. 3
Alexander Sax, Bradley Emi, Amir R. Zamir, Leonidas Guibas, Silvio Savarese, and Jitendra Ma- lik. Mid-level visual representations improve generalization and sample efficiency for learning visuomotor policies. In 2019. 3
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint, 2017. 6
James A Sethian. fast marching level set method for monotonically advancing fronts. PNAS, 93
141-1591- 1996. 2
13

---

Published as a conference paper at ICLR 2024
Dhruv Shah and Sergey Levine. graphic Hints. In RSS, 2022.
ViKiNG: Saion-Based Sclaee Navigation with Geo-
Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emergent correspondence from image diffusion. In arXiv:2306.33881, 2023.
Sebastian Thrun, Wolfram Burgard, Dieter Fox, et al. Probabilistic robotics, vol. I. MIT Press Cambridge, 2005. 2
Yanwei Wang, Ching- Yun Ko, and Pulkit Agrawal. Visual pre-training for navigation: What learn from noise? In NeurIPS Workshop, 2022.
P. Weinzaepfel, V. Leroy, T. Lucas, R. Brégier, Y. Cabon, V. Arora, L. Antsfeld, B. Chidlovskii, G. Csurka, and J. Revaud. CroCo: Self-Supervised Pretraining for 3D Vision Tasks by Cross- View Completion. In NeurIPS, 2022. 2, 3, 7,
Erik Wijmans, Abhishek Kadian, Ari Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra. Dd-ppo: Learning near- pear-perfect pointgoal navigators from 2.5 billion frames. In ICLR, 2019. 16
Qiaoyun Wu, Jun Wang, Jing Liang, Xiaoxi Gong, and Dinesh Manocha. Image- saage-goalunn navigation in complex environments via modular learning. IEEE Robotics Autom. Lett., ((3):6902-6900) 2022.
Fei Xia, Amir R Zamir, Zhiyang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: Real-world perception for embodied agents. In CVPR, 2018. 6, 16
Zhihui Xie, Zichuan Lin, Junyou Li, Shuai Li, and Deheng Ye. Pretraining in deep reinforcement learning: A survey, In ArXiv:2211.03955 2022.
Meng Xu, Youchen Wang. Bin Xu, Jun Zhang. Jian Ren, Stefan Poslad, and Pengfei Xu. critical analysis of image-based camera pose estimation techniques. In arXiv:220..05816, 2022. 3
Karmesh Yadav, Ram Ramrakhya, Arjun Majumdar, Vincent-Pierre Berges, Sachit Kuhar, Dhruv Batra, Alexei Baevski, and Oleksandr Maksymets. Offline visual representation learning for em- bodied navigation. In arXiv:2204.133. 132266, 2022. 3,
Karmesh Yadav, Arjun Majumdar, Ram Ramrakhya, Naoki Yokoyama, Alexei Baevski, Zsolt Kira, Oleksandr Maksymets, and Dhruv Batra. OVRL-V2: A simple state-of-art baseline for ImageNav and ObjectNav. In arXiv:2303.07799, 2023. 3,7,8
Joel Ye, Dhruv Batra, Abhishek Das, and Erik Wijmans. Auxiliary tasks and exploration enable objectgoal navigation. In ICCV, pp. 16097-16106, 30
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai. Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, Ce Liu, Mengchen Liu, Zicheng Liu, Yumao Lu, Yu Shi, Lijuan Wang, Jianfeng Wang, Bin Xiao, Zhen Xiao, Jianwei Yang, Michael Zeng, Lu- owei Zhou, and Pengchuan Zhang. Florence: A new foundation model for computer vision. In 1rXiv:2111.11432. 2021. 3
Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J. Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. Target-driven visual navigation in indoor seenes using deep reinforcement learning. In ICRA, 2017. 2, 3, 6, 7,8
14

---

Published as a conference paper at ICLR 2024
A
APPENDIX
All networks have been implemented in PyTorch, below we provide details the binocular encoder b, the monocular encoder m, and the recurrent poliey composed of a dynamics mapping f and the prediction head n.
A.1 THE BINOCULAR ENCODER
DEBiT's binocular encoder b follows the architecture in (Weinzaepfel et al., 2022), and the "Large" version DEBiT-L is equivalent to (Weinzaepfel et al., 2022), corresponding to the code available https: / github. com/ naver crocot the encoder & is ViT-Base model, i.e., composed of L = 12 self-attention blocks with H = 12 heads each and the embedding dimension = 768. The decoder D is composed of L 8 cross-attention blocks with
16 heads each and an embedding dimension d = 512. We will not detail the encoder part further, as it is very similar to standard (monocular) ViT. Concerning the cross-attention blocks used the decoder part, they are composed of the following layers:
A self-attention layer is applied to the features corresponding to the current view (poten- tially already enriched by features from the goal view in previous blocks), with pre-norm (LayerNor)) and skip-connection.
The resulting features are used as queries on the goal view features (used as keys and values, optionally also (reenommalized) in cross-attention layer, with skip-connection.
2-layers perceptron with dimensions projects the features of each patch, with pre-norm, GELU activation and kiip-connection.
2048 and back to 512 independently
Full details on these cross-attention blocks are available in (Weinzaepfel et al., 2022). Smaller DEBiT versions differ in the number of layers, heads and the embedding sizes, see Table 1.
A.2 THE PROJECTION y
The decoder blocks are followed by a single patch-wise linear layer d, which can also be seen as a 1 x 1 convolution the features of all patches in 2D. It projects them from dimension 512 to 64, before flattening them to a 3136-dim vector (given images of size 112 x 112 and patches of size 16 16).
A.3 AUXILIARY HEADS h FOR RPEV
For the second phase of pre-training, the result of the flattened projection of size 3136 is ReLU- activated and fed common 1024-dim linear layer, before being dispatched independent output layers for the predictions of relative camera translation t and rotation R, as well as goal visibility from current view v:
The translation head directly outputs 3D vector in the coordinate frame of the current view.
The rotation head outputs 9D vector which is reshaped 3 matrix, constrained to be a valid rotation matrix (using orthogonal Procrustes normalization) with a small regu- larization term added the loss.
The goal visibility head linearly outputs a single value with no activation constraining it to between and
A.4 THE MONOCULAR ENCODER m
The monocular encoder is a half-width ResNet-18 frequently used in prior work on visual naviga- tion (Mezghani et al., 2022; Majumdar et al., 2022b). It is very similar to a standard ResNet-18 (He al., 2016), only differing in 3 ways:
1. Instead using 64, 128, 256 and 512 channels in the 4 layers (of basic blocks each), the half-width ResNet-18 uses 32, 64, 128, and 256 channels.
15

---

Published as a conference paper at ICLR 2024
2. All BatchNorm2D layers replaced GroupNorm layers with 16 groups each.
3. The final (global pooling linear layer) is replaced by small "Compression" module which consist in: a 3x3 convolution (with padding) reducing the number of channels from 256 to 128, followed by LayezNorm and a ReLU activation, whose result is flattened and fed to a linear layer to produce a 512-dim flat embedding of the current (monocular) view.
A.5 THE RECURRENT POLICY (f.T)
The policy relies on a single-layer GRU as our recurrent state encoder. The 3 flat features vectors produced by the binocular, monocular, and previous action encoders concatenated and fed the GRU, whose output h4 is passed to 2 linear heads that respectively generates a softmax distribution over the action space (Actor head), and evaluation of current state (Critic head). This agent is structurally similar to the agent from (Ramakrishnan et al., 2022; Wijmans et al., 2019), which is used for the related woint-goal navigation task, and achieves 100% PointGoal success on the Gibson dataset (Xia et al., 20, but with different input modalities (RGB + ImageGoal instead of RGB-D PointGoal).
A.6 THE ANS ADAPTATION TO ImageGoal
Figure shows our adaptation of Active Neural SLAM (Chaplot et al., 2020b) to the ImageGoal task. The new components shown in orange. They include the target image, the binocular encoder as additional perception module, and a module switching between (1) navigation towards the predicted goal with the local policy and (2) exploration using the global+local policy, otherwise. Switching is done by thresholding the visibility prediction with a threshold T,, whose influence we tested in sensibility study below.
V
1
Reading
Estimate (1)
Global Policy
(AgE)
Neural SLAM USLAN)
Map (N])
Oesseuation.. (s)
fpia
Local Policy (st,
Figure 6: Active Neural SLAM + DEBÍT-L/0- - we use the binocular encoder b only of the DEBiT architecture. Figure reproduced from (Chaplot et al., 2020b), with additional parts from our adaptation to ImageGoal drawn in orange.
A.7 COMPUTATIONAL COMPLEXITY
Below we report computational complexity of the training pipeline in terms of processed frames per second (fps) single NVIDIA A100 GPU. This includes forward and backward passes well as the overhead of running simulation parallel Habitat environments. Note, that on a real robot only forward pass over the visual encoder and policy is required.
In terms complexity the correspondences themselves, it is the complexity of attention, which is quadratic in terms of tokens (which are patches) and linear in the embedding dimension, number of heads and number of layers. We have input images size 112 112 and patch size of 16 x 16 which gives 7 49 patches per image.
16

---

Published as a conference paper at ICLR 2024
Model
FP'
DEBiT-L
92
DEBiT-B
156
DEBIT-S
192
DEBIT-T
225
Table 6: Time complexity for different DEBÍT models.
A.8 THE INSTANCE- INSTANCE-IMAGENAN BENCHMARK
Dostance-Imagelle task has been trained and tested on the Habita--Matteport33]] (HM3D) Seman- ties v0.2 dataset that was provided for the Habitat Navigation Challenge (Habitat, 2023). The dataset contains 216 different scenes; the train/val/test split 145/36/35, Our model was trained on the 145 training scenes, we reported results for the 36 validation scenes. The remaining 35 test scenes are used by organizers internally; they are not publicly available.
A.9 VISUALIZATION OF CORRESPONDENCES
The visualization in Figure 4 is simple: have examined the last cross-attention layers, with attention summed over all heads, and looked at individual attention values. We have then picked the highest N attention values in this matrix and displayed their corresponding query-key pairs, i.e., drawing line between the cooreipon patches.
A. THE HWRNI 18 --ROOSS-ARSTTTS ATT ARCHITECTURE
The architecture of the binocular encoder for variant "hwRNI8+Cross-Ae used in Table 3 of the main paper is given as follows:
RGB
hwRN18
RGB spa feats
RGB flat
self-Artn
256
Rhaad:
shared weights
X-Attn
Feed Fio
Binoc, Jeats 16 256
Reeak
512
Iohe
Transformer Layer KfPayan
IngGrel 3. 112 112
Goal spu feuts 256
Goal faat feals 256
ImRNN18
In the sake of clarity, normalization and skip-connections in the transformer decoder are not repre- sented on the Figure. Contrary standard ViTs, patch size is 1 (because ResNet already reduced the input image size), no positional embeddings are added to the tokens, and no class tokens are used.
A.11 ABLATIONS AND SENSITIVITY ANALYSIS OF IN RPEV
Controlling Our RPEV model is trained to predict the visibility well the relative pose only on image pairs with visibility above = 0.2. We now propose ablations on the RPEV model where we remove the visibility predictions, as well as where we vary 7 to 0.1 and 0.4. Results are reported in Table 7. We report the visibility accuracy (predicted visibility in the range of 0.05 from the ground-truth
well the percentages correctly predicted poses within two error thresholds (1m& and 2m&209) computed over pairs with different visibility overlap r'. Overall, the per- formance close for all models. When model trained for a larger т than used for evaluation (T > T'), the performance is a bit worse given that the visibility might be too low compared to what have been seen during training, explaining the 2 or 3% lower values for the model trained with 0.4.
Summary: 7 controls the threshold during training, 7' controls the threshold in evaluation.
Removing visibility altogether - If pure RPE model were trained, i.e. RPEV without visibility, the model could be pretrained only on image pairs with overlap. ie. visibility > 0. Indeed, would be impossible for model to predict relative pose without any relevant information when the visibility is zero. On the other hand, when extracting features with this model for navigation, many input pairs would have overlap, i.e., visibility, setup that would not have been seen during
17

---

Published as a conference paper at ICLR 2024
the pre-training. To avoid this out-of-distribution setting, we believe it is important to also pretrain on pairs without overlap during RPEV, explaining why we include all experiments.
To verify that predicting visibility does not harm the RPE performance, have trained relative pose estimation model that is trained only on pairs with visibility above the threshold т, but without any visibility prediction supervision. These results provided in Table 7, first line, the impact negligible.
Table 7: Ablations on RPEV. We measure the impact of having the visibility and of varying 7, the threshold the visibility to decide whether to a given pair training and report the visibility accuracy as well as the percentages at correct poses at two thresholds, for different visibility thresholds 7' of minimum visibility between pairs. The impact of these variants is overall limited.
Visibility Head
% correct poses
Vis-ace
= 0.2
0.3
0.4 =
(%)
Im& 10"
2m&20'
1m&10" 2md220'
1m&
2m&220'
RPE (r 0.2)
X
92.4
96.8
93.5
97.4
93.5
97.5
RPEV (r = 0.2)
89.3
92.4
96.8
93.6
97.4
93.7
97.6
RPEV (T -0.1)
V
90.0
92.5
96.9
93.3
97.3
92.9
97.4
RPEV 0.4)
/
87.7
89.6
95.6
92.7
97.1
93.5
97.5
A.12 LIMITATIONS
Higher image resolutions than 112x112 could boost the goal recognition. On mnstance-Imagelle pre-training could sample image pairs with different camera intrinsics, as this is currently only learned through RL.
A.13 VIDEO
This appendix is accompanied by an additional video showing and explaining rollouts of the agents.
18


Reviews:
[{"review_id": "J2jCiz0znJ", "reviewer": "Reviewer_Q8jf", "summary": "The authors introduced pretext tasks and a dual visual encoder for ImageNav and Instance-ImageNav navigation in 3D environments, which provide rich geometric information and make it possible to address the challenging mono-view setting with end-to-end trained methods. Via experiments it is shown that peformance on competing methods and SOTA on both benchmarks is better.  The idea of cross-view completion and goal direction computation as pre-text for ImageGoal in contrast to ObjectGoal is the philosophical novelty in the work. Apart from that, the pipeline and architecture presented will aid future scope of research in the benchmark challenges.", "questions": "\"we split this path into 5 parts cor\" - any logic regarding the discreet steps evenly spaced?\nInstead of Active Neural SLAM, anything else has been tried out in the pipeline?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Fig. 3 is well described to give the view of the problem scope being breaked into cross view completion, relative pose estimation and visual navigation.\nThis is a well written paper with clear explainations and technical soundess.\nThe results sections and ablation studies uphold the claims.\nThis paper will help the ImageGoal community - hence recommended.", "weaknesses": "The last portion of supplementary video in terms of correspondence needs better representation and also that is the core focus.\nThe analyis of time complexity for doing correspondances in the benchmark and the distribution of work load should have helped understand the bottlenecks for a near real time system like robotic agents, even in embodied setups.\nA practical deployment in robotic setup should have confirmed the real world transfer applicability.\nI think Fig. 1 image you ahve search to related the chair with big picture - can any other image or better reoslution be used?\nSame with panaromic and mono view - please help in making sense of the image if at all included in body. If space contsraint, appendix referral is there for later sections, but introduction setup has to be clear.\nI think related work can only focus on the core related work, getting rid of object goal and general visual nav in such detail - this space can be used elsewhere in explaination later.\nNo limitations of the work is presented. Future gaps should be explained well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PZ4FmsOEmE", "reviewer": "Reviewer_Sgek", "summary": "This paper is concerned with image-goal navigation and proposes a new end-to-end method for this task. The main contribution of the work is the inclusion of a pre-training stage of two auxiliary tasks that help in learning relevant visual features to be used as a state representation to a recurrent policy. The method achieves state-of-the-art performance on publicy available datasets.", "questions": "Why is using panoramas treated as unrealistic with regards to robotic applications? RGB sensors (and even RGB-D) are relatively cheap and can be easily mounted on robots to cover a 360 degree field-of-view.\n\nThe authors cover some of the literature on pre-text tasks for improving sample efficienty but I think these are also worth discussing:\n[C] Ye et al, Auxiliary Tasks and Exploration Enable ObjectGoal Navigation, ICCV 2021\n[D] Sax et al, Mid-Level Visual Representations Improve Generalization and Sample Efficiency for Learning Visuomotor Policies, CoRL 2019\nEspecially regarding [D], shouldn't a representation trained on a multitude of vision tasks be more task-generalizable to just pre-training on relative pose estimation?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well written and easy to follow. The authors show that they understand the underlying challenges of this problem well and clearly explain their thought process behind the proposed approach.\n\nI agree with the authors that the bottleneck for these navigation problems is perception. I think the pretext tasks being proposed help in learning relevant representations for this task which would be otherwise very difficult to learn in typical end-to-end methods. Overall I think the paper proposes  a novel and sensible approach and moves the needle forward on this subject and improves the sample efficiency of these methods.", "weaknesses": "The title is somewhat misleading as the emergence of correspondences is really not the focus of the work, but more of an afterthought. The content of the paper might be mistaken as an investigation into this phenomenon that is carried out in this paper: \n[A] Tang et al, Emergent of Correspondences from Image Diffusion, arXiv, 2023.\nIn fact the emergence of correspondences from the learned representation is not really that surprising, given that the pre-training task is relative pose estimation. Even before the introduction of transformers, monocular pose estimation methods have shown that the representation learns to identify meaningful keypoints on objects. One example:\n[B] Mousavian et al, 3D Bounding box estimation using deep learning and geometry, CVPR 2017\n\nI appreciate the inclusion of the experiment where DEBiT was integrated with ANS for a direct comparison to a modular approach. However, I am surprised by the large performance gap to the proposed method. It seems unintuitive that a method that tries to map pixels directly to actions would outperform an approach that uses a map and de-couples the planning from the control. I think this should be looked more carefully to ensure fair comparison. Was the global policy of ANS finetuned with the frozen binocular encoder b and using the AdaptFormers? ANS was trained for object-goal so probably the Neural SLAM and global policy components need to be re-trained for the ImageGoal task. As it stands, the experiment is not convincing that end-to-end methods are better than modular approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rmqWVdwpNn", "reviewer": "Reviewer_isxL", "summary": "This paper improves the view correspondence encoding in image-goal navigation by pre-training a large binocular ViT model (DEBiT) with cross-view completion (CroCo) and relative pose estimation and visibility prediction (RPEV) from images in photorealistic indoor scenes. Specifically, the dual encoder in DEBiT takes binocular images as input, and the encoded features are merged with a decoder to model the correspondence between the two images. DEBiT is pre-trained with CroCo, followed by RPEV to estimate the relative distance, relative rotation, and overlap between two images. When addressing downstream Instance-ImageNav and ImageNav tasks, DEBiT takes the goal image and agent's observation as inputs, its network is adapted in tuning, and the output representations will be passed to the policy network for decision-making. Results show substantial improvement compared to previous approaches.", "questions": "Questions without a star (*) are not critical to my evaluation of this paper, but I still hope the authors can kindly and briefly respond to them. Please also respond to my concerns in Weaknesses.\n\n- (*) This paper mentioned depth inputs, but I wonder why depth images are not applied in the model since it might greatly facilitate learning view correspondence and identifying space and obstacles. (I might have overlooked some details; please correct me if I did.)\n\n- (*) How does the choice of $\\tau$ influence the pre-training and the downstream results? If two images are too distant away and have little overlap, will it be too noisy to learn? Or it might help the visual encoder to learn distant image correspondence and benefit exploration in navigation? Any numerical analysis on this?\n\n- (*) The paper claims the benefit of having visibility estimation but does not quantify its impact on downstream tasks. Any results for this?\n\n- Just curious, can CroCo and RPEV be trained simultaneously? \n\n- In Table 2, from Tiny to Large, the model size increases drastically, but the difference in navigation results is quite small; what might be the reason? Does it mean that a very large perception model might not be necessary? What would be the result of DEBiT-Tiny + Adapters?\n\n- Instance-ImageNav depicts targets viewed from a different camera; if this is the main reason why the proposed method gets much less improvement compared to ImageNav (an OOD situation as claimed), I wonder what if the images for pre-training are augmented with different camera parameters, (1) Is it still feasible to learn CroCo and RPEV? (2) Will it reduce the visual domain gap in downstream?\n\n- Are the visual encoders in OVRLs fine-tuned with the policy networks in downstream tasks? From their papers and Table 4, it seems Yes. But in the section Related Work, \"Once pre-trained, the encoder is often frozen before passing into a policy learning module.\"\n\n- About comparison experiments.\n    - Using adapters in the perception model largely improves the results. I wonder how much improvement it might bring to the previous approaches.\n    - In Table 3, for a fair comparison (in terms of pre-training tasks and #params), I think DEBiT-Tiny and RPEV-only should be listed, which I believe is more rigorous and can prove the same argument.\n    - (*) In Table 4, an important point to mention is the size of the visual encoders (#params) applied in each work. After scanning some papers in the Table, I believe there is a clear trend of larger-models-better-results. But it seems that DEBiT is relatively efficient, especially compared to OVRL-v2. I hope the authors can clarify this point, which I believe will also strengthen the argument.\n\nOther Suggestions:\n- Remove \"(Instance)-\" from the title.\n- I found a paper, \"Learning navigational visual representations with semantic map supervision (Hong et al., ICCV2023)\", which also focuses on learning better navigation visual encoder with view correspondence and uses two images for pre-training. It seems relevant and could be added to the references.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- This paper studies an important problem in visual navigation: learning high-capacity perception modules for modeling and reasoning view correspondence. It proposes DEBiT, which implements a dual encoder to enable early fusion of the images. It introduces CroCo and RPEV pre-training, which are highly relevant and have shown to be effective in addressing the problem. \n- Significant improvement is achieved compared to previous approaches, boosting the ImageNav and Instance-ImageNav results to 94% SR and 59.3% SR, respectively. The methods introduced in this paper and the resulting pre-trained models are very likely to inspire/to be used by future research in relevant fields.\n- The paper is technically sound; many important arguments and design choices are justified by experiments. Besides the key results, there are many highly constructive analyses/findings, such as \"directly training binocular encoder from scratch\", \"tuning visual encoder with downstream policy network\", \"early fusion vs. late fusion\", \"attention with scale changes\", etc., that are very valuable to future research.\n- Overall, this paper is nicely written; it is very compact, informative, and clear. All methods (and most of the implementation), visualizations, and discussions are clearly presented.", "weaknesses": "- The proposed DEBiT model seems to be limited in addressing image-goal navigation tasks (the image-nav task itself needs more justification), and it is unclear how it might benefit other visual navigation problems.\n    - However, I do believe that CroCo and RPEV can help in learning a general (and better) navigation-specific perception model (e.g., for obj-Nav, language-guided-Nav, Audio-Nav, etc.), while I am concerned that it might be much less effective compared to ImageNav which is defined to take two images as input.\n    \n- I am aware that ImageNav is an interesting visual navigation problem and has gained some research attention (publications), but I am still not convinced by its setting, especially how much practical value it might bring to real-world applications. [This question might be more appropriate for researchers who proposed ImageNav, but since this paper is devoted to ImageNav, I believe it must have a very strong reason and motivation behind.]\n    - The two ImageNav tasks consider indoor short-range navigation for finding static objects (Instance-ImageNav has paths with an avg. geodesic length of 12.41m); from the user's perspective (e.g., a household robot), giving instruction in the form of a specific image is unnatural.\n    - From the ImageNav data statistics and the visualization provided in this paper (supp. video), it seems that most of the paths have very few intersections, and the agent doesn't really need to explore the environment but keeps moving to new regions until the target is in its sight. This could be the reason why a very simple policy network, without methods like SLAM, can still lead to amazingly high performance (94% SR). I am concerned that the ImageNav task itself oversimplifies the problem, and this paper might overclaim the contribution to visual navigation research.\n\n- Some experiments are missing to justify arguments (see Questions below).\n\n- The limitation and future extension of this work are not discussed in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UuabeStU7O", "reviewer": "Reviewer_1KiF", "summary": "In this work, the authors introduce pretext tasks and a dual visual encoder for ImageNav and Instance-ImageNav navigation, offering rich geometric information and enabling the resolution of the challenging mono-view scenario with end-to-end trained techniques. \n\nThe method breaks down the problem into multiple training stages, demonstrating the emergence of solutions to the correspondence problem without explicit supervision. \n\nThrough experiments, the manuscript highlights the effectiveness of the proposed pretext tasks and a dedicated dual encoder architecture, surpassing competing methods and achieving  a novel state-of-the-art performance on both benchmarks. Additionally, the authors demonstrate seamless integration into a modular navigation pipeline of the proposed approach.", "questions": "Overall, I see an strong paper here.\nI would simply suggest to the authors that they provide explanations for the limitations I have described in the previous section.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- In the state of the art (Section 2), a detailed review of all the literature related to the problem investigated in this work is conducted. Furthermore, the main differences of the proposed model with respect to previous works are outlined.\n\n- The system proposed in Section 2 is novel. It describes an integration of subsystems/submodules that allows for an approximation to the problem, yielding highly promising results. Additionally, the proposed design can be considered innovative. The integration of Cross-View Completion (CroCo) into a navigation system as described has not been explored before, although it's true that this contribution is not the strongest of all. The second pretext task, relative pose estimation and visibility (RPEV) for navigation settings, is interesting and provides significant results. Overall, the originality of the work is considerable.\n\n- The experimental evaluation provides very interesting results in which the proposed model outperforms the state of the art (see tables 4 and 5). An ablation study is provided where the real impact of each of the contributions or parts of the model can be interpreted. Finally, qualitative evidence is presented on how the proposed pre-training process results in the emergence of correspondences between images by analyzing the attention of the last layers.", "weaknesses": "The experimental evaluation included in the paper has some minor limitations that should be addressed:\n- It is not clear on which databases or environments the system has been tested. For example, on what environments are the results reflected in Table 5 obtained? The interested reader has to review (Krantz et al., 2023) to know these details. Section 4, \"Experimental setup\" subsection needs to be improved.\n\n- I believe one of the most interesting contributions of the proposed model is that it generates correspondences between images due to how the pre-training is designed. Figure 4 shows some results or qualitative evidence in this regard. However, the paper does not explain in detail how this image is generated or how these correspondences are analyzed. The manuscript simply mentions that they \"visualize averaged attention of the last cross-attention layer of a DEBiT-L model\", but more details should be provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)