RANa: Retrieval-Augmented Navigation
Authors: Gianluca Monaci
Rafael S. Rezende
Romain Deffayet
Gabriela Csurka
Guillaume Bono
Hervé Déjean
Stéphane Clinchant
Christian Wolf
Venue: Accepted by TMLR ()

Abstract: Methods for navigation based on large-scale learning typically treat each episode as a new problem, where the agent is spawned with a clean memory in an unknown environment. While these generalization capabilities to an unknown environment are extremely important, we claim that, in a realistic setting, an agent should have the capacity of exploiting information collected during earlier robot operations. We address this by introducing a new retrieval-augmented agent, trained with RL, capable of querying a database collected from previous episodes in the same environment and learning how to integrate this additional context information. We introduce a unique agent architecture for the general navigation task, evaluated on ImageNav, Instance-ImageNav and ObjectNav. Our retrieval and context encoding methods are data-driven and employ vision foundation models (FM) for both semantic and geometric understanding. We propose new benchmarks for these settings and we show that retrieval allows zero-shot transfer across tasks and environments while significantly improving performance.

Published in Transactions on Machine Learning Research (11/2025)
RANa: Retrieval-Augmented Navigation
Gianluca Monaci, Rafacl S. Rezende, Romain Deffayet, Gabricla Csurka, Guillaume Bono, Hervé Déjean, Stéphane Clinchant and Christian Wolf All authors are affiliated with Nover Labs Europe https: // europe. naverlabs. com/ cww/ research/ http:r///pppllcccttooosooo/aaaa rana
Reviewed on OpenReview: https: // openreview. net/ for//frrm? id- OWCJ5JfsRR
Abstract
Methods for navigation based on large-scale learning typically treat each episode as a new problem, where the agent is spawned with a clean memory in an unknown environment. While those gencralization capabílitics to an unknown environment are extremely important, we claim that, in a realistic selting, an agent should have the capacity exploiting information collected during earlier robot operations. We address this by introducing new retrieval- augmented agent, trained with RL, capable of querying database collected from previous episodes in the same environment and learning how to integrate this additional context information. We introduce a unique agent architecture for the general navigation task, evaluated on ImageNav, Instance-ImageNav and ObjectNav. Our retrieval and context encoding methods are data-driven and employ vision foundation models (FM) for both semantic and geometric understanding. We propose new benchmarks for these settings and we show that retrieval allows zero-shot transfer across tasks and environments while significantly improving performance.
1 Introduction
A realistic setting for robot navigation is the "unboring" scenario, where one robot (or a fleet) is placed in its target sctting, startod-up, and navigates "out of the bor". The main appcal of this scenario is the lack of dependency environment preparation, like scanning the environment, installing external localization systems or generating floor plan prior to navigation. Most modern methods tackling this scenario, e.g. Wijmans ict al. (2019): : Chaplot et al. (2020a); Yadav et al. (2023a): Bono et al. (2024a), use lcarning-basse approaches in an episodic selling: every episode in the robot's 's life is treated as il it was its first. This obviously sub-optimal: during operation the scene explored, objects and their locations are observed, affordances found, failure cases encountered etc. Exploiting this information is crucial for designing efficient navigation agents that can contimuously improve.
Benchmarks like the k-item scenario BBeching et al. 2020b multi-object navigation Wani et al. 2020 or GOAT benchmark (Khanna et al. 2024) test the capacity of an agent to retain information from previous (subs)goals and trajectories. Structured latent memory trained end-to-end with RL performs best in these benchmarks (Marza et al. 2022). However, these architectures have only been tested on short (multi-)episodd of 500 to 2, 500 agent steps and are hardly suitable for a real continuous operation: transformer-baser agents with self-attention over time suffer from limited context length and the quadratic complexity of attention, while recurrent models are limited by the size of their latent memory and the network capacity growing quadratically with representation size (Jose et al. 2018). Alternative ways to integrate information from initial rollouts are topological maps (Savinov et al. 2018 Sridhar et al.) 2024). which require the estimation and generation of a structured graph model from a sequence of posed observations. The extension of these methods to multi-robot scenarios is anything but trivial.
In this work we propose simple data-driven approach to extend a stato-of-the-att navigation agent Bono
et al. 2024ah working out-of-the-box, with capabilities to store visual observations in global indexed

---

Published in Transactions on Machine Learning Research (11/2025)
ImageNev Instance-ImageeNv Task
ObjectNev
Task
Teak
ImageNay base arch.
R
Staroees +7.3
New LESA CheEk
New Lide carees
goal obs
Suczpess +3.3
Stacossa Hd.7
IC Base agent
M Add retrivved context
R-
Retrieve goat
(b)
cases
Semantic (DINOv2 Recrieval Contect Oener encocers Policy FM (CroCo)
R
π
act
DEBIFt augmented bassline!
Retrieval database
18.2
Semantic FM
Geom FM
Gurbel
D
24
Retrieval process
Context encoder
Graph Altn
10
(a) agent
(c) inference runtime
Retrieval-Augment Navigation (RANa). (a) We augment a navigation agent with context information retrieved from scene-specific retrieval database, fed with data by one or potentially multiple robots. We leverage semantic multi-modal foundation models for retrieval, and geometric foundation models for context encoding. (b) We tackle ImageNav, Instance- Instance-ImageNavv and ObjectNov tasks, demoustrating performance improvements (Success fx),, opening new use cases from existing models, and zero-shot applications with retrieval only added at test time (mD)). (c) Retrieval is efficient, negligible compared to vision models, with inference run-time in the same ballpark as non-angmented architectures like DEBiT.
Figure 1:
database, share them with a fleet, and retrieve and process them to improve performance during operation. The proposed architecture is general and can be used for different navigation tasks we evaluate our work on ImageNav, Destance-Imageln a semantic category, depending on the task, the retrieved information is always visual first person views (FPVs). Data-driven retricval mechanisms leverage vision foundation models (FM) like DINOv2 (Oquab) et al. 2024) to retrieve candidate views, while geometric FM, CroCo (Weinzaepfel et al.] 20222. integrates the retrieved context and provides directional information to the agent, cf. Fig. [p. Retrieval is also fast and keeps inference time in the same ball park of alternative baselines, cf. Fig. 1
and ObjectNav. While the goal may be specified either as an image or
This approach is scalable, as the dataset can potentially be very large and can be queried by any number of agents with optimized sub-linear time algorithms. Retrieving from an image database is also considerably simpler and more flexible than using involved memory structures such as metric or topological maps. While we can use any suitable vision or multi-modal FM represent, organize and retrieve images from an unstructured database, the latter approaches require pose to register observations and synchronization enable multi-agent. data collection: unless pose estimates absolute (which requires scene preparation), fusing new updates from multiple agents requires a jointly maintained map and distributed algorithms.
We propose the following contributions: (i) new navigation tasks, where the environment is augmented with a retricval database fed by one or multiple robots; (ii) a unified end-to-ond architecture for Retrioval-Augmente Navigation, RANa, for multiple tasks; (ili) data-driven retrieval mechanisms that do not require any meta- data beyond raw FPVs; (iv) mechanisms that leverage retrieved information either for additional context, or to replace the goal the fly, allowing for zero-shot applications, ef. Fig.
2 Related Work
Visual navigation The task of navigation has been addressed in robotics research using mapping and which requires solutions for mapping and localiza-
planning (Thrun et al. 2005 Macenski et al.] 2020).
tion Brrsson et al. 2017; Mur-Artal & Tardós 2017
Labbé & Michaud, 2019), planning (Sethian 1996
Rösmann et al. 2015). These methods depend on accurate sensor models, filtering, dynamical models and optimization schemes. In contrast, end-to-end models learn deep representations such as flat recurrett states Yaadav et al. 2023a Bono et al.) 2024a), occupancy maps (Chaplot al. (0200b), semantic maps (Chaplot et al. 2020a). latent metric maps (Henriques & Vodaldi 2018 Parisotto & Salakhutdinov 2018: Beeching et al.) 2020b Marza et al. 2022), topological maps (Savinov
Konolige 2000) and low-level control (Fox et al. 1997
et al.
2018 Beeching et al. 2020a Chaplot al.] 2020c
Sridhar et al. 2024 sell-attention (Fang et al. 2019:
2

---

Published in Transactions on Machine Learning Research (11/2025)
Du et al.
2021:
Chen et al.
2022
Reed et al.; 2022 Zeng et al. 2025) implicit representations (Marza
et al.
2023
Kwon et al. 2023).
Then, they map these representations into actions, using RL. (Jaderberg
itt al.
2017
Mirowski et al.
2017
Bono et al. 2024c), imitation learning (Ding et al. 2019) or by maximizing
navigability
Bono et al. 2024bb.
In this work; our goal to address end-to-end learning of representations and policies with RI. Although this problem nearly solved for PointNav, where the agent must navigate to a relative GPS coordinate (Savva et al. 2019 Wijmans et al., 2019 it remains challenge for other goal specifications.
Here we focus on ImageNav, where an image is provided as goal, requiring geometric understanding of the scene. A common variant is Instance- Instance-ImageeNar ImageNav (Krantz et al. 2023), where the goal image depicts a specific object in the enviromment and is taken with camera parameters different from those of the agent. Recently, the end-to-end model DEBIT (Bono et al. 2024a) achieved state-of- the-arr performance in ImageNav leveraging the goometric FM CroCo (Weinzaepfel ct al. 2022) to estimate relative poses between goal and agent's 's observation. In this work, we build upon DEBiT and augment it with retrieval, naturally exlending it to multi-robot setting and enabling zero-shot ObjectNav and Instance-Image.Val To the best of our knowledge other method performs all these tasks competently, the closest competitor being ZSON (Majumdar et al. 2022) which tackles ImageNav and zero-shot ObjectNav, with considerably lower performance (see Section 5
Retrieval-augnmented control- - The idea of retrieving information from a dataset of past experiences has been introduced in earlier works. Among them, episodic control methods for RL (Blundell et al. 2016 Pritzel et al. 2017 Lin et al.) 2018; Hansen et al.) 2019 He al., 2024: Hu al. 2021 act on successful experiences by re-employing Q-value estimates, enhancing sample efficiency during training. In contrast, exploit retrieved information from past experiences of a robot flect to enhance navigation performance and enable zero-shot lask generalization. Goyal et al. (2022) augment an RL agent with retrieval process parameterized as a neural network that has access to a dataset of past trajectories. Here instead, we take inspiration from Humphreys et al. (2022) that, to our knowledge, was the first work to propose a generic non-parameterized retrieval augmentation for an RL agent with previously learned embeddings. Alleviating the need to pre-train embeddings with RT, we use vision foundation models to process stored and retrieved images. This enables zero-shot transfer and allows to adapt the approach to various navigation tasks.
Retrieval-Aunmmented Planning (Kagaya et al. 2024 uses a memory of successful task executions, comprising plans, actions and observation sequences. Past experiences relevant to the task at hand are leveraged to guide planning with an LLM agent based on task constraints and state similarity. Xie et al. (2024) build a hicrarchical topological memory to prompt an LLM for reasoning and goal-finding in a navigable environment. In contrast, we only store images derived from unstructured observations without any associated metadata.
Navigation supported by prior rollouts Other navigation methods exploit previous observations to construct topological map of the explored scene, and use it to guide navigation (Savinov et al.) 2018
Chaplot et al.] 2020c Shah & Levine, 2022 Sridhar et al., 2024b. In contrast, our database is unstructured and we can query it to retrieve goal and context images with simple image retrieval. Also, here context
images are not used as navigation waypoints, but as recommendations, exploited or not by the agent, leading to navigation performance robust to potentially misleading context, as shown in Section 5
In Table D we compare RANa with selected navigation mothods discussed above in the continual navigation scenario. Most approaches require depth and pose and/or more involved (metric or topological) map updates, which need least to register observations. In contrast, RANa does not require depth or pose, the database update and retrieval is straighttorward, and naturally supports multiple data collection robots without the need of synchronization. While LLAM-based solutions (Engel et al. 2014 Mur-Artal & Tardós 2017) are well established for continual navigation, with RANa we propose a lightweight, data-driven solution that can be easily scaled up to multiple robots and can tackle ImageNav, Instance- Instance-ImageNNr and ObjectNav use cases.
3 Retrieval-augmenned agent
We target navigation 3D environments, where an agent tasked to navigate from a starting location to a goal and receives at each timestep t an image observation x,. Our method is general and the agent can be trained for diverse range of tasks, but without loss of generality in this presentation we focus on ImageNon,
3

---

Published in Transactions on Machine Learning Research (11/2025)
Method and
Img
Obj
Not required:
Easiness
Multiplo
representation
Nav
Nav
pose
depth
representation update
collectors
GOAT (Khanna et. al.
2024
V
X
K
map registration
X
3D Scene graphs Yim al
2024
X
V
X
X
graph-tma registration
X
Topo maps (Savinov al.
2018
Shah & Levine
2022
X
V
X
X
graph update map registration
X
Semantic maps [Chaplot et al.
2020a
X
V
X
X
X
Sem. Implicit MMaza et
X
V
X
X
X
V update pase graph map registration
X
Feature SLAM (Mur-Artal & Tardós 2017
X
V
Dense SLAM (Engel al.
2014
X
V
RANa
store (matrix update)
T order to support mulltiple collectors, a method should either be pose free, like RANa, or use distribated algorithoms to synchrondze, or absolate pose (and not episodsc).
Table 1: RANa compared to SOTA in continual navigation. RANa is a flexible, data-driven solution that can easily exploit data from previous episodes, in particular collected by multiple robots: it only requires storing images in dataset without any additional metadata, and pre-compute retrieval features, v.g.DINOv2. In case of dynamic contert (ef. Section 4.2 a similarity matrix, computed offline, is also incrementally updated when new elements are added.
where the agent receives goal image g. An extension to the ObjectNav task is presented in Section The action space is discrete, A =[MOVE FORMARD o. 25m, TURN LEFT 10", TURN RIGHT 10,, and STOP}. Navigation is considered successful if the STOP action is selected when the agent is within 1m of the goal position in terms of geodesic distance.
Our set-up based on the ImageNav task of the Habitat simulator and platform (Savva al.) 2019). We extend it with a retrieval dataset D and retrieval mechanism R, which will be detailed in Section The dataset D (xP) contains FPVs indexed by i and stemming from previous episodes from exploratory rollouts in the scene. While it is in principle possible to store additional meta-dala associated to data collection, like approximate pose, performed actions, reward, success or value (in an MDP sense, when available), in Section 5 we show that access to FPVs alone provides rich, useful information to the agent.
The agent queries this dataset DD with the retrieval mechanism R, which can be potentially used at each time step f. R uses the goal g: and optionally the current observation x2, to retrieve a set of FPVs R(x,, g) - (rr,1: Tr,2,... that is used context information to improve navigation. Depending on the use case, the use of observation x, in the retrieval process may be optional (as in the static contect in Section 4 if not provided, retrieval can be done once per episode, opposed to every time step.
We propose to integrate the retrieval mechanism R into a recurrent agent that maps observations and goals to actions. The non-angmented base agent can be described as
x; = x(xt)
11 perceive
g: = g(x,, g)
// compare obstgoal
h h h(b 1. Nt, g., (ax-1)) // update recurrent state
a, x (h+),
where x and g are trainable encoders, the update function of GRU (Cho ct al. 2014) which maintains hidden state h over time; is an embedding function, and æ is the policy. For clarity have omitted the equations of the gating functions.
Then, the retrieval-augmented agent is defined as:
X axx))
/1 perceive
g. = g(Xr. g)
/1 compare obsegoal
C, = (R(xx.g), X1,8)
// retrieve and ancoda context
h; = 1ohh-1, &x, Bis Cr (a) // update recurrent state
a, Thh),
act
where e an encoder which compresses the retrieved context: into compact form. It is trained end-to-end together with the agent, as described in Section 3.1 while the retrieval process R, described in Section
We denote functions with daki, tensors with bold face, and encoded tensors with <bold face.
4

---

Published in Transactions on Machine Learning Research (11/2025)
Task
Goal
Gool
dynyaam context
Oiservattoo
Oleevvalion
Retrieval database
Retrieval
hwAN18
ImageGoal
D
R
ImageGooll g
Goal g
GeomFM g
h
Policy
Context
Imagetioal
ag-1
Embed
GRU
Instance ImageGoal
GeomFM
Context
Frozee
GeomFM
encoder
Thnnneed
ObjectGoal "Find chair"
Obssivation
Trained
Figure 2:
We propose an architecture for Retrieval-Augmentt Navigation architecture RANa,, that addresses all use of Fig. Tb. Dashed arrows optional depending on the navigation task.
leverages pre-trained models. The observation encoder xr is implemented as a half-width ResNet-18 (hwRN18) architecture (He et al. 2016].
The image goal g is compared to the observation x, through a function ( g) implemented with a binocular visual encoder from Bono et al. (2024a) that returns an embedding representing information about the goal's dircction, pose, and visibility. This encoder was fine-tuned from the foundation model CroCo (Weinzaepfel et al. 2022); solving as pre-text tasks relative-pose and visibility estimation.
Additionally, an image goal gR can be retrieved from D given the task goal g, allowing the ImageNav model to address tasks in zero-shot way. For example, an ImageNav agent can perform ObjectNav by retrieving a
goal image gº from the database D given a goal category (x- x- -a)) In this case, the goal comparison function g is computed between x, and g (instead of the original goal g) as g(x, We explore this use of retrieval for zero-shot task generalization in Section 5.2
In summary, the agent receives at each timestep a new visual observation x which it encodes, compares with the goal g, and enriches with additional context information retrieved from the database D. This representation is then passed to recurrent policy. See Fig, 2 for visual overview.
3.1 Trainable context encoder
At each timestep t, the task of the context encoder e to extract useful information from the list R(x, g) (ri of retrieved context. images. To do so, we argue that the comparison of retrieved images, goal and current observation is of geometric nature and can be performed by leveraging a pre-trained poso-cstimator, or by geometric FM that could extract relevant directional information to be given to the agent. Therefore, we explore the same geometric FM CroCo used for the binocular encoder to translate the context (rt,a) into a set of embeddings ( by passing them through the binocular encoder along with the current observation and the goal image:
e = [g((x, at.a) g(g, r..
At this point, we obtain a list of embeddings et.n that benefit from the rich inductive bias provided by the geometric FM, including information about pose and visibility of the context image Ft,n from the observed image X2 and goal image g. The list is compressed into a single context embedding è₁ = c(R((,,,)) x,, that provides useful information to the agent, and for this we explore two variants.
Variant 1: Gumbel soft-max selector The agent selects single embedding from the list by predicting distribution over items in the retrieved list followed by Gumbel soft-mat loft-ax sampler (Jang et al. 2017
è (fann)a-)) a₂ Linear(er.n)
(2)
where a is the soft-max distribution over items the list. We perform the sampling and the computation of gradients using the Gumbel soft-mat trick (Jang et al. 2017). We motivate this choice from the pre-training of the geometric FM g in Eq. 1 the latent vector g(a, b) has been pre-trained to provide overlap
5

---

Published in Transactions on Machine Learning Research (11/2025)
information between images and b, which we conjecture to be highly correlated with the relevance of the corresponding context item. Additionally, since the weights oz are calculated independently, the model can be zero-shot applied to any context size at test time.
Variant 2: Attention-based encoder
transformer based model is potentially capable of integrating multiple context tokens {er,,) into a single fused embedding, effectively changing and enriching the original embedding space. We use transformer encoder layer with 2 layers of 4-heads self-attention, which receives as imput the N context tokens summed with positional embeddings, and predicts the context feature è by concatenating the output tokens and feeding them to a 2-layer MLP.
)
4 Retrieval mechanisms for RANa
The retrieval process performs a nearest neighbor search on image representations obtained with DI- NOv2 (Oquab et al., 2024) resulting in retrieved context. We investigate two context building strategies:
1. Static contert, where only the goal is used to build fixed context kept throughout the episode,
2. Dynamic contert, which also depends the current observation and is recomputed at each timestep.
4.1 Static context
We build the static context by ranking images from the dataset based on DINOv2 feature similarity with the goal and selecting the top-N nearest neighbors.
Diversity - providing multiple near-duplicates to the selector might not be very helpful. We therefore resort to Marimum Margin Relevance (MMR) from Carbonell & Goldstein (1998) re-ranking to increase diversity. Assuming relevance score vector (DINOv2 similarities) providing a ranked shortlist (top-100 in our experiments), and matrix Ω encoding the similarity for each pair of images in the ranked shortlist, MMR greedily selects at each step (rank) i the element T3 that maximizes the re-ranking criterion:
MMR(T.) (w(r.) - (1 max,,es._, (((,,,))()r))))))
where E [0, 1] is a mixture parameter (set to 0.5 in all our experiments) and S,-1 is the set of images already selected. We then select the top-N highest MMR. The impact of MMR is studied in Appendix [)
4.2 Dynamic context set
Dynamic context exploits the current observation at each time step, and we leverage this to use the context set to provide intermediate waypoint images to the augmented agent. Similar to topological maps we do this by building a graph and calculating shorlest paths. However, in stark contrast to classical topological maps, (i) the graph is built from the images of the retrieval dataset only and can be pre-computed, (ii) no pose information is required, we use visual similaritics only, and (iii) waypoint images are recommendations only, exploited or not by the agent through its retrieval context.
DINOv2-Graph - Given D, we build the affinity matrix Up containing the similarity between all pairs of images in D using DINOv2 feature cosine similarity (Oquab et al.) 2024). From Up we can derive a graph g (W whose nodes N are images x," E D and edges E are weighted by similarity. Then, we find in D the most similar images rt and rf to the observation X; and goal g, respectively. Finally, we populate the context R(x,,g) g) with rf rf and images sampled on the shortest path in the graph between them.
Fig. 8 shows three examples of shortest paths the HINOv2-Grapha in green. For comparison only, we also visualize the shortest paths obtained by the ground-truth (GT) "Pose Graph", not available to the agent. The path on the DINOv2-Graph well approximates the GT path. We provide more details about the graph construction and its relation to topological maps in Appendix C
Retricving the goal independent of the creation of a retrieval context, static or dynamic, zero-shot settings we also use retrieval replace goal g with a retrieved image goal R (R- D. x- --) setting g as the nearest neighbor of g from the dataset D.
6

---

Published in Transactions on Machine Learning Research (11/2025)
Figure 3: Shortest paths on DINOv2- G-rrpph and Pose Graph The most similar image r E D to the goal is highlighted red and localized with red star on the floorplan, while of & D, in blue, the most similar image to the current observation. Green crosses connects images along the shortest-path in the DINOv2-Graph (corresponding images in green) and purple crosses refer to poses on the Pose Graph, built using distances between GT camera. poses, with corresponding images highlighted in purple at the bottom of the map.
5 Experimental results
We train and evaluate our agents on the Habitat simulator and platform (avvaa et al. 2019 according to the standard ImageNav, Instance- Naa and ObjectNav task definitions.
Retrieval-augmented navigation benchmarks There is no publicly available benchmark that provides database of observations that can be exploited during training and evaluation. We propose to derive our benchmarks from standard Habitat tasks (Savva et al. 2019), modifying them as little possible to enable training and evaluating retrieval-angmented agents. We use the Gibson dataset (Xia et al. 2018), consisting of 72 train and 14 eval scenes, and for each scene, unless otherwise stated, we generate a retrieval set. 'D of as little as 1, 7,000 FPV images by letting agents navigate the environment. and store images in D. During training and evaluation, the retrieval-augee -uggeeetdd agent can select dataset images from the scene it operates Additional details on dataset creation are provided in Appendix A
Implementation details We base agents the DEBIT architecture by starting from the official codebase and weights provided by the author and extend them as described in Section 3 We consider two DEBIT backbones, base (DEBiT-b) with binocular encoder with 55M parameters, and tiny (DEBiT-t), with a 17M-parameters encoder, to allow faster execution of ablation studies. The corresponding retrieval- augmented models are RANa-b and RANa-t. In Section 5.2 we use the same agents and weights in zero-shot lashion for Instance- Instance- and ObjectNavv when the lask goal replaced with a retrieved image goal More details about the agent architecture are provided in Appendix B
Model training All models, retrieval-anngmeneee not, are trained with PPO up 200M steps. Unless specified otherwise, the geometric FM g and encoders x and are loaded from DEBiT and kept frozen, as is DINOv2. For retrievalaaugmenter agents, the context encoder and policy network π are learned from scratch. Compared to DEBIT, the GRU memory of RANa receives the additional context input ét, therefore we expand the DEBIT GRU imput matrix to reach the required dimension and finetune it. Further details and ablations of this choice are reported Appendix D The reward definition inspired by PointNav (Chattopadhyay let al. 2021) and ImageNav (Bono et al. 2024a) rewards and given as:
K snocess
A Geo
X,
where K=10, A Geo is the increase in geodesic distance to the goal, and slack cost X=0.01 encourages efficiency.
Retrieval We consider two context encoding approaches, Gumbel and Attention, and two retrieval mechanisms, static based on DINOv2, and dynamic based on DINOv2-Graph. The context size is N8 in all experiments (sce Appendix D for ablations). For dynamic variants requiring DINOv2 to be applied to each
hhttp:://github.ccmmmmaa
7

---

Published in Transactions on Machine Learning Research (11/2025)
Model
Retrieval
Context Enc.
SR
SPL
DEBiT-t [Bono et al.
2024a)
79.9
51.4
RANa-t static
DINOv2
Gumbel
83.0
52.3
12
RANa-t RANa-t
DINOv2
Attention
81.9
54.3
12
DINOv2-Graph
Gumbel
82.0
51.5
RANa-t dynamic
DINOv2-Graph
Attention
84.7
63.7
DEBiT-b (Booo al.
2024a)
83.4
56.8
02
RANa-b static
DINOv2
Gumbel
88.3
57.6
02
RANa-b
DINOv2
Attention
84.3
58.3
RANa-b
DINOv2-Graph
Gumbel
89.7
60.4
RANa-b dynamic
DINOx2-Graph
Attention
90.7
71.8
Table Effect of retrieving context, RANa-t and models.
Model
Odom
SR
SPL
Binocular encoder
OVRL Yadav et al.
2023b)
54.2 27.0
Finetuned
VCI-ViT-L Majumda el
2023
Finetuned
OVRL-v2 Yadav et al.
2023a)
82.0 58.7
Finetuned
ZSEL (Al-Halah et al.
2022)
X
29.2
21.6
Obs. & policy frozen, goal from scratch
ZSON Majumdaa
2022
x 36.9
28.0
Observation finetuned, goal frozen
PGPrompt (Sun ot ol.
2024a
X 92.3
68.5
Trained from scrotch
DEBiT-b no adaplers (Bono al.
2024a
X 83.0
55.6
Frozen
RANa-b dynamic
X 90.7
71.8
Frozen
DEBiT-L w. adapters (Bono et 2024a)
X
94.0
71.7
Finetuned with adapters Finetuned with adapters
RANa-L w. adapters dynamic
X
94.5
76.2
Table 3: ImageNav state-of-the-art results. "this work sets the Habitat simulation parameter sliding=True,
simplifying the task significantly and making it incomparable with other methods (cf. (Monaci et al., 2025
observation, again leverage retrieval to speed up RL training significantly: during training, we replace observations by their nearest dataset image, allowing to use pre-computed DINOv2 features.
Metrics Navigation performance is evaluated by success rate (SR), i.e., the percentage of episodes terminated within a distance of <lm from the goal after the agent has called the STOP action, and SPL (Anderson et al. 2018 i.e., SR weighted by the optimality of the path:
SPL
100
Σ S
C
max((,, E?)
where S; is binary success indicator in episode i, E is the agent path length and E the shortest path length.
5.1 Retrieval improves navigation
Table 2 shows results of the proposed RANa agents with retricved context on the ImageNav task, exhibiting significant performance improvements across all settings compared to the DEBiT baselines For static context retrieval the Gumbel encoder shows better results, arguably because there is no inherent ordering in the context and the retrieved CroCo-based features contain strong information about the relative pose between context elements and observation. To simplify notation, from now on we label the combination of static context. retrieval and Gumbel encoding static. The attention-based encoder works better for the dynamic graph-based context, where reasoning about the order of context elements becomes important. We label this graph-based context combined with altention encoding dynamic. It obtains the best results, with a huge boost in SPL (+15 and +12.3 points) compared to the DEBiT-b and DEBiT-t baselines, respectively.
Comparison to ImageNav SOTA - In Table 3 we compare RANa with statc-of-tho-art (SOTA) ImageNav models. These typically finetune (with a navigation loss) visual encoders based on CLIP (Majumdar et al.
TTo make baselines comparable, loaded checkpoints continued training for the same additional 200M steps, updating GRU and policy only.

---

Published in Transactions on Machine Learning Research (11/2025)
Model
Nr Par
SR
SPL
Tot. runtime (ms)
DINOv2 Retrieval Context
H+x
CRU+T
CroCo
DEBiT-I
27M
79.3
50.0
13.2
1.7
1.1
10.4
DEBiT-b
66M
83.0
55.6
25.2
1.7
1.1
22.4
RANa-t static
32M
83.0
52.3
24.4
0.1
0.1
0.4
1.7
1.2
20.9
RANa-t dynamic
34M
84.7
63.7
38.5
11.9
0.5
2.2
1.7
1.2
21.0
Table 4: Inference runtime of different model components, in ms, timed one Nvidia H100-80G GPU. Retrieval overhead is negligible, most runtime is spent on image feature computations (DINOv2, CroCo), which increase with retrieval because of additional operations needed to select and compare retrieved elements.
DB size
100
1k
10k
20k
1k
81.7
(50.1) 82.7 (52.6) 6)
82.1 (51.2)
82.8 (51.9)
10k
80.6 (49.4) 83.0 (52.3)
79.8 (46.3)
82.2 (51.5)
Table 5: Impact of retrieval database size SR (SPL) for RANa-t with static context using different data sizes, combinations of train and test conditions.
2022 VIT Yadav et al., 2023b Majumdar et al. 2023 Yadav et al. 2023a) and geometric FMs (Bono et al. 2024a used for observation-gon goal comparison. Not only RANa considerably boosts the performance of the strong DEBiT-b model, but it also enhance the SOTA performance of DEBiT-Large (Bono et al. 2024a) finetuned with adapters (Chen et. al.] (002)). While SR is only marginally improved, arguably because it is already very high (94) and remaining failure cases are not solvable using context, path efficiency (expressed by SPL) increases significantly, from 71.7 to 76.2 points.
Retrieval is efficient Table 4 shows model size and runtime of different components of selected DEBiT and RANa models. Retrieval time is negligible and scales favorably with the dataset size thanks to efficient. approximate nearest neighbor search libraries such as Faiss (Douze et al. 2024). Inference time is slightly higher for RANa because the encoder с computes CroCo features between context images and observation and goal (while DEBIT only runs CroCo between observation and goal). Still, RANa-t is smaller and faster than DEBiT-b while achieving similar performance, suggesting that retrieval augmentation can be a frugal alternative to model scaling. The variant with dynamic conlext adds the compute time required to run DINOv2 on each observation, but its runtime remains in the same ballpark of other agents.
100 dataset images suffice to improve performance Table shows that the size of the retrieval dalabase can be small, 100 images per scene sullice for the considered Gibson scenes, and performance plateaus at 1K images. Additional experiments for mestance-Image.ie and ObjectNav are presented in Appendix D
The geometric FM provides essential directional information - and plays a crucial role in the effectiveness of RANa. We validate this intuition by training a RANa-t model with slatic context where the frozen CroCo encoders g in Eq. (I) are replaced by a hwRN18. This network has the same architecture of the encoder æ, but takes input 9x112x112 tensor formed by chamnol-coneatenatin observation, goal and context, and outputs the context features er,n fed to the Gumbel selector. This encoder is trained together with the rest of the RANa model. Table Ü that this model achieves 79.1 SR, against 83.0 SR of RaNaat: in this setting, the agent is not capable of exploiting context information and performance remain at DEBIT-t baseline level, validating the importance of the geometric pre-training of CroCo FM in Bono et al. 2024a).
Context robustness and failure cases 7 studies the robustness of RANa-t potentially misleading context elements: (i) in domain: expected retrieved elements, and (ii) random: random images from the scene dataset. Providing random context images results in performance drop of both RANa static and dynamic to baseline levels. These results support two key properties of the proposed approach: the context brings useful
Model
Retrieval Features
SR
SPL
DEBiT-t
79.9
51.4
RANa-t static
CraCo Geometric FM
83.0
52.3
25
RANa-t static
hwRN18
79.1
48.2
Table 6: Impact of Geometric FM on context representation.
9

---

Published in Transactions on Machine Learning Research (11/2025)
Model
in domain
random
RAN statie
83.0 (52.3) 77.0 (47.1)
RANa-t dynamic
84.7 (63.7) 7)
80.3 (50.2)
Table Impact retrieved context SR (SPL. for RANa-t models on different test conditions.
Otsemssion
Apent path
Fallre
Figure 4: A failure case: RANa static reaches the zone of the goal image (bottom on the map) and then starts turning around, circling through context items, while the DEBIT baseline (on the right) quickly navigates the goal.
information and RANa learns to exploit it, and also, RANa can filter out irrelevant context elements and is robust to misleading random retrieval. In fairness, we observe rare occurrences in which access to context deteriorates the performance of the bascline agent. In these cases RANa seems "distracted" by context items, and Fig. shows typical instance of this phenomenon: the baseline DEBiT agent (trajectory on the right panel) quickly gets to the goal, while RANa reaches the goal area, but starts turning on itself, cycling multiple times through relevant, but "distracting", context images, leading to time-out.
Training curves- RANa agents are trained starting form DEBiT checkpoints, training curves are stable and quickly converge to high values. Fig. 5 shows the evolution of Return and Success Rate (SR) during training for R.ANa-t with static Gumbel context encoder, in pink, and DEBiT finetuned for additional 200M steps, in gray. The behavior of the two models is comparable, but RANa achieves higher values in both metrics.
Examples of retrieved context Fig- shows three types of context for = 8. Top 8 most similar dataset images the goal in DINOv2 feature space (top) and top 8 MMR most similar images filtered by MMR (middle), are static context types, t.e.they are fixed along one episode and depend only the goal image, shown on the right highlighted in red. DINOv2- Graph (bottom) is dynamic context, recalculated at each step and depending on the observation (left, highlighted in blue).
The top most similar images to the goal (top) are very similar to each other as well, and might not be very useful. Filtering images using MMR increases diversity and performance (cf. Appendix D for a qualitative analysis). The dynamic context (bottom) is the most informative, as it provides some form of "waypoints"
Success Rate
0.8
Relurn
0.6
0.4
0.2
40M
80M
120M
160M
200M
training steps
training steps
Figure Training curves: Return and Success Rate for RANa-t static (pink) and DEBiT (gray).
10

---

Published in Transactions on Machine Learning Research (11/2025)
Static, top
Static, top MMR
Dynamic, DINOvB- Graph
Figure Context types top most similar dataset images to goal (red, on the right), top MMR, most. similar images filtered by MMR, and dynamic, the only one depending the observation (blue, left) well.
ObjNav SR
IIN
Model
Extra sensors
SPL
SR
SPL
ZSEL
HAl-Halah et al. 2022)
11.3
ZSON
Majumdar et al.
2022
31.3
12.0
16.1
Mod-INN
Krantz al.
2023
Depth, odom
56.1
23.3
DEBiT-b
31.9
15.8
55.4
25.5
RANa-b
36.6
17.9
58.7
26.8
Table 8: sero navigation results using retrieved image goal g for Gibson ObjectNov and
HM3D Instance- Naageeen and comparison to state-of-the-art. obtained with camera intrinsics and rohot size of the original Instance- Nam task specification.
from the current location to the goal indeed this approach achieves the best navigation performance, shown in Tables and 3
5.2 Retrieval allows zero-shot generalization
By retrieving the image goal g from the database, it is possible to apply any ImageNav model to the ObjectNav and Instance-ImageNi tasks. Table compares the performance zero-shot architectures in: (i) Gūbson ObjectNan, variant introduced in Al-Halah et al. (2022) for ImageNNa agents, consisting of 10000 episodes over 5 Gibson scenes and containing the 6 object categories of HM3D ObjectNav (chair, bed, plant, sofa, le, loilet), and (ii) HM3D Mislance-Inoge.ee robot size to fit the ImageNav configuration. In both cases we use retrieval databases of 50, 000 images, it is important to retrieve an image goal gº as close as possible to the real goal to avoid stopping too far from it. We ablate this parameter in Appendix D and show that RANa already achieves SOTA zero-shot Instence-TmageNNar performance with as little as 5, 000 retrieval images. Tn Imstance-ImageNee we replace the image goal with the closest image in DINOv2 feature space from the retrieval set. In zero-shot ObjectNaa, where the goal is object category, retrieve the top 9 closest dataset images using OpenCLIP (Ilharco) et al. 2021 feature similarity, and re-rank them al each step to select as goal gR the element with the highest. CO- visibility with the current observation (the remaining 8 images constitute the context). This is achieved using the frozen co-visibility auxiliary loss head that was used to train DEBiT models. This step is needed since in ObjectNav any instance of a target object is a valid goal.
(Krantz et al.] 2023) with modified camera intrinsics and
Retrioval enables the use of DEBIT-b for both tasks, achicving strong performance in line with best existing methods (for reference, zero-shot DEBIT-b without retrieved goal image achieves 10 SR and 3.1 SPL on Instance-Image.Vrt RANa outperforms the feature-matching based method of Krantz et al (2023 which uses additional depth and odometry sensors and is the current SOTA in zero-shot restance-Imagelle as well as ZSON (configuration A) of Majumdar et al. 2022 in zero-shot Gibson ObjectNav.
Fig. 7 shows an example of successful zero-shot Instance- Instance-ImageNNane ImageNavigation, where our RANa agent first correctly picks from the context the door near the goal I, and then view of the room where the target is 2. leading to success. contrast, the DEBiT-b baseline without context does not see the target inside the and passes by, exploring the wrong side of the house and failing to complete the episode.
11

---

Published in Transactions on Machine Learning Research (11/2025)
Indtance Retrevadg
Besaline without context
Failure
Contoxt
Success
Figure 7: Zero-shot lssanne--mmageNav episode by RANa-b and by DEBÍT-b baseline (top-right box). The green camera indicates the pose of the retrieved image goal g the pink ones indicate the poses of selected context items.
Retrieved
ImgGoal
Retrieved
ImgGoal
Context
Context
Model SPL
DEBIT-b
40.5 17.1
RANa-b
16.5 17.0
Success
Fa llure
Figure 8: Navigation with real goal images, retrieving new goals: 2 episodes shown, 200 episodes evaluated.
5.3 Retrieval allows bridging the sim-to-real gap
We evaluate our -- RANa-b agent on a small Pistance-Image-Ve datasct of 200 episodes where goal images are taken in a real office environment. We manually annotate the positions of 20 real images a simulated 3D reconstruction of the environment. We generate a retrieval set D of 10k images from uniformly-sampled navigable poses, rendered in simulation using the agent configuration. Simulating navigation from 10 random starting poses towards each of the 20 real image goals, RANa reaches 46.5 success, vs. 40.5 of DEBIT-b (see inlet Table in Fig. S). The Figure illustrates a challenging success case (left), where the cactus viewing angle is far from any view the agent could capture, and (right) an interesting failure case, where the retricval returned as goal g2 an image of a different instance of the same microwave model al a different position. Interestingly, some retrieved context images depict the correct goal, and RANa appears to able to exploit this information as succcods in 3/10 episodes fcaturing this goal.
6 Extension to ObjectNav
We exlend the proposed approach to ObjectNav, where the goal is a textual label of an object category, by slightly modifying the RANa architecture. Since ObjectNav requires semantic understanding the scene and common-sense reasoning, a large body of research builds semantic maps (Gadre et al., [022 Chaplot al. 2020a) and/or leverages an LLM to guide navigation based on a scene's common sense semantics (Yu
et al.
2023 Cai et. al. 2024 Kuang et. al. 2024
Yin et al. 2024 Few approaches address this task in
12

---

Published in Transactions on Machine Learning Research (11/2025)
Model
Extra semsors w/ map w/ LLM
SR
SPL
PixNav CCai et al.
2024
Pano RGB
X
/
37.9
20.5
ESC Zhou al
2023
Depth, odom
/
/
39.2
22.3
L3MVN Yu et al.
2023
Depth, odom
/
/
54.2
25.5
CoW (Gadre et
2022
Depth, odom
/
x
6.1
3.9
GoW Gadre et al.
2023
Depth, odom
/
x
32.0
18.1
ProcTHOR (Deikke et al.
2022
Depth, odom
x
13.2
7.7
OVRL-v2 Yadav et al. 2023a
Odom
x
x
64.7
28.1
7SON MMjjumdar et al.
2022
X
X
25.5
12.6
PST.
Sun et al. 2024b)
X
X
42.4
19.2
DEBIT-t ObjectNav
X
X
42.9
23.0
RANa-t ObjectNav
X
X
52.1
26.9
Table 9: DEBiT-t and RANa-t ObjectNav compared to state-of-the-art methods.
zero-shot fashion, training their agents without ObjectNav rewards and leveraging CLIP encoders for goal representation (Sun al.) 2024b Majumdar et al. 2022 Gadre et al., 2023). Here we do not aim achieve srate-of-the-ar results, as is unlikely for our simple approach to compete with methods using LLMs and VLMs with billion of parameters. Instead we want to showcase the effectiveness and ficxibility of our method.
In order to do that, we simply modily few components of the RANa-t static architecture: to account for the textual nature of the goal we use the multi-modal FM OpenCLIP (Ilharco et al. 2021) to compute the goal representation & and the similarity mcasure used for retrieval. Besides, ObjectNav has two additional actions, LOOK UP and LOOK DOWN, that are added to the action space. We train and lest this agent, as well as a baseline as DEBiT-t ObjectNav), on the HM3DSem dataset (Ramakrishnan et al. 2021 Yadav al.) 2023c) with 6 object categories (chair, bed, plant, sofa, fv, toilet) using the standard Habitat ObjectNav task definition and following the same procedure described earlier.
not augmented with retrieval (we refer
Table 9 compares the results of RANa-t and DEBiT-t ObjectNav models with competing approaches. RANa can exploit retrieved context also in this setting and improves the DEBiT baseline SR by almost +10 points. Compared to methods with similar sensor settings (only one RGB camera), RANa-t outperforms previous SOTA by +9.7 on SR and +7.7 on SPL. We also compare RANa-t to methods that use extra sensors, such as depth camera, GPS+compass, or camera sensors, as well as LLM-guided navigation. Here RANa-t shows competitive results, outperforming several strong baselines such ESC (Zhou et al. 2023 and PixNav (Cai
et al. 2024
7 Conclusion
We proposc general retrieval-augmeenee navigation agent, trained with RL, that is able to retrieve and act on images stored in a large, global database of robot observations. We leverage pre-trained foundation models to allow the agent to query and process contextual information that can be useful for solving its target task. Augmenting SOTA ImageNavagenn with a context retrieval module improves navigation performance while adding minimal overhcad, which showcases the ficxibility and effectiveness of the proposed method. Moreover, we demonstrate the approach for Objecl agent, and we show how to use retrieval across domains and modalities to apply existing agents to new tasks in a zero-shot fashion.
This work opens the possibility of letting navigation agents benefit from the large amount of unstructured data collected during the operation of robot fleets. Future work will explore the integration metadata to the database and data-driven query mechanisms that allow an agent to target the information it needs.
References
Ziad Al-Halah, Santhosh Kumar Ramakrishnan, and Kristen Grauman. Zero experience required: Plug & play modular transfer learning for semantic visual navigation. In IEEE/CVP /CVF Conference on Computer Vision and Pottern Recognition (CVPR), 2022.
13

---

Published in Transactions on Machine Learning Research (11/2025)
Peter Anderson, Angel X. Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, and Amir Zamir. On Evaluation of Embodied Navigation Agents. arXiv:1807.0657, 2018.
Edward Beeching, Jilles Dibangoye, Olivier Simonin, and Christian Wolf. Learning to Plan with Uncertain Topological Maps. In European Conference on Computer Vision (ECCV), 2020a.
Edward Beeching, Jilles Dibangoye, Olivier Simonin, and Christian Wolf. EgoMap: Projective Mapping and Structured Egocentric Memory for Deep RL. In European Conference Machine Learning and Principles and Praclice of Knowledge Discovery in Dalabases (ECMI-PKDD), 2020b.
Charles Blundell, Benigno Uria, Alexander Pritzel, Yazhe Li, Avraham Ruderman, Joel Z. Leibo, Jack Rac, Daan Wierstra, and Demis Hassabis. Model-Free Episodic Control. arXiv:1606.04460, 2016.
Guillaume Bono, Lconid Antsfeld, Boris Chidlovskii, Philippe Weinzaepfel, and Christian Wolf. End-to-End (Instance)-Image Goal Navigation through Correspondence as an Emergent Phenomenon, In International Conference on Learning Representations (ICLR), 2024a.
Guillaume Bono, Leonid Antsfeld, Assem Sadek, Gianluca Monaci, and Christian Wolf. Learning with a Mole: Transferable Latent Spatial Representations for Navigation without Reconstruction. In International Conference on Learning Representations (ICLR), 2024b.
Guillaume Bono, Hervé Poirier, Leonid Antsfeld, Gianluca Monaci, Boris Chidlovskii, and Christian Wolf. Learning to Navigate Efficiently and Precisely in Real Environments. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024c.
Guillaume Bresson, Zayed Alsayed, Li Yu, and Sébastien Glaser. Simultaneous Localization and Mapping: A Survey of Current. Trends in Autonomous Driving. IEFF Transactions Intelligent. Vehicles, 2(3): 194 220, 2017.
Wenzhe Cai, Siyuan Huang, Guangran Cheng, Yuxing Long, Peng Gao, Changyin Sun, and Hao Dong. Bridging Zero-shot Object Navigation and Foundation Models through Pixel- Guided Navigation Skill. In IEEE International Conference on Robotics and Automation (ICRA), 2024.
Jaime Carbonell and Jade Goldstein. The Use of MMR, Diversity-based Reranking for Reordering Documents and Producing Summaries. In ACM SIGIR Conference on Research and Development in Informalion Retrieval (SIGIR), 1998.
Devendra Singh Chaplot, Dhiraj Gandhi, Abhinav Gupta, and Ruslan Salakhutdinov. Object Goal Navigation using Goal-Oriented Semantic Exploration. In Annual Conference on Neural Information Processing Systems (NeurIPS), 2020a.
Devendra Singh Chaplot, Dhiraj Gandhi, Saurabh Gupta, Abhinav Gupta, and Ruslan Salakhutdinov. Learning To Explore Using Active Neural SLAM. In International Conference Learning Representations (ICLR), 2020b.
Devendra Singh Chaplot, Ruslan Salakhutdinov, Abhinav Gupta, and Saurabh Gupta. Neural Topological SLAM for Visual Navigation. In IEEE/CVF Conference on Compuler Vision and Pallern Recognition (CVPR), 2020c.
Prithvijit Chattopadhyay, Judy Hoffman, Roozbeh Mottaghi, and Aniruddha Kembhavi. RobustNav: Towards Benchmarking Robustness in Embodied Navigation. In International Conference on Computer Vision (ICCV), 2021.
Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. Think Global, Act Local: Dual-scale Graph Transformer for Aibon-and-Languagee Navigation. In IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. Tn International Conference Learning Representations (ICLR), 2023.
14

---

Published in Transactions on Machine Learning Research (11/2025)
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning Phrase Representations using RNN Machine Translation. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014.
for Statistical
Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Kiana Ehsani, Jordi Salvador, Winson Han, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. ProcTHOR: Large-Scale Embodied AI Using Procedural Generation. In Annual Conference on Neural Information Processing Systems (NeurIPS), 2022.
Edsger Wybe Dijkstra. Note on two Problems in Connexion with Graphs. Numerische Mathematik, 1: 269-271, 1956.
Yiming Ding, Carlos Florensa, Pieter Abbeel, and Mariano Phielipp. Goal-conditioned Imitation Learning. In Annual Conference on Neural Information Processing Systems (NeurIPS), 2019.
Matthijs Douze, Alexandr Guzlva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The Faiss library. arXiv:2401.08281, 2024.
Mazaré,
Heming Du, Xin Yu, and Liang Zheng. VTNet: Visual Transformer Network for Object Goal Navigation. In International Conference on Learning Representations (ICLR), 2021.
Jakob Engel, Thomas Schöps, and Daniel Cremers. LSD-SLAM: Largo-scale Direct Monocular SLAM. In European Conference on Computer Vision (ECCV), 2014.
Kuan Fang, Alexander Toshev, Li Fei-Fei, and Silvio Savarese. Scene Memory Transformer for Embodied Agents in Long- Horizon Tasks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Dieter Fox, Wolfram Burgard, and Sebastian Thrun. The Dynamic Window Approach Collision Avoidance. IEFE Robotics & Automation Megazine, 4(1):23-33 33, 1997.
Samir Yitzhak Gadre, Mitchell Wortsman, Gabriel Ilharco, Ludwig Schmidt, and Shuran Song, CLIP on Wheels: Zero-Shot Object Navigation as Object Localization and Exploration. arXiv:2203.10421, 2022.
Samir Yitzhak Gadre, Mitchell Wortsman, Gabriel Ilharco, Ludwig Schmidt, and Shuran Song. CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Anirudh Goyal, Abram L. Friesen, Andrea Banino, Theophane Weber, Nan Rosemary Ke, Adria Puigdomenech Badia, Arthur Guez, Mchdi Mirza, Peter C. Humphreys, Ksenia Konyushkova, Laurent Sifre, Michal Valko, Simon Osindero, Timothy Lillicrp,, Nicolas Heess, and Charles Blundell. Retrieval Augmented Reinforcement Learning. In International Conference on Machine Learning (ICML), 2022.
Steven Hansen, Pablo Spreehmann, Alexander Pritzel, André Barreto, and Charles Blundell. Fast Deep Reinforcement Learning using Online Adjustments from the Past. In Annual Conference on Newral Information Processing Systems (NeurIPS), 2019.
Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefei Cao, Ashish Shah, Abhinav Shrivastava, and Ser- Nam Lim. MA-LMM: Memory-Augmented Large Multimodal Model for Long-Tery Term Video Understanding. In IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recogrittin (CVPR), 2016.
João F. Henriques and Andrea Vedaldi. MapNet: An Allocentric Spatial Memory for Mapping Environments. In IEEE/CVE Conference on Compuler Vision and Pallern Recognation (CVPR), 2018.
Hao Hu, Jianing Ye, Guangxiang Zhu, Zhizhou Ren, and Chongjie Zhang. Generalizable Episodic Memory for Deep Reinforcement Learning. Tn International Conference Machine Learning (ICMT), 2021.
15

---

Published in Transactions on Machine Learning Research (11/2025)
Peter C. Humphreys, Arthur Guez, Olivier Tieleman, Laurent Sifre, Théophane Weber, and Timothy Lillicrap. Large-scale Retrieval for Reinforcement Learning. In Annual Conference on Newral Information Processing Systems (NewrIPS), 2022.
Gabriel Ihaarco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Haunaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. URL https://doi. wttg/....0/////no.d... 5281/22nodo. 5143773
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z. Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement Learning with Unsupervised Auxiliary Tasks. In International Conference on Learning Represenlations (ICLR), 2017.
Eric Jang, Shixiang Gu, and Ben Poolc. Categorical Reparamcterization with Sumbel-Softmax. In Interna- lional Conference on Learning Representations (ICLR), 2017.
Cijo Jose, Moustapha Cissé, and Francois Fleurct. Kronecker Recurrent Units. In International Conference on Machine Learning (ICML), 2018.
Tomoyuki Kagaya, Thong Jing Yuan, Yuxuan Lou, Jayashree Karlckar, Sugiri Pranata, Akira Kinosc, Koki Oguri, and Yang Wick, Felixand You. RAP: Geetrival-Augmeentee Planning with Contextual Memory for Multimodal LLM Agents. In NewrIPS Workshop on Open- World Agents (OWA), 2024.
Mukul Khanna, Ram Ramrakhya, Gunjan Chhablani, Sriram Yenamandra, Theophile Gervet, Malthew Chang, Zsolt Kira, Devendra Singh Chaplot, Dhruv Batra, and Roozbeh Mottaghi. GOAT-Bench: A Benchmark for Multi-Modal Lifelong Navigation. In IEEE/CVF Conference on Computer Vision and Pollern Recognilion (CVPR), 2024.
Nuri Kim, Obin Kwon, Hwiycon Yoo, Yunho Choi, Jeongho Park, and Songhwai Oh. Topological Semantic Graph Memory for Image- Goal Navigation. In Conference on Robot Learning (CoRL), 2023.
Kurt Konolige. A Gradient Method for Realtime Robot Control. In IEEE/RS, RRSJ International Conference on Intelligent Robols and Systems (IROS), 2000.
Jacob Krantz, Theophile Gervet, Karmesh Yadav, Austin Wang, Chris Paxton, Roozbeh Mottaghi, Dhruv Batra, Jitendra Malik, Stefan Lee, and Devendra Singh Chaplot. Navigating to Objects Specified by Images. In IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Yuxuan Kuang, Hai Lin, and Meng Jiang. Opw: Towards Open-Set Zero-Shot Object Navigation via Vision Language Foundation Models. In International Conference on Learning Representations (ICLR), 2024.
Obin Kwon, Jeongho Park, and Songhwai Oh. Renderable neural Radiance Map for Visual Navigation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Mathieu Labbé and François Michaud. RTAB-Map as an Open-Source Lidar and Visual SLAM Library for Large-Scale and Long-Ten Term Online Operation. Journal of Field Robotics, 36(2):416-445 2019.
Zichuan Lin, Tianqi Zhao, Guangwen Yang, and Lintao Zhang. Episodic Memory Deep N-Newwrrks. In International Joint Conference on Artificial Intelligence (IJCAT), 2018.
Steve Macenski, Francisco Martín, Ruffin White, and Jonatan Ginés Clavero. The Marathon 2: A Navigation System. In FEEE/RS] International Conference on Intelligent Robots and Systems (IROS), 2020.
Arjun Majumdar, Gunjan Aggarwal, Bhavika Devnani, Judy Hoffman, and Dhruv Batra. ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddings. In Annual Conference on Neurol Information Processing Systems (NewrIPS), 2022.
Arjun Majumdar, Karmesh Yadav, Sergio Arnaud, Yecheng Jason Ma, Claire Chen, Sneha Silwal, Aryan Jain, Vincent-Pierre Berges, Picter Abboel, Jitendra Malik, Dhruv Batra, Yixin Lin, Oleksandr Maksymets, Aravind Rajeswaran, and Franziska Meier. Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence? Tn ICLR Workshop Reincarnating Reinforcement Learning, 2023.
16

---

Published in Transactions on Machine Learning Research (11/2025)
Pierre Marza, Lactitia Matignon, Olivier Simonin, and Christian Wolf. Teaching agents how to map: Spatial reasoning for multi-object navigation. In IEFE/RSJ International Conference on Intelligent Robols and Systems (IROS), 2022.
Pierre Marza, Laetitia Matignon, Olivier Simonin, and Christian Wolf. Multi-Object Navigation with dynamically learned neural implicit representations. In International Conference on Computer Vision (ICCV), 2023.
Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andy Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukeuoglu, Dharshan Kumaran, and Raia Hadsell. Learning to navigate in complex environments. In International Conference on Learning Representations (ICLR), 2017.
Gianluca Monaci, Philippe Weinzaepfel, and Christian Wolf. What does Really Matter in Image Goal Navigation? arXiv:2507.0166; 2025.
Raul Mur-Artal and Juan D Tardós. ORB-SLAM2: An Open-Source SLAM System for Monocular, Stereo, and RGB-D Cameras. IEEE Transactions on Robotics, 33(():12 1262, 2017.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po-Yao Huang, Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski.
DINOv2: Learning Robust Visual Features without Supervision. Transactions on Machine Learning Research Journal (TMLR), 3(1), 2024.
Emilio Parisotto and Ruslan Salakhutdinov. Neural Map: Structured Memory for Deep Reinforcement Tearning. In International Conference on Learning Representations (ICLR), 2018.
Alexander Pritzel, Benigno Uria, Sriram Srinivasan, Adrià Puigdomènech, Oriol Vinyals, Demis Hassabis, Daan Wierstra, and Charles Blundell. Neural Episodic Control. In International Conference Machine Learning (ICML), 2017.
Qi Qian, Yuanhong Xu, and Juhua Hu. Intra-Modal Proxy Learning for Zero-Shot Visual Categorization with CLIP. In Annual Conference on Neurol Information Processing Systems (NeurIPS), 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), 2021.
Santhosh Kumar Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alexander Clegg, John M Turner, Eric Undersander, Wojeiech Galuba, Andrew Westbury, Angel X Chang, Manolis Savva, Yili Zhao, and Dhruv Batra. Habital--atterport 3D dataset (HM3D): 1000 large-scale 3d environments for embodied AI. In NewrIPS Datasets and Benchmarks Track, 2021.
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gómez Colmenarejo, Alexander Novikov, Gabriel Barth- maron, Mai Giménez, Yury Sulsky, Jackic Kay, Jost Tobias Springenberg, et al. A generalist agent. Transactions on Machine Learning Research Journal (TMLR), 2022.
Christoph Rösmann, Frank Hoffmann, and Torsten Bertram. Timed-Elastic- for time-optimal point-to- point nonlinear model predictive control. In European Control Conference (ECC), 2015.
Nikolay Savinov, Alexey Dosovitskiy, and Vladlen Koltun. Semi carametramttric Topological Memory for Navigation. In International Conference on Learning Representations (ICLR), 2018
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A Platform for Embodied AT Research. Tn International Conference on Computer Vision (ICCV), 2019.
17

---

Published in Transactions on Machine Learning Research (11/2025)
James A Sethian. A Fast Marching Level Set Method for Monotonically Advancing Fronts. Proceedings of the National Academy of Sciences (PNAS), 98(4):1591-159h 1996.
Dhruv Shah and Sergey Levine. ViKiNG: Vision-based Kilometer-scale Navigation with Geographic Hints. In Robotics: Science and Systems (RSS), 2022.
Ajay Sridhar, Dhruv Shah, Catherine Glossop, and Sergey Levine. NoMaD: Goal Masked Diffusion Policies for Navigation and Exploration. In IEEE International Conference on Robotics and Automation (ICRA), 2024.
Xinyu Sun, Peihao Chen, Jugang Fan, Jian Chen, Thomas Li, and Mingkui Tan. FGPrompt: Finc- aine-grained Goal Prompting for Image-goal Navigation. In Annual Conference on Neural Informalion Processing Systems (NewrIPS), 2024a.
Xinyu Sun, Lizhao Liu, Hongyan Zhi, Ronghe Qiu, and Junwei Liang. Prioritized Semantic Learning for Zero-shot Instance Navigation. In European Conference on Computer Vision (ECCV), 2024b.
Sebastian Thrun, Wolfram Burgard, and Dieter Fox. Probabilistic Robotics. MIT Press Cambridge, 2005.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Annual Conference Neural Information Processing Systems (NeurIPS), 2017.
Saim Wani, Shivansh Patel, Unnat Jain, Angel X. Chang, and Manolis Savva. Multion: Benchmarking semantic map memory using multi-object navigation. In Annual Conference on Neural Information Processing Systems (NeurIPS), 2020.
Philippe Weinzaepfel, Vincent Leroy, Thomas Lucas, Romain Brégier, Yohann Cabon, Vaibhav Arora, Leonid Antsfeld, Boris Chidlovski,, Gabricla Csurka, and Jérôme Revaud. CroCo: Self-Supervised Pretraining for 3D Vision Tasks by Cross View Completion. In Annual Conference on Neural Information Processing Systems (NeurIPS), 2022.
Erik Wijmans, Abhishek Kadian, Ari Morcos, Stefan Lee, Trfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra. DD-PPO: Learning Near-Perfect GointGoal Navigators from 2.5 Billion Frames. In International Conference on Learning Representations (ICLR), 2019.
Rey Reza Wiyatno, Anqi Xu, and Liam Paull. Lifelong Topological Visual Navigation. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2022.
Fei Xia, Amir R. Zamir, Zhiyang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson Env: real-world perception for embodicd agents. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018
Quanting Xie, So Ycon Min, Tianyi Zhang, Aarav Bajaj. Ruslan Salakhutdinov, Matthew Johnson-Roberson, and Yonatan Bisk. Embodied-RAG:: General non-parametric Embodied Memory for Retrieval and Genera- tion. arXiv:2409.18313, 2024.
Karmesh Yadav, Arjun Majumdar, Ram Ramrakhya, Naoki Yokoyama, Alexei Baevski, Zsolt Kira, Oleksandr Maksymets, and Dhruv Batra. OVRL-V2: A simple Statc-of-art Bascline for ImageNav and ObjectNav. arXiv:2303.07798, 2023a.
Karmesh Yadav, Ram Ramrakhya, Arjun Majumdar, Vincent-Pierre Berges, Sachit Kuhar, Dhruv Batra, Alexei Baevski, and Oleksandr Maksymets. Offline Visual Representation Learning for Embodied Navigation. In ICLR Workshop on Reincarnating Reinforcement Learning (RRL), 2023b.
Karmesh Yadav, Ram Ramrakhya, Santhosh Kumar Ramakrishhan, Theo Gervet, John Turner, Aaron Gokaslan, Noah Maestre, Angel Xuan Chang, Dhruv Batra, Manolis Savva, Alexander William Clegg, and Devendra Singh Chaplot. Habitat-Matterpor 3D Semantics Dataset. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023c.
18

---

Published in Transactions on Machine Learning Research (11/2025)
Hang Yin, Xiuwei Xu, Zhenyu Wu, Jie Zhou, and Jiwen Lu. SG-Nav: Online 3D Scene Graph Prompting for LLM--ased Zero-shot Object Navigation. In Anmual Conference on Newral Information Processing Systems (NeurIPS), 2024.
Bangguo Yu, Hamidreza Kasaei, and Ming Cao. L3MVN: Leveraging Large Language Models for Visual Target Navigation. In IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), 2023.
Kuo-Hao Zeng, Zichen Zhang, Kiana Ehsani, Rose Hendrix, Jordi Salvador, Alvaro Herrasti, Ross Girshick, Aniruddha Kembhavi, and Luca Weihs. PoliFormer: Scaling On-Policy RL with Transformers Results in Masterful Navigators. In Conference Robot Learning (CoRL), 2025.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid Loss for Language Image Pre- Pre-Training. In International Conference on Computer Vision (ICCV), 2023.
Kaiwen Zhou, Kaizhi Zheng, Connor Pryor, Yilin Shen, Hongxia Jin, Lise Getoor, and Xin Eric Wang. ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigation. In International Conference on Learning Representations (ICLR), 2023.
19

---

Published in Transactions on Machine Learning Research (11/2025)
A Retrieval dataset creation
We use the Habitat simulator, platform and task definitions from Savva et al. (0019) For Image.Nav, we use the Gibson dataset (Xia et al. 2018), consisting of 72 train and 14 eval scenes, and the ImageNav agent configuration: height 1.5m,, radius 0. 0.1m, 112x112 RGB camera with 90 horizontal field-of-view (hfov) at 1.25m from the ground. To train and test the ObjectNavv variant in Section 6 we use 80 train and 20 eval scenes of the HM3DSem-v0.22 @Rmaakishnan et al. 2021 Yadav et al., 2023ch as in Cai al. (2024) and agent of height 0.88m, radius 0.18m with x12x 112 RGB sensor with 79 hfov placed at 0.88m from the ground.
For each scene, we generate a retrieval dataset D usually containing 000 FPV images as follows:
(i) we randomly choose start and goal viewpoints such that the goal is navigable from the start,
(ii) we make an agent follow the shortest path from start to goal, and
(iii) after each action, we save the agent's RGB obscrvation as an image in D.
We repeat this process until the dataset contains the desired number of images. We randomly choose the distribution of starts and goals such that the agent roughly covers all navigable areas, and that the path between pairs of viewpoints are not too close to one another to simulate a prior offlinc exploration of the scene.
B RANa agent architecture
In this section we detail some of the architecture choices made for our method.
Observation encoder processes the RGB input image x, E 1xx112x112 It is implemented with a half-width ResNet-18 (He et al. 2016) and generates an output feature & E R512
compares the observation X, to the goal g- For ImageNav we use the frozen binocular encoder of DEBiT (Bono et al. 2024a), based on the Geometric Foundation Model (FM) CroCo (Weinzaepfel et al.) 2022). This encoder is a Siamese encoder E applied decoder D which combines the output of the two encoders. Both E and D are implemented with a ViT architecture with self-attention layers, and D also adds cross-attention. The output of the decoder D is further compressed by fully connected layer (frozen, from the DEBiT model) that projects the flattened output of D into g. E R3336 For details about the binocular encoder we refer to Bono et al. (024a) and Weinzaepfel et al. (2022)
Goal comparison function
both x, and g, and a
In ObjectNav we use OpenCLIP (Tharco et al. 2021 to encode the visual observation x, and the textual goal g. This process produces two features ely (x,) the output g. E 1024
R2512 and clr(g) E R2522 which are concatenated to generate
Action embedding- encodes the previous action a into 32- 32-dimensional feature vector å and it is taken from the DEBiT (Bono et al. 2024a) model. For ImageNan, we load DEBiT action encoder and freeze it during training, while for ObjectNav, we trained it from scratch, since the two tasks do not share a common action space. Indeed, the task definition of ObjectNav requires two additional actions, LOOK UP and LOOK DOwN.
Gumbel context encoder Variant of the trainable context encoder (Section 3.1 selects one element. & among the context features J based on a learned relevance estimate {an The retrieved context images are compared to the goal image (if available) and to the observation by the same frozen binocular encoder of DEBIT described above, leading to the N context features (et, each of dimension DD, where D = 6272 for ImageNav or Mestance-ImageNNe ObjecINav. A fully connected layer Linear(D, (using a synlax inspired by PyTorch) estimates the relevance {an)N for each context element and the item with the largest relevance is selected using the Gumbel soft-max sampler [Jang et al. 2017). In the ImageNav or Instance- Instance-ImogeNau cases, only the binocular feature representing the comparison between observation and context items is selected, so in all cases the context. feature é, R3136
(when goal image is available), and D = 3136 for
20

---

Published in Transactions on Machine Learning Research (11/2025)
Figure 9: Examples of shortest paths on DINOv2-Graph (SWG) and on Pose Graph (PS) The most. similar image rt D to the goal is highlighted in red and localized with red blue, is the most similar image to the current observation. Green crosses connects images along the shortest-path in the DINOv2- Graph (corresponding images highlighted green on top of the map) and purple crosses refer poses on the Pose Graph, with corresponding images shown with purple contour at the bottom of the map.
the floorplan, while of D, in
Attention-basel context encoder Variant 2 of the trainable context encoder creates context feature by computing the self-attention over the context elements (e computed as described in the previous paragraph. In this case the context encoder first reduces the dimensionality of each context feature to IR 128 through a fully connected layer Linear(D, 128) (D = 6272 for ImageNav or Instance-ImageNaa, and D 3136 for ObjectNavv), then processes them with standard two-layer transformer encoder network (Vaswani el. al,, 2017) with attention heads and foodforward dimension 256. The N context features are concatenated into feature vector of size RNx12 and processed by 2-layer MLP(128, 3136) with hidden dimension 128 and output 3136, such that the context encoder output è; has the same dimension the Gumbel context encoder.
Recurrent memory The flatten vectors X,, E.. å and é; are concatenated and fed to a single-layer GRU Cho et al. 2014) with hidden state h, E R512 This GRU is an "extended" version of the baseline agent's GRU without context, which is finetuned. However, since RANa concatenates the context feature č; the other inputs, we cannot directly finetune the GRU as is. We modify the GRU of the base model by padding the input weight matricos of the first layer, W W W70 (and corresponding bias vectors 6.. 62 b9,) along the input dimension with random values. The remaining GRU matrices initialized to the values of the baseline checkpoints and finetuned from there.
Policy The hidden state h, is fed to linear policy composed of a linear Actor head that generates a softmax distribution over actions and a linear Crilic head that evaluates the current state.
C DINOv2-Graph
In this section we provide details about the construction of the DINOv2-Graph context. The graph is built from the affinity matrix Ω2 containing the similarity between DINOv2 (Oquab et al.] (024] features of all pairs of images in D, where the nodes are images xP E and edges are weighted by similarity. We use Dijkstra's algorithm (Dijkstra 1956) (from the SciPy python package) select the shortest path between two nodes.
21

---

Published in Transactions on Machine Learning Research (11/2025)
SWG
SBG
DWG
PG
RANa-b (Gumbel)
89.7 (60.4) 89.6 (60.5) 88.7 (60. 7)
89.7 (60.7)
RANa-b (Attention)
89.8 (71.0) 89.6 (70.9) 90.7 (71.8)
91.9 (69.8)
RANa-t (Attention)
84.7 (63. 7) 83.5 (61.1) 1 86.8 (64.8)
87.4 (62.2)
Impact of the graph structure on RANa with DINOv2-Graph SR (SPL) for ImageNav with retrieval using different graph structures at test time: Sparse Weighted Graph (SWG), Sparse Binary Graph (SBG), Dense Weighted Graph (DWG) and Pose Graph (PG). All RANa models were trained with the same SWG graphs built on the training scenes. bolded best results. Note that the PG variant is not considered as it uses the camera poses of images in D; we only add it for comparison.
Table 10: M
Selecting a path on this semantic graph (DiNOv2-Graph) is not necessarily equivalent to a "physical path" (see examples in Fig. 9 In order to enforce selection of edges between images that have high overlap probability (i.e. represent the same part of the scene) we made two further modifications to the above graph. First, as we obscrved that a DINOv2 similarity score above th = 0.75 between L,2 normalized features reflects, in most cases, & good overlap between the two images, we only kept edges for which the similarity was above this threshold, removing all the others. Second, in order to enforce Dijkstra's 's algorithm to select paths with high similarity (low cost), we used the weights Wij /(1 $iz), where 8tj is the DINOv2 similarity between the L2 normalized features.
During both training and inference, at each step, the agent finds in D the most similar images r and of to the observation x, and goal g, respectively, and computes the shortest path between rÇc and rf on the DINOv2 Graph. Let P {rT", r2,, Fn. rf] be the set of nodes on the shortest path and C' the context size. If + 2 > C, we sample images randomly from P and update all images in the context. If n + 2 < C, we use all images in P update n + 2 images in the context. If no path was found, = 2 hence we only update two elements of the context with rf and r..
A Analyzing the choice of the graph structure - We trained our models on the Gibson dataset where we had around 10k images per scene in the retrieval dataset D. We selected the checkpoint obtained at 130M steps for all models. During testing we used set of [D] = 1k images per scene. In this section we study how the graph structure affects performance. The default model, used for the results reported in Table _ of the main paper, builds weighted graph by removing edges with similarity below th = 0.75 and uses Wij- ((1- (51-8,, 8tz) as weights. We call this Sparse Weighted Graph (SWG). We evaluate three additional strategies to construct the LANOv2-Graph. (i) We consider the same graph, but instead of using Way = (11 we use Way = 1 as edge weights, and we call this model Sparse Binary Graph (SBG). (ii) We also consider a similarity threshold th = 0.4 which yiclds a much denser graph, called Dense Weighted Graph (DWG). (iii) Finally, we compare these results with "pose graph", where instead of using DINOv2 features, we use distances between camera poses to build the affinity map. In this case we consider a binary graph where we cut edges for which the distance between the camera poses is above Im. We denote the Pose Graph with PG.
From the results shown in Table 10 we can draw the following observations. First, using a pose aware graph (PG) does not necessarily improve the accuracy over using DINOv2-Graph, and when it does, the improvement is rather small. Note however that the pose of the observation and goal images were not used to select the start and end points of the shortest path, rxr and r these were retrieved using DINOv2 feature similarity and we only replaced the DINOv2-Graph with the Pose Graph. The second observation we can make is that the graph structure does not have big impact the model with Gumbel selector, but a bigger one on the model with attention- based context encoder. This is not surprising as the Gumbel selects an image from the context (path) and use it independently from other context clements, while Attention combines information from all images within the context (path), hence it is more important how we fill up the context. Finally, we observe that the Dense Weighted Graph (DWG) performs better than the Sparse Weighted Graph (SWG). We conjecture that the reason that the model was trained with D of size 10k images and we rarely had no connection between two requested nodes (r rf), which occurs more often at test when only 1 images are available. Using a denser graph (DWG) appears to help overcoming these cases.
22

---

Published in Transactions on Machine Learning Research (11/2025)
context size
20
24
4 80.8 (52.3) 80.5 (51.6) 80.4 (52. 7) 81.0 (52.3) 80.9
(52.0)
81.7 (54.0)
8 80.4
(49.8) 83.0 (52.9) 80.6 (50.6) 84.0 (53.0) 82.0 (59.2) 2) 82.9 (53.4)
12 81.9 (50.0) 80.0 (49.2) 81.9 (51.2) 80.6 (50.6) 81.6 (50.8) 82.4 (52.2)
Table 11: P Impact of context size for RANa-t SR (SPL) for ImageNav with retrieval over different context sizes, combinations of train and test conditions.
context size
12
16
20
24
51.6 (26.1) 50.8
50.88 (25.
49.8
(25.8) 49.8 (25.8) 49.0
(24. 50.6
(25.8)
50.5 (25.9) 51.9 (26.4)
52.1 (26.9)
50.3 (26.0 6)
51.6 (26.5)
51.1 (26.6)
51.4 (26.4) 51.4 (26.9) 50.9 (26.7) 51.2 (27.3) 50.9 (26.7) 51.6 (27.4)
Table 12: P Impact of context size for RANa-t (SPL) for ObjectNav with retrieval over different context. sizes, combinations of train and test conditions.
Similarities and differences with topological maps
The proposed graph representation is related
to topological map structures (Savinov et al. 2018 Beeching et al. 2020b Chaplot et al. 2020c Wiyatno
jel al. 2022; Kim et al., 2023 Sridhar el al. (2024). However, there are several critical differences between our model and traditional topological map based methods. First, we do not use the topological map to localize the agent, plan, and act to reach the "next waypoint".. Instcad, the information gathered from planning is passed to the agent as a "recommendationn in the form of the retrieval context, to be integrated into decision making through learning: our agent is a full fledged policy which has access to memory, goal and a series of images along the path, which arguably enables more informed decision- decision-making, making, and improved robustness and navigation performance. Additionally, our graph is connected based only on visual similarity rather than predictions from temporal distances. While it is debatable whether this approach is inherently superior, it offers a distinct advantage: we do not require trajectories build the graph but we can also build it from a large number of isolated images. The main drawback of our method is that we rely on the assumption that visual proximity is proxy for spatial proximity, which is not always true (see Fig. 9h. Future work could attempt to learn to take into account the uncertainty in this assumption through data-driven planning, as in Beeching et al. (2020b).
D Ablation experiments
In this section, provide additional ablation sludies that could not fit into the main document. As for the ablation experiments in the main paper, all experiments are conducted on RANa-t models.
Impact of context size - studied in Table 11 for ImageNav (Bx) and in Table 12 for ObjectNas (о
with slatic context. One interesting advantage the Gumbel selector that allows for context different. sizes during evaluation, independent of the context size used during training. With that said, this parameter has relatively minor impact on performance, with larger training context sizes showing positive performance trends. The size N = 8 is choice that works well both for ImageNav and ObjectNev and balances navigation performance with execution time. For ObjectNav, we observe more pronounced performance increase for larger context sizes during evaluation, so we use N = 8 during training and N = 12 during evaluation.
Impact of context diversity for Gumbel context encoder Table 13 evaluates the behavior of RANa-t ImageNav with static context, when using different strategies to create the context, (i) top 8 MMR: most similar database images filtered by MMR as in Section 4.1 (ii) top 8: most similar database images to the goal in DINOv2 feature space, and (iii) top 1: the closest element from the retrieval dataset.
Filtering the context with MMR is important, the agent appears to rely on diverse context items; removing MMR degrades performance. top 1 always uses the best element from the context, which is very similar to the goal image. In this case performance go back to baseline levels, confirming the importance of diversity in the retrieved elements.
Impact of retrieval database size on zero-shot Instance-ImageNNu (s- H is shown on Table 14 Here the goal image R is retrieved from the database, and thus retrieval accuracy is crucial for success and
23

---

Published in Transactions on Machine Learning Research (11/2025)
top 8 MMR
top 8
top 1
top 8 MMR
83.0 (52.3)
81.5 (51.0)
79.6 (48.7) 7)
top 8
80.7 (51.1)
81.4 (48.1)
80.2 (51.2)
Table 13: Impact of MMR filtering on retrieved context SR (SPL) for ImageNav with and without MMR, combinations of train and test conditions ( (in-domain performance shaded in pink ). Context diversity from MMR is important and delivers the best performance (SR=83.0). When the model trained with MMR is tested without it, performance drops.
DB size
1k
6k
10k
20k
50k
100k
SR (SPL)
47.6 (20.2) 56.2
(24.4) 56.4
(23.8)
57.4
(25.0) 58.7 (26.8) 59.1 (25.9)
Table 14: s-D Impact of retrieval database size in zero-shot SR. (SPI. in Instance- Instanoe-Image Tmggeaangaaainonnnnnnnnnn increases with dataset size.
avoid terminating an episode too far from the actual goal. Table 14 shows this phenomenon, with zero-shot Instance-Inun ImaageNe performance increasing with retrieval dataset size. However, please note that RANa achieves navigation performance in line with SOTA using retrieval dataset containing only 5, 000 images.
A Analysis of "oracle" context for ImageNav ( is carried out in Table 15. In this experiment we study which information is most useful and can be exploited by RANa, by building contexts using privileged information in simulation therefore in these cases the retricval dataset D is not used. Oracle captures a panoramic view of the goal location and is created by collecting N context images at the goal position while rotating the camera view by 45 clock-wiss wisee for each element. This context represents a soft upper bound for the static context in Section 4.1 where fixed set of items is chosen per episode based on their similarity with the goal image. Oracle 2 context is richer and it is generated by computing the shortest path from the agent position to the goal at each step and collecting views along this path. This is soft upper bound for the DINOv2-Graph dynamic context (Section 4.2) which is dynamically generated at each step on the context graph. Table 15 shows that, as expected, both types of oracle context can provide large gains, especially in SPL, é.e. navigation efficiency. It suggests that there little room for improvement in success (SR) using static goal context, while more structured information, e.g. paths, derived from data in D can improve navigation considerably.
Impact of using pre-trained GRU weights As mentioned earlier all RANa agents GRU finetuned from the basclinc agent's GRU. We evaluate the impact of this choice by training the RANa-t static GRU from scratch while keeping all other settings fixed. Table 16 shows that finetuning the GRU form pretrained weights those of DEBiT (Bono et al. 2024ah in this case improves navigation performance. We also observe faster convergence during training, which might in part explain the performance increase.
Context selection criteria All ImageNav experiments reported in the paper use a feature vector for each context. element which is obtained by concatenating two features: (i) the output of the Geometric FM g comparing context element and observation, and (ii) the output of g comparing context element and goal. We evaluate the impact of this choice by training a RANa-t static model which selects context elements based only on feature (i), which compares context elements to the observation. The performance of this variant and that of the bascline which uses both observation and goal, is displayed in Table 17
It shows that information
Context
Type
SR
SPL
DINOv2 goal
static
83.0
52.3
Panorama (oracle 1)
static
83.4
71.6
HINOv2-graph
dynamic
84.7
63.7
Shorlest. path (oracle 2)
dynamic
98.3 87.9
Table 15: D Navigation with "oracle" context SR (SPL) for RANa-t using privileged information to build oracle context.
24

---

Published in Transactions on Machine Learning Research (11/2025)
Model
SR.
SPL
GRU finetuned
83.0
52.3
GRU from scratch
81.4 51.0
Table 16: D+ Impact of GRU finetuning SR. (SPL) for RANa-t with slatic conlext and GRU finetuned
trained from scratch.
Model
SR.
SPL
context selection based on obs and goal
83.0
52.3
context sclection based on obs only
81.0
50.3
Table 17: D+ Impact of context selection SR (SPI.) for RANa-t with static context and elements selected based comparisons with only the observation, both goal and observation.
provided by comparing the context to the goal is useful and can be exploited by RANa to navigate more successfully and efficiently to the goal.
Context building for ObjectNavv - Table 18 shows the bohavior of RANa-t ObjectNav (+ when adopting
different context building approaches, (i) rand: 8 random elements from the retrieval dataset, (ii) top cls: most similar database images to the object goal encoder in CLIP feature space using CLIP similarity as metric, and (iii) top wo: most similar database images using CLIP score softmar normalization. While CLIP allows to easily compare any open-vocabulary object category with retrieval gallery images, it has been observed in the context of wero-shot image classification that directly ranking images by similarity to the category is suboptimal (Qian et al. (2023). To address this issue for ObjectNav, we leverage the fact that we often target a pre-defined set of classes O (e.g. 6 in HM3D), and rescale the score of each image D via softmax:
wx(x) = edl((ox)) /
where cls(o, x) denotes the CLIP similarity of image and object category o and (x)x) its normalized similarity.
The observed behavior is similar to ImageNon. First, RANa-t model trained with context with randomly selected images from the scene gallery performs worse than a model trained with either top 8 images. Nevertheless, this model still outperforms our baseline trained without retrieval (SR=42.9 and SPL-23.0 for the non-retrieval baseline) and is more robust to random context images during evaluation. For RANa-t models trained with retrieval top 8 context, we observe improved performance when using softmax normalized scores.
Impact of retrieval dataset size for ObjectNav is shown in Table 19 Results demonstrate that unlike ImageNav (Table 5). ObjectNav seems to need larger retrieval datasets both at training and at testing. Performance degrades significantly when the datasct D at training only has 1, 000 images. At test, using larger retrieval datasets leads to better performance, with metrics degrading significantly with D containing 100 items.
Impact of CLIP variants for ObjectNav is evaluated in Table 20 In this experiment, we build retricval datasets D of size 10k images for each of the 100 HM3DSem-v0.22 scenes used for ObjectNav (80 train, 20 test). For each of the 6 target object categories (chair, bed, plant, toilet, tu monitor, sofa) and for each
rand
top cls
top 8 Wo
rand
46.8 (23.8)
47.7 (25.0)
46.1 (23.1)
lop 8 ols
44.3 (23.0)
48.0 (25.4)
48.5 (26.0)
top Wo
44.4 (21.4)
51.3 (25.9)
51.9 (26.4)
Table 18: P Impact of retrieval data selection SR (SPL) for RANa-t ObjectNav with different context building strategies, combinations of train and test conditions.
25

---

Published in Transactions on Machine Learning Research (11/2025)
DB size
100
1k
10k
50k
1k
43.8
(21.7)
45.3
(22.4)
46.8
(23.2)
47.1
(23.2)
10k
46.6 (24.9) 50.3 (26.4)
52.1
(26.9)
51.8 (27.2)
Table 19: p
Impact of retrieval database size SR (SPL) for RANa-t ObjectNav with retrieval using different database sizes, combinations of train and test conditions.
Vision-language model
softmax
top
top
8 top 20
OpenCLIP (Iharcc et al. 2021 X
40.6
76.0
88.5
OpenCLIP (Ilharco al. 2021
/
78.1
95.8
97.9
CLIP (Radford et al. 2021
/
67.7
88.5
90.6
SigLIP (Zhai et al. 2023
/ 70.8
91.7
94.8
Table 20: Retrieval performance of CLIP variants. Retrieval considered success if the retrieved image is of the correct category and if it is closer than 1m to the target object.
scene, we do search in the retrieval dataset D by querying the CLIP-encoded goal category (t.e. the CLIP text encoder for the sentence "a picture of a calegory inside a house") and comparing it with CLIP encoded dataset images of the scene, providing top 1, top 8 and top 20 success percentages. retrieval is considered successful if the retrieved image (whose ground-truth pose known); closer to one of the viewpoints of a object instance of the correct category than the success threshold of the ObjectNavv task, í.e. 1m. The Table shows that OpenCLIP performs better than alternative CLIP models, namely OpenAl's original CLIP weights (Radford et al.) 2021) and SigLIP (Zhai et al. 2023), all with the softmax normalization being used. As observed carlier, ranking images by CLIP similarity to the category does not perform well (Qian et al. 2023 since the number of (arget classes is known (6 in all our experiments), rescaling the score of each image in 'D via softmax significantly boosts performance.
E
Qualitative analysis
E.1
Examples of retrieved context for ObjectNav
Fig. 10 displays examples context for RANa ObjectNav (= using similarity in CLIP feature space for the six different goal categories, chair, bed, plant, sofa, to monitor and toilet, Results accurately represent instances of the target object present in the scene, with the exception of the monitor class where we observe some errors, especially in scenes where monitors are absent.
E.2 Navigation examples
In Fig. 11 show examples of our agent RANa-b static navigating to two challenging ImageNav goals. The figures visualize the observation (left), goal (highlighted in green), N 8 context elements, with the one selected at the given time step highlighted in red, and the agent path drawn on the scene floorplan on the right. Each row of pictures focuses on one time step along the episode and are ordered in time from top to bottom. In this figure and the following ones, time is not sampled uniformly: we selected time steps, typically towards the end of the episode, where the agent makes interesting context selections to solve the episode.
The episode in Fig, II (left) is challenging because the environment is rather complex, the goal image is ambiguous and it is "hidden" in terrace at the extreme perifery of the navigable space. The baseline DEBiT-b agent, without context, does its best to explore the apartment and gets close to the terrace door where the goal is located, but docs not manage to navigate to it. RANa-b selects from the context an image (highlighted in red) that contains the railing of the terrace that helps our agent solve this complex navigation episode. The navigation episode on the right complex for similar reasons: large complex environment and goal image with very little information. RANa capable of exploiting the few relevant context items, by first selecting the context image containing the door the room where the goal located, and then the cabinet, whose corner is visible on the goal image.
26

---

Published in Transactions on Machine Learning Research (11/2025)
Goal
Context
"chair"
"bed"
"plant"
"sofa" "
"TV"
"toilet"
Figure 10:
Examples of context for different ObjectNav categories.
Oiswvalion
Gowl
Contac
Agent pah
Ossevvatio
Gowl
Connect
Agent path
Faluxe
DEBIT bosoline contenl
Faluur
DEBI's bazeline no comtea!
Figure 11: + RANa-b successfully navigates to challenging goals in two ImageNav episodes The agent observation is the left, next it is the goal image highlighted green, then there N context elements, with the one selected by the RANa agent at the current time step highlighted in red, and the right the agent path drawn on the floorplan the scene, with the goal position indicated by red square. At the bottom show the trajectory of the baseline DEBiT-b agent on the same episode.
27

---

Published in Transactions on Machine Learning Research (11/2025)
Agent path
Agont
Sucooss
ME
FATUDa
no baseline
E
no
HR RANa-b successfully navigating to two goals in zero-shot The agent observation is on the left, next to is the original goal image from the Instance- Iustance-ImageNeee Imageear task (in cyan) and the retrieved goal image g" highlighted in green. The remaining data is organized and visualized as Fig.
Figure 12:
ImageNav episodes
Observation
Agent
Observation
Agent
Time
DEBIT-b baseline
DEBIT-b baseline no context
no context
RANa-b successfully navigates real-world goals in two zero-shot Instance- Instance-ImageNNrt episodes- In this case the original image goal (highlighted in cyan) is taken with a smariphone in a real environment. Again we retrieve goal image g" from D and the agent navigates 3D model of the environment.
Figure 13: -
Fig. 12 shows RANa-b ImageNav agent successfully navigating in two zero-shot Instance-ImageNie episodes. Results are visualized in the same way as in Fig. [1] with the exception of the goal image. In this case we have fact two goal images, one from the original task (highlighted with different pose and intrinsics of the agent sensor, and the goal image g highlighted in green, which is retrieved from the scene database R. In both cases, RANa selects context items that represent elements visible in our agent observation (the toilet on the left and the door on the right) that guide the agent inside the room where the goal is located. Interestingly, in both cases the baseline agent passed along the same path, but did not enter the room, arguably because did not have access to the context.
cyan) which taken with sensor
In Fig. we present an even more challenging scenario, where RANa navigates in two zero-shot Instance- ImageNov episodes, where the goal image captured with a smartphone in real environment. As mentioned in the main text, assume having access to this environment, and that 3D model of the scene is available.
28

---

Published in Transactions on Machine Learning Research (11/2025)
The agent navigates to the retrieved goal image g from the database D in the simulated reconstruction of the scene. The example on the left is particularly interesting because the retrieved goal image, in green, is not the correct instance of the original goal. However, guided by several context items that contain instances of the original goal image, RANa succeeds in reaching the goal. In the navigation episode on the right, our agent picks relevant context elements containing the chairs of the meeting room where the goal positioned, and enters the room solving the navigation episode.
29


Reviews:
[{"review_id": "WpmCqMFWsp", "reviewer": "Reviewer_8vmt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a retrieval design for augmenting navigation agent. The agent is trained via RL to solve navigation tasks, the paper proposes to augment the agent's context information with database retrieval. The paper presents solid technical design and empirical results, showing the promise of the approach.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper's writing is relatively easy to follow. The paper lays out details on the technical designs of the retrieval systems, and how it gets integrated into a canonical RNN based navigation agent. The paper also carries out fairly extensive empirical study that showcases the gains the retrieval entails when trained in the navigation environment.", "audience_yn": "Yes", "audience_explanation": "The paper presents findings in the utility of retrieval mechanism for RL training, which definitely can be of interest to certain TMLR readers.", "requested_changes": "I have a few technical questions regarding the paper.\n\n=== ***static vs. dynamic context*** ===\n\nExamining results in Table 2, it seems that the static context brings about most of the gains while the dynamic context only improves slightly on top. Does it mean that for the sake of simplicity, it suffices to use static retrieval for augmented context, to gain most of the performance improvements? I wonder why dynamic context would not obtain further gains given the additional amount of flexibility it entails.\n\n=== ***RL training curves*** ===\n\nSince the paper is centered on training RL with augmented input via retrieval, I think it's valuable to put RL training curves into the main paper. All results in the paper show the final performance of the trained system, and hence does not reflect how augmented input helps with the performance. For example, do we expect speed improvement in learning? Do we expect better asymptotic performance during training? Are hyper-parameters more robust for the training curves due to retrieval?\n\n=== ***Training retrieval process*** ===\n\nI wonder if there is potential for training retrieval process itself, in addition to the encoder network that encodes the retrieved inputs? It seems that most of the prior work on this fixes the retrieval process, so does this current work. Just wonder if the authors have any thoughts regarding the challenges and potential with this approach.\n\n=== ***Transformer architecture vs. rnn*** ===\n\nWhile experiments with recurrent networks are certainly technical solid, as done in this work, I wonder whether the authors have considered the impact of retrieval on architectures, e.g., how it impacts the system of the base agent is transformer. An argument is that the recurrent architecture might be bottlenecked by the capacity of the hidden state, and hence retrieved inputs allow for improvements. However, when using transformer, the hidden state bottleneck is alleviated and I wonder if the gains entailed by retrieval will diminish as a result.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "h4xYt5kgdM", "reviewer": "Reviewer_ufQa", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The majority of the existing works consider robots to be new to the environment, without taking the prior observations and interactions into account. This work proposes retrieval-augmented navigation (RANa) to allow an agent to store and leverage visual observations from a retrieval database during operation. Retrieval-augmented agent involves a retrieval process which will retrieve the relevant context and the agent will make the next move based on the given context. The experimental results on ImageNav, Instance-ImageNav and ObjectNav demonstrate that RANa reaches new state-of-the-art results with a relatively fast inference speed.", "claims_and_evidence_yn": "Yes", "claims_explanation": "**Strengths**\n1. The use of prior visual observations is smart and interesting.\n2. The retrieval database, though small (i.e., 1K), brings significant performance improvements on ImageNav, Instance-ImageNav and ObjectNav.\n3. The model can reach impressive zero-shot navigation performance.\n\n\n**Weaknesses**\n1. The success of RANa is highly dependent on the retrieval database. Can the model automatically update or add new visual observations into the retrieval database?\n2. The database is still very small, and has limited scalability (Table 5). I understand that this may be attributed to limited context variety in the same environment. However, enlarging the retrieval database may be helpful for generalizability. Can the authors show the impact of retrieval database size on other datasets? If it does not bring improvement, then my question is: why doesn't a larger retrieval database bring further improvement?\n3. What is the impact on inference speed with different sizes of retrieval database. Based on Table 5, the inference speed is significantly slower due to the DINOv2 computation (dynamic variant), and the affinity matrix is a significant computation cost (especially if a larger retrieval database is needed). I am also confused why DINOv2 is a computation-heavy component, given the elaboration on the last sentence in “Retrieval” subsection on Page 7.", "audience_yn": "Yes", "audience_explanation": "See above.", "requested_changes": "1. More clarification is needed for “store” in Table 1", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tS8hRckizS", "reviewer": "Reviewer_mTWV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary of Contributions:**\n\nThis paper introduces a retrieval-augmented navigation (RANa) agent which can perform navigation tasks better by querying a database of previous observations and using it as additional contextual information. \n\nThe authors turn the simple idea of using a retrieval database into something powerful by storing observations as simple raw first person view (FPV) images and then using foundational models to encode that data. \n\nThe authors then introduce a set of benchmarks that provides a database of observations to evaluate retrieval-augmented agents. These benchmarks are based off standard Habitat Simulator tasks. \n\nThe authors also show that the RANa agent performs significantly better than other baseline models and also has zero-shot generalization capabilities.\n\n**Strengths:** \n\n- The paper is well-written and easy to follow\n- The paper takes the simple idea of having a retrieval database and makes it into something powerful by making use of modern foundational models\n- The authors also show that the database can be shared across agents, and is scalable to very large number of images\n- Simplicity of the method: The authors store simple, raw images into the database without any complicated metadata or pose information and show that it can still work effectively by leveraging foundational models and a smart retrieval mechanism. This aligns well with the modern deep learning paradigm where large-scale raw data can be converted into useful representations.\n\n**Weaknesses:** \n\n- In the case of RANa agent, the performance of the agent will be closely tied to information obtained through the database. So it is very likely that there could be failure cases which cause the agent to go astray. The authors do briefly discuss about a failure case in Fig 12 in the Appendix but I feel that there should be more discussions on this.\n- The retrieval basically works on visual similarity so one possible failure case could be if there are images in the database which are visually very similar but not relevant to the goal task. Wouldn’t the agent be led astray by the “additional context” in this scenario?\n- One other interesting experiment to see would be the impact of “stale memory” in the agent’s database. For example, once a large enough samples for the database are collected and then the positions of the items in the simulator are modified slightly will the agent be led astray by the stale images in the database or is the agent robust enough to find the still find the goal. The authors do touch upon this a bit in the “microwave experiment” under the sim-to-real section but similar experiments could be directly performed in the simulator just to specifically test the robustness of the agent.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The authors clearly support their claims but conducting extensive empirical experiments and presenting their results in the paper in section 5.", "audience_yn": "Yes", "audience_explanation": "This paper introduces a unique method of foundational model based retrieval mechanism to significantly improve the performance of RL agents for the navigation task, so in my opinion it has novelty, strong results, and thus many people in TMLR's audience should be interested in this.", "requested_changes": "I feel that the paper is already strong enough to be accepted at TMLR, but it would be great if the authors could add a bit more discussion related to failure cases and robustness as mentioned in the weaknesses section.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)