EfficientNav: Towards On-Device Object-Goal Navigation with Navigation Map Caching and Retrieval
Authors: Zebin Yang
Sunjian Zheng
Tong Xie
Tianshi Xu
Bo Yu
Fan Wang
Jie Tang
Shaoshan Liu
Meng Li
Venue: NeurIPS 2025 poster ()
Keywords: Object-Goal Navigation
Navigation Map
Memory Caching
Memory Retrieval
Primary area: deep_learning
TL;DR: We propose EfficientNav, enabling on-device memory-augmented object-goal navigation system for zero-shot in-door navigation.

Abstract: Object-goal navigation (ObjNav) tasks an agent with navigating to the location of a specific object in an unseen environment. 
Embodied agents equipped with large language models (LLMs) and online constructed navigation maps can perform ObjNav in a zero-shot manner. However, existing agents heavily rely on giant LLMs on the cloud, e.g., GPT-4, while directly switching to small LLMs, e.g., LLaMA3.2-11b, suffer from significant success rate drops due to limited model capacity for understanding complex navigation maps, which prevents deploying ObjNav on local devices.
At the same time, the long prompt introduced by the navigation map description will cause high planning latency on local devices.
In this paper, we propose EfficientNav to enable on-device efficient LLM-based zero-shot ObjNav. To help the smaller LLMs better understand the environment, we propose semantics-aware memory retrieval to prune redundant information in navigation maps.
To reduce planning latency, we propose discrete memory caching and attention-based memory clustering to efficiently save and re-use the KV cache.
Extensive experimental results demonstrate that EfficientNav
achieves 11.1\% improvement in success rate on HM3D benchmark over GPT-4-based baselines, 
and demonstrates 6.7$\times$ real-time latency reduction and 4.7$\times$ end-to-end latency reduction over GPT-4 planner. Our code is available on https://github.com/PKU-SEC-Lab/EfficientNav.

EfficientNav: Towards On-Device Object-Goal Navigation with Navigation Map Caching and Retrieval
Zebin Yang!.
Sunjian Zheng
Tong Xie Shaushan Liu
Tianshi Xu
Bo Yu
Fan Wang"
Jie Tang*
Meng Li
Institute for Artificial Intelligence, Peking University
School of Integrated Cireuits, Peking University
Shenzhen Institute of Artificial Intelligence and Robotics for Society
Schooo of Computer Science and Engineering, South China University of Technology Beijing Advanced Innovation Center for Integrated Circuits
Abstract
Object-goal navigation (ObjNav) tasks an agent with navigating the location of a specific object in an unseen environment. Embodied agents equipped with large language models (LLMs) and online constructed navigation maps can perform ObjNav in a zero-shot manner. However, existing agents heavily rely on giant LLMs on the cloud, e.g., GPT-4, while directly switching to small LLMs, e.g., LLaMA3.2- lb, suffer from significant success rate drops due to limited model capacity for understanding complex navigation maps, which prevents deploying ObjNav on local devices. At the same time, the long prompt introduced by the navigation map description will cause high planning latency on local devices. In this paper, we propose EfficientNav to enable on-device efficient LLM-based zero-shot ObjNav. To help the smaller LLMs better understand the environment, we propose semantics-aware memory retrieval to prune redundant information in navigation maps. To reduce planning latency, we propose discrete memory caching and attention- based memory clustering efficiently Extensive experimental results demonstrate that EfficientNav achieves 11.1% improvement success rate HM3D benchmark over GPT-4-based baselines, and demonstrates 6.7x real-time latency reduction and 4.7x end-to-end latency reduction over GPT-4 planner. Our code is available https://gitubb.co//KKK ISEC-Lab/EffiiieeNNaa
and re-use the KV cache.
Introduction
Aiming to navigate agent to object specified by its category in unknown enviromment, object- goal navigation (ObjNav) presents a crucial yet challenging task for embodied agents [ 58 43 25)
To enhance the generality of ObjNav, recent studies have proposed leveraging the common-sense reasoning capabilities of large language models (LLMs) to achieve long-term planning in a zero-shot manner [40] 54 65 Specifically, LLMs act planners and determine series of navigation
sub-goals to guide the agent to the final destination [[] In such systems, navigation goals, maps of typically provided as context for the LLMs [10 91 g.
explored areas, and current observations
This information can be conceived as memory for LLMs, providing information of explored areas and visited objects facilitate planning and decision preciiion-maiing.
Corresponding author, Emails: boyu@ www..cdu.c..oo meng. edu.en
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Parameters
KV Cache
10
(22GB, 69%)
Parameters
10
Memory of Orin
LLaMA 00 Meta
10
Others
109
NVIDIA J etson Orin: 32GB
GPT-4V
PaLM-E
GPT-3.5
Liama 3.2-11b
LLaVA-7b
(a)
(b)
LLaVA-340
L30
LLaVA-70
LLaVA-34D
LLaVA-7b
10
KV Cache (GB)
Recompute Latency
Longer Real-time Latency
64
32
16
30% Memory of Orin
0
5
10
15
20
25
30
35
40
5
10
15
20
25
30
Navigation Steps
Navigation Steps
(c)
(d)
Figure 1: (a) Local devices (e.g.. NVIDIA Jetson AGX Orin) exhibit constrained memory capacity (32GB). (b) Only smaller LLMs, such as -lama-3.2-111 lb, can be used as the planner due to memory limitations. (c) The KV cache of map information accumulates with navigation steps, exceeding the memory capacity. (d) Recomputing the KV cache incurs long real-time latency.
Though promising, LLM-based Nav systems face significant challenges due to the substantial memory requirements and computational complexity of LLMs. To ensure task success, giant LLMs, such as GPT-4, are typically employed, necessitating offloading these models to the cloud 74, 66]. Such computing systems not only suffer from high communication latency, 36 90], but also raise privacy concerns 30 [6 and incur extremely high computational costs 69] Hence, computing
63
91
which degrades real sal-timme performance [26, 53 59
44
46
18 33
52
paradigm shift from offloading LLMs to the cloud towards accommodating LLMs on local servers or embedded devices, i.c., on-device computing, is required for ObjNav systems 62].
However, enabling on-device ObjNav presents two primary design challenges, largely constrained by limited memory resources. First, only smaller LLMs can be deployed on local devices, thus causing planning performance drops. For example, as shown in Figure [ (a) and (b), the NVIDIA Jetson AGX Orin (24] only has 32GB of DRAM, which can only accommodate smaller LLMs such as LLaMA-3.2-1I Since larger LLMs typically have higher model capacity, enabling them to handle more complex and longer navigation tasks, on-device LLMs struggle achieve the same level of performance as their cloud-based counterparts [8 40]. Second, as context information accumulates during the navigation process, its corresponding KV-cache also grows and can eventually exceed the memory constraint (in Figure (c), we set 30% of the Orin memory as the KV cache memory constraint, as the model weights also need to be stored memory). And re-computing the KV-cache significantly slows down LLM decoding, introducing substantial compute latency (Figure (d)).
In this paper, we propose EfficientNav, on-device memory-augmented ObjNav system that enables efficient zero-shot in-door navigation. We observe that describing the navigation map can lead to long prompt lengths, and the corresponding KV-cache often cannot be fully stored due memory constraints. Consequently, we select only a portion of the map description and load its KV cache for the LLM. However, the description selection will change the context order and the prefix of the corresponding KV cache. To enable reusing the retrieved KV cache despite the changing prefix, we propose discrete memory caching. This involves clustering the map information into groups and calculating the KV-cache for each group independently. Then, KV-caches within the same group are subject to cross-attention. To mitigate the impact of ignoring cross-attes attention between groups, we propose attention- based memory clustering. This technique uses LLM attention mechanisms to cluster related information into the same group. Furthermore, we observe that smaller LLMMs may struggle to fully understand complex navigation maps. To address this, we propose semantics-awart memory retrieval to prune redundant map information during the group selection process, thereby improving the performance of smaller LLMs.

---

We summarize our contributions as follows: We propose discrete memory caching to prevent saving the KV cache of the whole map description and meet the memory constraints. We propose attention-based memory clustering to reduce the impact of ignoring cross-attention between groups. We propose semantics-aware memory retrieval to efficiently select the relevant groups and prune redundant map information. We conduct extensive experiments and show that EfficientNav can achieves 1 11.1% % success rate improvements over GPT-4-based methods on HM3D dataset, and show 6.7x real-time latency and x end-to-end latency reduction over GPT-4 planner.
2 Related Work
LLM-based object goal navigation, According to the data processing pipeline, LLM-based ObjNav can be categorized into two paradigms: end-to-end and modular approaches. The end-to-end method, such Na Vid
83 directly converts RGB-D inputs into robot control policies, but incurs significant
training costs 6 27 The modular approach, which is more prevalent due to simplicity, involves using LLMs a high-level planner, and lower-level controller handled by the robot itself. 440 554
The LLM planner generates sub-goals to reach the final goal conditioned on the environment and instructions. The controller [56 [88. usually small neural network, finds trajectory from the current place to the sub-goal and controls the robot's motion in a real-time manner. We focus on optimizing the LLM planner, since it is the primary efficiency bottleneck due its long inference
latency and high memory cost (76] 10].
Memory Augmentation for object goal navi- gation. In zero-shot ObjNav tasks, the robot faces inefficiencies during sub-goal planning due to its limited field of view (7]. To achieve efficient environment exploration, mechanisms for memorizing navigation history are crucial for ObjNav 86 60 88) Maps built during navigation are typically leveraged to explicitly memorize the semantics and locations of visited areas and objects. While occupancy maps [40] can be used for memorization, graph-based nav- igation maps [I0 I 91 68 are increasingly
Description:
Novigation Map Description
(ddject dest cooloon (...). (Clpect powdior (-8
Hnsoisction (-200 (ahnn)
Detection
Instruction
Trajectory
narigation
abowe
dosorkiton Siffront
+
the ensirament Desed ans mommeraal
thaa you hace newn Your good is first the
LLM Planner
the
you have Irredted to betore Trajectory: (-100 (shama)
(-ill
The neal subgost in slns predion Di. My
Figure 2: Software flow of basdd Obv
favored for their ability to explicitly represent the semanties and locations of visited objects, which can be seamlessly integrated with 4LM--ased ObjNav 8 47). Hence, in this paper, we focus on the graph-based navigation maps.
Efficient LLMs. Many approaches have been proposed to enable efficient LLM inference on
devices with limited computational resources, including quantization I12 34 71
28
32 17). pruning
41221 [4] 41 [45., knowledge distillation [35] 50 73]. etc. However, these methods are not specifically optimized for ObjNav. In particular, they do not address the trade-off's between LLM size and the accuracy of the task planner, nor the challenge of efficient KV-cache saving and retrieval as
the navigated map expands ObjNav scenarios. We will further discuss them in Section 3.1
Comparisons with prior methods. Our ap-
proach employs a modular architecture that uses a graph-based navigation map to represent the semantic and spatial information of the navi- gated area. It leverages I.LMs to reason based on this graph, current observations, and the goal to generate the next navigation target. While aligned with many prior architectural designs, our approach is distinguished by its efficient and highly accurate on on-device ObjNav, as shown in Table This is achieved through our efficient KV-cache retrieval and pruning mechanism. No- tably, our method is orthogonal to and compat- ible with classic efficient L.LM techniques such as quantization, knowledge distillation, etc.
Table 1: Comparison with prior methods.
Method
Zero- shot
LLM
On On-device Inforence
Memory Augmenied
VIKING 55
NaVid [3
Vicuna
x
Skip SCAR
37
Navigation
2
GPT-4
X
InstructNas +M
GPT-4V
MapGPT 10
GPT-4
X
LFG 51
GPT-1
X
Imagine Before Go
GPI-4
X
SayNav [0
GPT-4
Efficientaov (Ours)
LLsMA
3

---

Map Description
Inatruction
Trajectory
Prefiling Latency
Decoding Latency
4930
Litency
1.54
0:0- Map compledty LUM Biee
gataa Staps
20
Complee
30
19
20
30
20
28
LUN Dox
LLaWA 180
LLahie 341
LLWVA-TD
LLAV 130
(a)
(b)
Figure 3: (a) Average length of different parts of prompts across navigation steps. Prompt lengths vary slightly across models due to divergent sub-goal selections influencing map complexity. (b) On-device LLM planning latency across navigation steps.
Attendic Notee 0,87X
Attention Score
Antentiinn
Attention Nobs
Adbrttion
Masse
0.358
Navigriion 0.
Object
Ofject
Objods
Intruction
Oijects
Olject
Objects Intruttion
(M) Atanisnndoi coinitinddiittiion alter
Altention dutribztion altur 50slaps
(a)
Figure 4: (a) Attention score on the most promising sub-goal (ground-trutt object) in different navigation steps. Attention distribution after (b) 10 steps and (c) 30 steps using LLaVA-34p in sub-goal planning. As the amount of map information increases with the progression of navigation, LLM can not fully understand the navigation map to focus on the most promising sub-goal. We observe similar problems in other small open-source LLMs such as LLaMA and Vicuna. The ground-truth object chosen by GPT-4o.
3
EfficientNav: Memory Augmented On-device Navigation System
3.1 Challenges and Overview
ObjNav software pipeline. In our ObjNav, the goal-finding process is composed of iterative sub- goal finding tasks until the final goal reached. Figure illustrates our ObjNav's navigation pipeline for finding sub-goal, and the ultimate goal. The first step involves generating semantic and distance information from RGB-D sensor captures using detection models, c.g., Grounding Dino [37]. Next, this semantic and spatial information is organized into the graph-based navigation map (we show an example in Appendix B). Third, the updated map and goal instructions are provided to the LLM for goal planning. Finally, if the target object is already present in the map, choose it as the next goal; otherwise, the planner chooses a promising object in the map as the next sub-goal and continues detecting new environmental information after reaching the sub-goal. The computation challenges of this method lies efficiently and accurately running the LLM as the map expands, which we discuss follows.
Challenge 1: tight memory constraints of local device limit the saving of KV cache of the navigation map description. As discussed in Section and shown Figure (a), with the progression of navigation, the amount of map information rapidly increases. thus introducing long prompt length, usually up to thousands of tokens. Re-computing the KV cache in each planning process will cause a long prefilling time, which is shown in Figure 3 (b). To avoid this cost, [22 saves the KV cache of history information in server memory, while shown in Figure (c). this can not meet the memory constraints of local devices. So new memory caching mechanism is needed.
Challenge 2: the tight memory constraint forces to use smaller LLMs, which have poorer model capacity and cause success rate degradation. To meet the memory constraints of local devices, usually tens of gigabytes, we need to use smaller LLMs, e.g., L.L VA-7b. However, directly using them to replace giant LLMs, such GPT-4, will cause significant success rate drop 8 29 In practice, we find this comes from that the smaller LILM with lower capacity can not fully understand complex navigation maps. With the progression of navigation, the LLM can not pay attention to the

---

Detection
Your goal to find thelsiet
Planning
clock
Stnc
Mirror
sub-goal
Attention-baue Memory Clustering (Sec. 3.3)
Mince Sink,
Uodels the EV aicha rewdy detected otpeets
Discrete Memory Caching (Sec 5.2)
Cock
Growpt
Group2
Groap3
Relevant groups are Group land
Figure Overview EfficientNav. Sim. stands for similarity.
[Gao al.2624) (Aa500]
EfficientNev (A5020)
[Gao et al 2024) (Orin)
EtficinnNaa
Toilet TV
Bed
16
Selected Times
12-
KV Cicho
0.4
1n
0.0
2
4
8
10
11
12
13
14
15
16
Groupp Stored in Storage Units
Group
(a)
(b)
Figure (a) Memory transfer time of LLaVA-7b when using low-speed storage units. (b) For different final goals, the selected times of different groups are different, thus showing different relevance. For each final goal, the navigation task is conducted 15 times with different starting points. The groups are selected by GPT-4o.
important information among the huge amount of map information and choose the correct sub-goal, which shown in Figure 4 So a memory selection strategy needed to remove the redundant information for the smaller LLM.
EfficientNav Overview. Based on these observations, we propose EfficientNav to enable efficient zero-shot ObjNav local devices. demoonsseesssssttes the overview of EfficientNav. To meet the memory constraints, we propose discrete memory caching to cluster objects into groups and calculate the KV cache of each group individually. Then only select portion of groups to LLM and load their KV cache to device memory. To accurately cluster information and improve success rate, we propose iitention-based memory clustering, using LLM attention to guide group clustering. To help the smaller LLMs better understand the navigation map, in the group selection process, we propose semantic-aware memory retrieval efficiently prune redundant map information.
3.2 Discrete Memory Caching
As discussed in Section 3.1 the memory constraints of local devices prevent full storage of the KV cache of navigation map descriptions. To mitigate this, [13] offloads the KV cache in high-capacity but low-speed storage units, e.g., CPU host memory for NVIDIA RTX A6000 or disk for Jetson Orin, and loads the KV cache on-demand during decoding. However, since local memory all the KV cache, this method requires repeated loading the KV cache to device memory in each decoding phase. However, each planning process, L.LM needs decode multiple tokens (around 40 in practice). The frequent communication between device memory and storage units will also cause huge latency, which is shown in Figure (a).
not retain
To meet the memory constraint while avoiding re-computation and frequent memory transfers, we propose discrete memory caching strategy. Building [33] store the KV cache in low-speed storage units and load the required KV cache into device memory, but introduce critical optimization. As the navigation map contains redundancy, we only select partial important information according to the memory budget and load the KV cache only once, thus reducing memory transfer cost.
However, when selecting information, the order of context in the prompt will change. When pre- calculating the KV cache, existing works [38 89 calculate the full attention the whole context. In this approach, only the KV cache of its longest common prefix with the selected information can be
5

---

Ataasion of each group is outcatted sulvstaally the cache dacsupled with groep order.
cachee for layers cesting device momory.
Group
Group
Crose-ttivetie Indeesn diffcrett groupe ane Igrord.
Group
Clucter anely denecced otjects high attention scone.
nenlyy acted objects
Group
cachee Instrccioon and re re-computed
Group
nevell plaveig
Group 3-
Group
Group
Group 4-
Esceeding the sttention threstodd
Iogustinn
Trijctory
aradel
(a)
(b)
(c)
Figure 7: (a) When using discrete memory caching, the KV cache of existing groups can be reused in LLM planning. (b) When groups with newly added objects are selected during LLM planning process, the KV cache of these objects can be directly calculated and saved without impacting existing KV cache. (c) When using attention-basee memory clustering, the newly detected objects will be clustered into different groups according to attention distribution.
reused. The KV cache after the changed position needs to be recomputed. To address this, we cluster objects in the navigation map into groups and compute KV cache of each group individually, as shown in Figure 7 (a). Then, only partial groups are selected to prompt the LLM planner. Hence, we can decouple the group KV cache and the group order, thus making full usage of the saved KV cache and avoiding re-computation. As shown in Figure in each navigation step, newly detected objects will be added to existing groups, and need to add their KV cache the group's KV cache. To avoid changing context order within the group and impacting the existing KV cache, we concatenate the information of new objects at the end of the group information. As shown in Figure (b), in the planning process, when groups with newly added objects are selected, the KV cache of these objects can be directly calculated and saved, without introducing extra computation. We follow the position encoding strategy in LLM inference as 15
However, discrete memory caching will cause the ignorance of cross-attention between groups, leading to LLM performance drop [9]] We also need to ensure that the groups with important information are not removed in group selection. Therefore, correct memory clustering and memory selection mechanism is very important. We will explain our design in Section 3.3 3land 3.4
3.3 Attention-based Memory Clustering
to reduce the impact of ignoring cross-attention and maintain a high success rate, we need cluster the information accurately and efficiently. Existing works [22] 75) partition context into uniform-length chunks, neglecting the relationships between objects in the navigation map. For example, the relationship between the oven and the pot is closer than the relationship between the oven and the bed. By grouping objects with closer relationships together and calculating their cross-attention, LLMs can better understand the navigation map and abstract the environment of larger area. Meanwhile, the grouping granularity is also important. In contrast, if the granularity is too coarse, the KV cache size of each group will become large. Hence, only a few groups can be selected for the LLM, which will enhance the difficulty of selection and may neglect important information.
As discussed in Section 3.2
To adaptively control object clustering and group granularity, we use the attention of L.LM itself to cluster the newly detected objects into different groups As shown in Figure 7 (c), in the clustering process, input the information of existing groups and the newly detected objects into LLM, and only infer a few layers (in practice, we find around 1 To layers of the whole model is enough). If the average attention between a newly detected object and existing group exceeds specific threshold, we cluster this object into the group. Otherwise, the remaining objects will be added to a new group. For the first group. simply cluster the detected objects in the first navigation step into group.
By using attention-based memory clustering, we can reduce the difference between discrete attention and full attention. Note that the clustering process will not introduce much extra computation and latency. This is because we only compute the first few layers of LLM,, and the KV cache of existing groups has been pre-computed and its cache of the first few layers are kept device memory.

---

3,4 Semantics-aware Memory Retrieval
As discussed in Section 3.1 the smaller LLM cannot fully understand complex navigation maps. Thus, a memory selection mechanism is needed to remove redundant information. [54] empirically selects environmental information based on object positions. However, we observe that when the final goal changes, the relevance of each group to the final goal also changes, shown in Figure 6 (b). This empirical selection can not adapt to different final goals, thus showing lower performance. At the same time, it does not consider different device memory budgets. 8 uses LLM to select relevant information, while this will introduce extra LLM calls and long real-time latency.
Based on this, we propose semantics-awan memory retrieval, efficiently selecting groups according to semantic information adapt different sub-goals. As the task of removing redundant information is much easier than finding specific sub-goal to improve efficiency, we conduct group selection and sub-goal planning in a small-large model collaboration mode. For easier group selection, we use CLIP model [ instead of LLM, which only has around 100M parameters and lower inference latency. For the harder sub-goal planning, we use I.M to select one specific object as the sub-goal. In the information selection process, we use CLIP to encode the object information in each group and the final goal. We then calculate the similarity between the encoding results of the final goal and each group, which is viewed as the probability for a group to include potential sub-goals, We consider the group relevant to the final goal only if the probability is larger than a threshold. To adapt to different devices with different memory budgets, we formulate the group selection as knapsack problem:
maximize
P=
Σ 2m1
(P threshold) 24
n Mr- M4 M, x2 E {0,1)
subject to
where P. denotes the probability include sub-goals in group i, M; and M denotes the memory usage of KV cache of group i and device memory budget, z; denotes whether to select group i, n denotes the number of groups. After solving the knapsack problem, we can select truly relevant information for LLM and meet the memory budget at the same time.
By using semantics- aware memory retrieval, we can efficiently select relevant groups adapting to different final goals and devices. Note that in each planning process, we only update the decoding results of the groups with newly detected objects. As the inference time of CLIP is much lower than LLM, the latency of the selection process is negligible. At the same time, between two adjacent planning processes, the navigation map will not change much. So it is likely that some group both selected in the current step and last step. Hence, their KV cache has been loaded into device memory, thus reducing the KV cache loading time, which we will further show in our experiments.
4
Experiments
4.1 Experiment Setup
We evaluate EfficientNav the HM3D dataset 3 based on the Habitat simulation platform 61.. In the simulation platform, the robot can access RGBD observation of the environment. In each task, the robot is placed ent starting point in the environment and is only instructed to find specific object, e.g., "TV", "chair", "sofa", "bed", "toilet", "plant" etc., which harder than tasks giving detailed, step-by-stey step directions 22] 48 For accuracy. we report two metrics: i) the average success rate (SR), and ii) the success rate penalized by path length (SPL), which both evaluates the accuracy and the efficiency of robot trajectory. For system efficiency, we report two metrics: i) real-time latency (RtL): the average robot planning time
Table SR and SPL. comparison.
differ-
DD-PPO 67
27.9
14.2
SemExp 9
37.9
18.8
OVRL
62.0
26.8
ZSON
25.5
12.6
PixeINav
GPT-4
ESC
5
39.2
22.3
VaraNav 65
GPT-3.5
42.0
26.0
LLaVA Plamner334b
30
LLaVA-34b
42.7
21.0
L3MVN h0
RoRERTa--arge
InstmerrNa
GPT-4V
LPG 4
GPT-4
68.9
36.0
EfideentNar-11
LLaMA3.2-1Ib
74,2
39.5
EMMcionNNav33bb
LLaVA-34b
7

---

in one navigation step. ii) End-to-end latency (E2EL); the average latency for the robot to complete one navigation task (including multiple navigation steps and robot moving time). We implement our methods on 4 L.Mss, LLaVA-7b, LLaVA-13b, LLaVA-34b, and LLaMA3.2-1 2-11-3bb211bbb. 11b on NVIDIA A6000 GPU and Jetson Orin. We show the network architecture and implementation details in Appendix [ and G
We also show an example of EfficientNav pipeline in Section B and
4.2
Main Results
Comparison with state-of-the-art. The comparison of SR and SPL is shown in Table Compared with learning- based methods, EfficienNav achieves 18.0% SR and 14.7% SPL improvements over the prior-art art method OVRL (72), without any training cost. This demonstrates the advantage of LLM- based navigation system, as discussed in Section [ Compared with zero-shot methods, EfficientNav achieves 1 11.1% % SR and 5.5% SPL improvements over LFG (54), which utilizes GPT-4. Compared with naive VA-34b planner [TO), EfficientNav achieves 37.3% SR and 20.5% SPI. improvements. This is because we use semantics- memory retrieval, which helps the LLM focus on the most relevant groups and removes redundant information.
The latency comparison is shown in Table E Compared with GPT-4 planner [10), Efficient- Nav achieves 6.7 real-time latency reduction and 4.7x end-to-end latency reduction, by sav- ing the communication time the cloud server. Compared with naive LLaVA planner, Effic- cientNav achieves 8.8x and 6.5x real-time latency reduction and 3.7x and 4.4x end-to- end latency reduction on LLaMA3.2-1 lb and LLaVA-34b. This mainly comes from using dis- crete memory caching avoid re-computation in the prefilling stage of planning. Prior-art so- lutions such as vllm can accelerate LLM inference, but they can not solve the problem of high re-computation cost, which is the main bottleneck. Compared with vllm, EfficientNav achieves 6.5x and 5. x real-time latency reduction and 3. x and 3.8x end-to-end latency reduction on I.L.M3.2-11 1b and VA-34b3
Table Average latency comparison on A6000.
Method
LLM
EZEL
GPT-4 Planner [0
GPT-4
5.80s
59.34
LLaMA Planner-
10
11aMA3.2-11b
3.07
46.40s
vilm 3D
LLaMA3.2-11b
2.27
39.78s
EfficientNav-I 1lb (Ours)
LLaMA3,2-11b
0.35:
12.70s
LLaVA Planner-344 34b [d]
LLaVA-34b
5.63%
55.326
vilm 30
LLaVA-34b
4.43s
47,95s
Effiien (Ours)
LLaVA-34b
0.87%
12,51s
Real-time latency in different navigation steps. The amount of map information in- creases in each navigation step. As shown in Figure 8 when using traditional serving methods [30], the increasing prompt length will introduce high recomputation cost and in- creasing real-time latency. However, when us-
LLaVA Planner-7b (Orin)
EfficientNav-7b (Orin)
LLaVA Planner-4bb
EfficientNav-34 EfficientNav-79
Planner-7b
8-
Real-time Latency
ing EfficientNav, the real-time latency stabi- lizes after a certain navigation step, as we use semantics-aware memory retrieval select par- tial groups for LLM according to memory bud- get. With the same amount of information given to LLM, EffcientNav also shows lower real-time latency, as discrete memory caching saves the re-computation time of LLM prefilling.
Number of Navigation Steps
Figure 8: Comparison of real-time latency in dif- ferent navigation steps on A6000 and Jetson Orin.
Planning latency breakdown. Figure 9 shows the latency breakdown of one naviga- tion step. Compared to traditional serving meth- ods 30), by avoiding re-computation, discrete memory caching can significantly reduce the prefilling time of planning by around 20x. And semantics-awart aware memory retrieval removes the redundant information, thus reducing the prompt length and reducing the computation cost and latency of the decoding stage of planning. The latency of memory clustering and memory selection is negligible, as discussed in Section 3.3 and 3.4
Memory Clustering
Memory Selection Dacode Staga of Planning
Profil Stage Painiing
LLaVA Planner
EfficientNe
2
Latency (8)
Figure 9: Planning latency (s) breakdown for LLaVA planner and EfficientNav A6000.
8

---

Numberr or
1.0
Number Grouge
1.0
0.6
06
0u4
04
18
110
00
00
1J
16
24
12 20
01
26 24
-0
Device Nemory Budget
Device Memorry Budget
Device Memory (Gdget(G8)
(a) LLova-77b
(b) LLaVA-110
LL)VA43333b
Figure 10: Comparison of memory caching distribution and memory cache hit rate across different device memory budgets for KV cache, using (a) LLaVA-7b, (b) LLaVA-13b, and (c) LLaVA-34b. Cache hit rate is defined as the proportion of selected groups that have been stored in device memory.
Memory caching distribution and cache hit rate of EfficientNav. When using EfficientNav, in each planning process, we only select relevant groups for LLM. With larger device memory budgets for KV cache, we can store more groups in device memory. As discussed in Section 3.4 some selected groups may be just stored in device memory, so we do not need reload these groups from storage units. We define the proportion of these groups as the cache hit rate. A high hit rate will lead to low caching loading time. As shown in Figure 10] the cache hit rate of EfficientNav increases with the device memory budget and rapidly reaches a high level. This is because when the memory budget increases, more relevant groups stored in device memory. At the same time, the semantics of each group will not change much between adjacent planning processes. So it is likely that some relevant groups are just loaded to device memory in the last few steps, leading to high cache hit rate.
4.3 Ablation Study
Individual influence of our meth- ods. Table 4 shows the individual influence of our proposed methods. In naive LLaVA planner, we empirically cluster and select environmental infor- mation according to object positions By using discrete memory caching, compared with naive L.LaVA planner [0), we achieve 2.3x real- time latency and 1.5x end-to-end la- tency reduction, by re-using KV cache of navigation map description and avoiding re-computation. After attention-based memory clustering is used, objects in each selected group have high correlation, which helps the LLM to better understand the environment and reduces the impact of ignoring cross-attention at the same time, thus improving the success rate. When using memory retrieval, we can select the most related groups, which further improves the success rate. We also achieve 2.7x real-time latency and 2.6x end-to-end latency reduction, by selecting groups according to memory budget and avoiding frequent communication with storage units.
Table 4: Ablation study on EfficientNav methods with LLaVA-34b.
Method
SR
SPL
RtL
E2EL
LLaVA Planner
42.7
21.0
5.63s
55,32s
85 [4]]
+Discrete Memory Caching
43.1
21.0
2.42s
36.94s
-Attention-base Memory Clustering
63.3
34,2
2.32s
32.58s
*Semanties-aware Memory Retrieval
80.0
41.5
0.87s
12.51s
Impact of device memory budget for KV cache. In semantics-aaee awarr memory retrieval, select groups according to the device memory budget. Here we evaluate the impact of memory budget on success rate and latency us- ing LLa As shown in Table on most cases, our method shows good robustness. How-
Table 5: Ablation study on different device mem- ory budgets for KV cache with LLa VA-34b.
Memory Budget
SR
SPL
RtL
E2EL
16GB
74.7
37.3
0.59s
14.24s
ever, when the memory budget for KV cache is
24GB
79.0
39.1
0.71s
14.29s
small, fewer relevant groups can be selected. Hence, the potential sub-goal may be ignored, which causes success rate drops and longer end- to-end latency. For a larger memory budget, more relevant groups can be selected. While the longer prompt length may cause longer planning time, which will slightly increase the real-time latency.
32GB
80.0
41.5
0.87s
12.51s
40GB
80.3
41.9
0.93s
11.72s
Comparison with sparse attention approaches. Our discrete memory caching is similar to adaptive sparse attention methods 220 31 84). However, the goals of general adaptive sparse attention and EfficientNav are different: EfficientNav clusters objects into different groups and calculates

---

the attention of each group individually. This structured sparse approach can decouple the group order and KV cache computation of each group, thus enabling KV reuse when we retrieve different groups in navigation planning and re- ducing real-time latency. However, for general adaptive sparse attention methods, their purpose to minimize the difference between sparse
Table 6: Comparison with sparse attention ap-
proaches with LLaVA LL-44b34b.
Method
SR
SPL
Real-time latency
Minference 20
35.3
18.0
3.88s
FlexPrefill [1
36.7
20.1
3.02s
EfficientNay
80.0
41.5
0.87s
attention and full attention and accelerate the attention calculation. The comparison with sparse Just using adaptive sparse attention methods shows huge accuracy drops. This is because these methods minimize the difference between sparse attention and full attention. But without memory retrieval, even full attention shows a low success rate, as discussed in Section [.1] At the same time, without discrete memory caching, these methods need to recompute the KV cache of the navigation map description because map changes, thus showing longer real-time latency.
attention approaches is shown in Table 6
5 Conclusion and Limitations
This work proposes EfficientNav, which enables efficient zero-shot Obj local devices. We propose discrete memory caching to enable re-using KV cache and avoid re-computation LLM planning. We also propose attention-based memory clustering to reduce the impact of ignoring cross-attention. To improve planning performance smaller LLMs, we propose semantics-aware memory retrieval to remove the redundancy in navigation map. EfficientNav overcomes all existing baselines and enables running zero-shot ObjNav on local devices. While our study shows promising results, it has some limitations. We mainly focus LLM-based navigation in this paper, while the inference speed of LLM can not reach that of small models even after acceleration. And EfficientNav should be carefully used in applications that need extremely low real-time latency.
6 Acknowledgments
This work was supported in part by NSFC under Grant 62495102, Grant 92464104, and Grant 62341407, in part by the National Key Research and Development Program under Grant 2024YFB4505004, in part by Beijing Municipal Science and Technology Program under Grant Z241100004224015 in part by 111 Project under Grant B 180011, and in part by Longgang District Shenzhen's 's "Ten Action Plan" " for Supporting Innovation Projects under Grant LGKCSDPT202400> and LGKCSDP202400
References
[1] Dong An, Hanqing Wang, Wenguan Wang, Zun Wang, Yan Huang, Keji He, and Liang Wang. Etpnav: Evolving topological planning for vision-language navigation in continuous environments. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[2] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. ssiion-and-langaaa navigation: Interpreting visually-grounded navigation instructions in real environments. Proceedings of the IEEE conference computer vision and pattern recognition, pages 3674-3683, 2018
[3] Szot Andrew, Yadav Karmesh, Clegg Alex, Berges Vincent-Pierre, Gokaslan Aaron, Chang Angel, Savva Manolis, Kira Zsolt, and Batra Dhruv. Habitat rearrangement challenge 2022.
tat htt.a//bb.......or//caaaaeee////
2022.
[4] Abrar Anwar, John Welsh, Joydeep Biswas, Soha Pouya, and Yan Chang, Remembr: Building and reasoning over long-horizon spatio-temporal memory for robot navigation. arXiv preprint arXiv:2409. /3682 2024.
[5] Aram Bahrini, Mohammadsadra Khamoshifar, Hossein Abbasimehr, Robert J Riggs, Maryam Esmaeili, Rastin Mastali Majdabadkohne, and Morteza Pasehvar. Chatgpt: Applications, opportunities, and threats. In 2023 Systems and Information Engineering Design Symposium (SIEDS), pages 279-279. IEEE, 2023.
10

---

[6] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choro- manski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, al. Rt-2: Vision-language- action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307. 15818, 2023.
[7] Wenzhe Cai, Siyuan Huang, Guangran Cheng, Yuxing Long, Peng Gao, Changyin Sun, and Hao Dong. Bridging zero-shot object navigation and foundation models through pixel-guided navigation skill. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 22344.23. IEEE, 2024.
[8] Yihan Cao, Jiazhao Zhang. Zhinan Yu, Shuzhen Liu, Zheng Qin, Qin Zou, Bo Du, and Kai Xu. Cognav: Cognitive process modeling for object goal navigation with llms. arXiv preprint arXiv:2412. 10439, 2024.
[9] Devendra Singh Chaplot, Dhiraj Prakashchand Gandhi, Abhinav Gupta, and Russ R Salakhut- dinov. Object goal navigation using goal-oriented semantic exploration. Advances in Neural Information Processing Systems, 33:4247-4255 2020.
[10] Jiaqi Chen, Bingqian Lin, Ran Xu, Zhenhua Chai, Xiaodan Liang, and Kwan- Yee Wong. Mapgpt:
Map-guided prompting with adaptive path planning for vision-and and-language navigation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9796-9810, 2024.
[11] Vishnu Sashank Dorbala, Gunnar Sigurdsson, Robinson Piramuthu, Jesse Thomason, and Gaurav S Sukhatme. Clip-nav: Using clip for zero-shot vision-and-languag navigation. arXiv preprint arXiv:2211. 2022.
[12] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprini arXiv:2210.1732 2022.
[13] Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, and Pengfei Zuo. [Cost-Effiieent) large language model serving for multi- turn conversations with (CachedAttention). In 2024 USENIX Annual Technical Conference (USENIX ATC 24), pages 11 111-126. 2024.
[14] Shanggian Gao, Chi-Heng Lin, Ting Hua, Zheng Tang, Yilin Shen, Hongxia Jin, and Yen-Chang Hsu. Disp-llm: invension-independeee structural pruning for large language models. Advances in Neural Information Processing Systems, 37:72219-72244 2024.
[15] In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong. Prompt cache: Modular attention reuse for low-latency inference. Proceedings of Machine Learning and Systems, 6:325-336 2024.
[16] Meng Hao, Hongwei Li, Hanxiao Chen, Pengzhi Xing, Guowen Xu, and Tianwei Zhang. Iron: Private inference transformers. Advances in neural information processing systems, 15:15718-15731, 2022.
[17] Coleman Hooper, Schoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Sophia Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length I1m inference with kv cache quantization. Advances in Neural Information Processing Systems, 37:1270-1303, 2024.
[18] Xiaoyang Hou, Jian Liu, Jingyu Li, Yuhan Li, Wen-jie Lu, Cheng Hong, and Kui Ren. Ciphergpt: Secure two-party gpt inference. Cryptology ePrint Archive, 2023.
[19] Junhao Hu, Wenrui Huang, Haoyi Wang. Weidong Wang, Tiancheng Hu, Qin Zhang, Hao Feng, Xusheng Chen, Yizhou Shan, and Tao Xie. Epic: Efficient iositionnindeeeeneett context caching for serving large language models. arXiv preprint arXiv:2410.15332, 2024.
[20] Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H Abdi, Dongsheng Li, Chin-Yev Yew Lin, et al. Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention. Advances Neural Information Processing Systems, 37:52481-525155 2024.

---

[21] Yikun Jiang, Huanyu Wang, Lei Xie, Hanbin Zhao, Hui Qian, John Lui, et al. D-llm: A token adaptive computing resouree allocation strategy for large language models. Advances in Neural Information Processing Systems, 37:1725-1749, 2024.
[22] Chao Jin, Zili Zhang, Xuanlin Jiang, Fangyue Liu, Xin Liu, Xuanzhe Liu, and Xin Jin. Ragcache: Efficient knowledge caching for retrieval-augmented generation. arXiv preprint arXiv:2404. 12457, 2024.
[23] Zhao Kaichen, Song Yaoxian, Zhao Haiquan, Liu Haoyu, Li Tiefeng, and Li Zhixu. Towards coarse-grained visual language navigation task planning enhanced by event knowledge graph. arXiv preprint arXiv:2408.02553. 2024.
[24] Leela S Karumbunathan. Nvidia jetson agx orin series, 2022.
[25] Serge Kernbach and Kristof Jebens. Development of a cost-efficient autonomous mav for an unstructured indoor environment. arXiv preprint arXiv:1111.0500, 2011.
[26] Oussama Khatib. Real-time obstacle avoidance for manipulators and mobile robots. The international journal of robotics research, 5(1):90-99) 1986.
jouraaall
[27] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti. Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, al. Openvla: An open-source vision-language-actionn model. arXiv preprint arXiv:2406.092240 2024.
[28] Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. Squeezellm: Dense-and and--aasd- quantization. arXiv preprint arXiv:2306.0766) 07629, 2023.
[29] Taewoong Kim, Byeonghwi Kim, and Jonghyun Choi. Multi-modal grounded planning and efficient replanning for learning embodied agents with a few examples. arXiv preprint arXiv:24/2. 17288, 2024.
[30] Woosuk Kwon, Zhuohan Li. Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 61 1666, 2023.
[31] Xunhao Lai, Jianqiao Lu, Yao Luo, Yiyuan Ma, and Xun Zhou. Flexprefill: A context- aware sparse attention mechanism for efficient long-sequens. inference. arXiv preprint 2rXiv:2502.20766, 2025.
[32] Shiyao Li, Xuefei Ning, Ke Hong, Tengxuan Liu, Luning Wang, Xiuhong Li, Kai Zhong, Guohao Dai, Huazhong Yang, and Yu Wang. Llm-mq: Mixed-precisiin quantization for efficient Ilm deployment. In NeurlPS 2023 Efficient Natural Language and Speech Processing Workshop, pages 1-5, 2023.
[33] Chenqi Lin, Tianshi Xu, Zebin Yang, Runsheng Wang, Ru Huang, and Meng Li. Fastquery: Communication efficient arXiv:2405.1 16241, 2024.
eeffefcc embedding table query for private Ilm inference. arXiv preprint
[34] Lin, Jiaming Tang, Haotian Tang, Shang Yang. Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device IIm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87-100, 2024.
[35] Jiaheng Liu, Chenchen Zhang, Jinyang Guo, Yuanxing Zhang, Haoran Que, Ken Deng, Jie Liu, Ge Zhang, Yanan Wu, Congnan Liu, et al. Ddk: Distilling domain knowledge for efficient large language models. Advances in Neural Information Processing Systems, 37:98297-98319 2024.
[36] Runze Liu, Jianlei Yang, Yiran Chen, and Weisheng Zhao. eslam: An energy-efficient ac- celerator for real-time orb-slam on fpga platform. In Proceedings of the 56th Annual Design Automation Conference 2019, pages 1-6, 2019.
12

---

[37] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li. Hao Zhang, Jie Yang. Qing Jiang, Chumyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European conference on computer vision, pages 38-55. Springer, 2024.
[38] Shu Liu, Asim Biswal, Audrey Cheng, Xiangxi Mo, Shiyi Cao, Joseph E Gonzalez, lon Stoica, and Matei Zaharia. Optimizing Ilm queries in relational workloads. arXiv preprini arXiv:2403.05821, 2024.
[39] Yaotian Liu and Jeff Zhang. Skip-scar: A modular approach to objectgoal navigation with sparsity and adaptive skips. arXiv preprint 1rXiv:2405.14154 2024.
[40] Yuxing Long, Wenzhe Cai, Hongcheng Wang, Guanqi Zhan, and Hao Dong. Instructnav: Zero-shot system for generic instruction navigation in unexplored environment. arXiv preprint arXiv:2406.04885 2024.
[41] Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 20221002221222 2023.
[42] Arjum Majumdar, Gunjan Aggarwal, Bhavika Devnani, Judy Hoffman, and Dhruv Batra. Zson: Zero-shot object-goal navigation using multimodal goal embeddings. Advances in Neural Information Processing Systems, 35:32340- -23522 2022.
[43] Ali Reza Manashty, Amir Rajabzadeh, and Zahra Forootan Jahromi. A scenario- based mobile application for robot-assisted smart digital homes. arXiv preprini 1009::1009.5 5399,, 2010.
[44] Yongguo Mei, Yung-Hsiang Lu, Yu Charlie Hu, and CS George Lee. Deployment of mobile robots with energy and timing constraints. IEEE Transactions on robotics, 22(3):507-522, 2006.
[45] Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. Compact language models via pruning and knowledge distillation. Advances in Neural Informa- tion Processing Systems, 37:4107-441102, 2024.
[46] Sean Murray, William Floyd-Jones, Ying Qi, George Konidaris, and Daniel J Sorin. The microarchitecture of real-time robot motion planning accelerator. In 2016 49th Annual JEEE/ACM International Symposium on Microarchitecturre (MICRO), pages 1-12. IEEE, 2016.
[47] Dujun Nie, Xianda Guo, Yiqun Duan, Ruijun Zhang, and Long Chen. Wmnav: Integrat- ing vision-language models into world models for object goal navigation. arXiv preprint 25X20:2503.02247, 2025.
[48] Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang. William Yang Wang, Chunhua Shen, and Anton van den Hengel. Reverie: Remote embodied visual referring expression in real indoor environments. In Proceedings of the IEEF/VV Conference on Computer Vision and Pattern Recognition, pages 9982-9991, 2020.
[49] Abhinav Rajvanshi, Karan Sikka, Xiao Lin, Bhoram Lee, Han-Pang Chiu, and Alvaro Velasquez. Saynav: Grounding large language models for dynamic planning navigation in new environ- ments. In Proceedings of the International Conference on Automated Planning and Scheduling. volume 34, pages 464-474, 2024.
[50] Mrigank Raman, Pranav Mani, Davis Liang, and Zachary Lipton. For distillation, tokens are not all you need. In NeurlPS 2023 Workshop on Instruction Tuning and Instruction Following. 2023.
[51] Ram Ramrakhya, Eric Undersander, Dhruy Batra, and Abhishek Das. Habitat-web: Learning embodied object-search strategies from human demonstrations scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5173-5183, 2022.
[52] Xuanle Ren, Zhaohui Chen, Zhen Gu, Yanheng Lu, Ruiguang Zhong, Wen-Jie Lu, Jiansong Zhang, Yichi Zhang, Hanghang Wu, Xiaofu Zheng, et al. Cham: A customized homomor- phic encryption accelerator for fast matrix-vector product. In 2023 60th ACM/IEEE Design Automation Conference (DAC), pages 1-6. IEEE, 2023.
13

---

[53] Deval Shah, Ningfeng Yang, and Tor M Aamodt. Energy-efficient realtime motion planning. In Proceedings of the 50th Annual International Symposium on Computer Architecture, pages
1-17,201 2023.
[54] Dhruv Shah, Michael Robert Equi, Blažej Osiński, Fei Xia, Brian Ichter, and Sergey Levine. Navigation with large language models: Semantic guesswork as heuristic for planning. In Conference Robot Learning, pages 2683-2699. PMLR, 2023.
[55] Dhruv Shah and Sergey Levine. Viking: Vision-based kilometer-scale navigation with geo- graphic hints. arXiv preprint arXiv:2202.11271, 2022.
[56] Dhruv Shah, Ajay Sridhar, Nitish Dashora, Kyle Stachowicz, Kevin Black, Noriaki Hi- rose, and Sergey Levine. Vint: A foundation model for visual navigation. arXiv preprint arXiv:2306. 14846, 2023.
[57] Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. Mpnet: Masked and permuted pre-training for language understanding. Advances in neural information processing systems, 33:16857-16867, 2020.
[58] Ajay Sridhar, Dhruv Shah, Catherine Glossop, and Sergey Levine. Nomad: Goal masked diffusion policies for navigation and exploration. In 2024 IEEE International Conference Robotics and Automation (ICRA), pages 63-70. IEEE, 2024.
[59] Amr Suleiman, Zhengdong Zhang, Luca Carlone, Sertac Karaman, and Vivienne Sze. Navion: A 2-mw fully integrated real-time visual-inertial odometry accelerator for autonomous navigation of nano drones. IEEE Journal of Solid-State Circuits, 54(4):1
2019.
[60] Leyuan Sun, Asako Kanezaki, Guillaume Caron, and Yusuke Yoshiyasu. Enhancing multimodal- input object goal navigation by leveraging large language models for inferring room noom-ojject relationship knowledge. Advanced Engineering Informatics, 65:103135, 2025.
[61] Andrew Szot, Alexander Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Singh Chaplot, Oleksandr Maksymets, et al. Habitat 2.0: Training home assistants to rearrange their habitat. Advances in Neural Information Processing Systems, 34:251-266, 2021.
[62] Emil Talpes, Douglas Williams, and Debjit Das Sarma. Dojo: exa-scale computer. In 2022 IEEE Hot Chips 34 Symposium (HCS), pages 1-28. IREE Computer Society, 2022.
The microarchitecture of tesla's
[63] Naoki Wake, Atsushi Kanehira, Kazuhiro Sasabuchi, Jun Takamatsu, and Katsushi Ikeuchi. Gpt-4v (ision) for robotics: Multimodal task planning from human demonstration. arXiv preprint arXiv:2311.12015 2023.
[64] Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distillation for task aas-agnostic compression of pre-trained transformers. Advances in neural information processing systems, 33:5776-5788 2020.
[65] Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy. Videoagent: Long-form video understanding with large language model as agent. arXiv preprint arXiv:2403.10511 2024.
[66] Renjie Wei, Songqiang Xu, Linfeng Zhong, Zebin Yang, Qingyu Guo, Yuan Wang, Runsheng Wang, and Meng Li. Lightmamba: Efficient mamba acceleration on fpga with quantization and hardware co-design. In 2025 Design, Automation & Test in Europe Conference (DATE), pages 1-7. IEEE, 2025.
[67] Erik Wijmans, Abhishek Kadian, Ari Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra. Dd-ppo: Learning near-perfect pointgoal navigators from 2.5 billion frames. arXiv preprint arXiv: 19 2019.
[68] Pengying Wu, Yao Mu, Bingxian Wu, Yi Hou, Ji Ma, Shanghang Zhang, and Chang Liu. Voronav: Voronoi-based zero-shot object navigation with large language model. arXiv preprint arXiv:2401.02695. 2024.
14

---

[69] Xiaodong Wu, Ran Duan, and Jianbing Ni. Unveiling security, privacy, and ethical concerns of chatgpt. Journal of Information and Intelligence, 2(2):102-115. 2024.
[70] Xiaoshi Wu, Feng Zhu, Rui Zhao, and Hongsheng Li. Cora: Adapting clip for open-vocabulary detection with region prompting and anchor pre-mate m-tthhhhinn In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7031-7040, 2023.
[71] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087-38099, PMLR, 2023.
[72] Karmesh Yadav, Ram Ramrakhya, Arjun Majumdar, Vincent- Berges, Sachit Kuhar, Dhruv Batra, Alexei Baevski, and Oleksandr Maksymets. Offline visual representation learning for embodied navigation. In Workshop on Reincarnating Reinforcement Learning at ICLR 2023,
2023.
[73] Runming Yang, Taiqiang Wu, Jiahao Wang, Pengfei Hu, Yik-Chung Wu, Ngai Wong, and Yujiu Yang. Llm-neo: Parameter efficient knowledge distillation for large language models. arXiv preprint arXiv:2411.06832 2024.
[74] Zebin Yang, Renze Chen, Taiqiang Wu, Ngai Wong, Yun Liang, Runsheng Wang, Ru Huang, and Meng Li. Mcubert: Memory-efficiett bert inference on commodity microcontrollers. arXiv preprint arXiv:2410. 1795777777 2024.
[75] Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. Cacheblend: Fast large language model serving with cached knowledge fusion. arXiv preprint arXiv:2405. 16444, 2024.
[76] Hang Yin, Xiuwei Xu, Zhenyu Wu, Jie Zhou, and Jiwen Lu. Sg-nav: Online 3d scene graph prompting for Ilm-based zero-shot object navigation. Advances in Neural Information Processing Systems, $7:5285-5309 2024.
[77] Naoki Yokoyama and Schoon Ha. Film-nav: Efficient and generalizable navigation via vlm fine-tuning. arXiv preprint arXiv:2509, 16445,, 2025.
[78] Naoki Yokoyama, Ram Ramrakhya, Abhishek Das, Dhruv Batra, and Schoon Ha. Hm3d-ovon: dataset and benchmark for open-vocabulary object goal navigation. In 2024 IEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pages 5543-5550, IEEE, 2024.
[79] Bangguo Yu, Hamidreza Kasaci, and Ming Cao. L3mvn: Leveraging large language models for visual target navigation. In 2023 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 3554-3560. IEEE, 2023.
[80] Wenxuan Zeng, Meng Li. Haichuan Yang, Wen-jie Lu, Runsheng Wang, and Ru Huang. Copriv: network/protocol co-optimization for communication-effiieee private inference. Advances in Neural Information Processing Systems, 36:78906-789277 2023.
[81] Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. Long-clip: Unlocking the long-text capability of clip. In European conference on computer vision, pages 310-325. Springer, 2024.
[82] Jiazhao Zhang, Kunyu Wang, Shaoan Wang, Minghan Li, Haoran Liu, Songlin Wei, Zhongyuan Wang, Zhizheng Zhang, and He Wang. Uni-navid: A video-based vision-language-action model for unifying embodied navigation tasks. arXiv preprint arXiv:2412.06224, 2024.
[83] Jiazhao Zhang, Kunyu Wang, Rongtao Xu, Gengze Zhou, Yicong Hong, Xiaomeng Fang, Qi Wu, Zhizheng Zhang, and Wang He. Navid: Video-based vim plans the next step for vision-and-language navigation. arXiv preprint arXiv:2402. 15852, 2024.
[84] Jintao Zhang, Chendong Xiang, Haofeng Huang, Haocheng Xi, Jun Zhu, Jianfei Chen, et al. Spargeattention: Accurate and training-free sparse attention accelerating any model inference. In Forty-second International Conference Machine Learning.
15

---

[85] Junbo Zhang and Kaisheng Ma. Mg-vin: Benchmarking multi- and long-horizon vision- language navigation with language enhanced memory map. 2024 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 7750-7757. IEFF, 2024.
[86] Sixian Zhang, Xinhang Song, Yubing Bai, Weijie Li, Yakui Chu, and Shuqiang Jiang. Hierarchi- cal object-to-zone graph for object navigation. In Proceedings of the IEEE/CVF international conference computer vision, pages 15 1130-154400 2021.
[87] Sixian Zhang, Xinhang Song, Xinyao Yu, Yubing Bai, Xinlong Guo, Weijie Li, and Shuqiang Jiang. Hoz++: Versatile hierarchical object-to-zone graph for object navigation. IEEE Transac- lions on Pattern Analysis and Machine Intelligence, 2025.
[88] Sixian Zhang, Xinyao Yu, Xinhang Song, Xiaohan Wang, and Shuqiang Jiang. Imagine before go: Self- self-suuer generative map for object goal navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16414-16425, 2024.
[89] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Jeff Huang, Chuyue Sun, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. Efficiently programming large language models using sglang. arXiv preprint arXiv:2312.07104, 2023.
[90] Shuzhang Zhong, Zebin Yang. Ruihao Gong, Runsheng Wang, Ru Huang, and Meng Li. Propd: Dynamic token tree pruning and generation for IIm parallel decoding. In Proceedings of the 43rd IEEE/ACM International Conference on Computer-Aided Design, pages 1-8, 2024.
[91] Gengze Zhou, Yicong Hong, and Qi Wu. Navgpt: Explicit reasoning in vision-and-language navigation with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 7641-7649, 2024.
[92] Kaiwen Zhou, Kaizhi Zheng, Connor Pryor, Yilin Shen, Hongxia Jin, Lise Getoor, and Xin Eric Wang. Ese: Exploration with soft commonsense constraints for zero-shot object navigation. In International Conference on Machine Learning, pages 42829 42842. PMLR, 2023.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the work in Section 5 and Appendix !
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
This paper does not involve theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide detailed information about experiments the Section appendix and code.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification:
We will release of code and data.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https wwt.mmmssssooooooiiooooooo for more details. / //nips.cc/publ http:l///giiese//deuuoioiie
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify all the training and test details in Section appendix, and source code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We repeated experiments with several runnings and found error bar is relatively small.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
19

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provided sufficient information on the computer resources in the main text and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't ' make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httppp////eerrrs...cooooooootttt
Answer: [Yes]
Justification: The research conducted the paper conformed, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This paper discusses both potential positive societal impacts and negative societal impacts of the work in the Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We describe safeguards for responsible release of models in the broader impacts section.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly credited the creators original owners of assets (e.g., code, data, models), used in the paper and conformed the license and terms.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, papersswi swiprswitee thcode. thcoodee com/ datasttsss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
21

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification: submission.
We communicated the details of the dataset/codeemmodll part of their
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve study participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve study participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
22

---

for For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: Our paper does not involve LLMs as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non ntn-standard components.
Please refer our LLM policy what should or should not be described.
23

---

A LLM inference and KV Cache
The generative inference process of LLMs is divided into two stages, the prefill stage and the decode stage. In the prefill stage, the prompt sequence is taken input X, and GEMMs are performed to compute the query (Q X WW)) key (K = X W k), and value (V X Wv) matrices
for each Transformer block. These computed key and value matrices are cached as KV cache for
QK"
subsequent steps. The attention output is derived as Attn(Q, Q,QKKKW V) softmax
) which
VD
further produces the first output token.
The decode stage iteratively generates token per step via General Matrix-Vectte operations. At step 1, the newly generated token updates the cached Key and Value tensors through concatenation: K: concat K 1- new) and Ve concat 1-1 V new). The updated cache enables efficient computation of self-attention over the extended sequence, yielding the next token. This autoregressive process continues until termination. Both stages leverage cached key-value pairs avoid redundant computation.
B Object Navigation Flow
As discussed in Section 3.1 our ObjNav, the goal-finding process is composed of iterative sub-goal finding tasks until the final goal is reached. Figure IT shows example of our ObjNAv. After reaching sub-goal, the system generates semantic and distance information of the main objects from RGB-D B sensor captures using detection models. Next, this semantic and spatial information organized into the graph-base based navigation map, and the updated map and goal instructions are provided to the LLM for goal planning. Here also show example of the LLM prompt and output in Section C
MApeet 7 T2b6 poston 1114, 107.15 8 oosionn
Eioeo cins papiioant 15.2 350 Bipe doos posiion $2.00
Eoot roles dostion (8.51 200
south (181 (Mbpct dosk poo on 1150 951)
joaest pokiori
1283
project tir eigestanc pooline (781 54313
jrasel po (274-15256
pootions
(4eect 1: doox posilon/55A71.100 (1,1
Navigation Map Description: (nlywc 1: daoc, postion 71 (000))000ppp.... (object 2: solee, postion (9.91 28% (utject 3: sink, preftio 18.
74, 30%6, (object beiteube, poséitr:11é, 166 (05) dolject 8: dook posiion (68 60 222_59j./obped jobject paint, pooel ond (128, Aojjeejjed 7: table, postion: 114, 107, 1833 conject B: sofa, postion: 1131 94, 220 lobject B: couch. positon: (120, 125, 19%, (object 10: desk.
postiio:: (169, 100 (0l.2611.ppope (poject 11: chair, position: (190.6 360 (oaject 12: dressing, postoon]243 23 281), 280. joeject 13: nighssaand. poséllio::
(oojuct bed, posétion: (274, 48,25%
(more objects)
nevigetion
devecristion
sperslic object
sudl-goel
belone:
Trajectory: visited anject 1.6.7.10.
Outpuic The next subgoal bed position (274,)22228888880 48,25.
Figure 11: An example of our ObjNAv flow.
C Detailed Planning Example
We now further provide more detailed example illustrate the memory and prompt for EfficientNav.
1. Navigation Map Description: the navigation map records the objects and their position. To facilitate memory retrieval, organize these objects into groups. Hence, example of the navigation map is as follows:
24

---

Navigation Map Description
Object Group 1: (object: bathtubs, position::54.71,555)) (object: toilet, (9,11))):
(object: sink, position: (5, 74, 3,)), [object: towel, (160tton:(161055,0))
Object Group 2: {object: door, (6666)9):(66662.55))) (object: paint, position::125,27.999
{object: table, position:(114,1000 11,,007,,1)) (object: sofa, position:(131 (,9,22))) (object: couch,
position: (120,125,19)) (object: desk, position: (159,104,55), (object: chair, position:
(196,58,34)
Object Group [object:
dressing, position:(243.2322))) (object: nightstand, position:
(288, 54,31)), {object: bed, position: (274,48,25) (more groups)
Note that in order to reduce the computation cost of the navigation map, we compute the KV cache of each group only once and store the KV cache in memory. As the robot explores the environment, new objects will be added to different groups, whose KV cache will be updated accordingly.
2. Memory retrieval: for given target, we then leverage semantic-aware memory retrieval to select relevant groups from the navigation map. Then, the KV cache of corresponding groups will be loaded to device memory. Thereby, memory retrieval not only reduces the memory loading cost but also helps the LLM planner to focus on the relevant environment to make better decisions. Let's assume the Group 2 and 3 in the navigation map above are selected.
Planning: Given the retrieved KV cache, we now query the LLM plan for the exploration. The prompt for the LLM is as follows:
Prompt for Planning
Object Group 2: {object: door, (66,11on:(6662..59)) [object: paint, (12it,on:(12527,,,
{object: table, position:(114,107, [object: sofa, position: (131,94,22)), {object: couch,
position: (120,125,19)). fobject: desk, position: 105,5)04,55), (object: chair, position:
(196,58,34)
Object Group [object:
dressing, (o31t1oo::243,33229)) (object: nightstand, position:
(288, 54,31)), [object: bed, position: (274,48,25)
Instruction: You are a navigation robot. The above a description of different objects in the environment that you have seen. Your final goal is to find the TV in the environment. Based on the environmental information, please choose one specific object to travel to as your sub-goal, following such format: "The next subgoal is xxx position (xx, xx)". Here the objects that you have traveled before:
Trajectory: You have visited the door at position (66,62,59) and the dressing at position (243,23,29).
Given the prompt above, the LLM generates the following instruction:
Planner Output
The next subgoal is sofa position (131,94,22).
The robot will thereby follow the instruction to find the sofa at position (131, 94, 22). The process is repeated until the robot finds the target object.
D Network Architecture
We benchmark EfficientNav using four models: LLaVA-7B, LLaMA-3.2-11B. LLaVA-13B, and LLa LLaVB,34B, each comprising LLM backbone and ViT-based vision encoder. As detailed in Table 7 their architectures scale systematically.
For LLM Backbones, LLaVA-7b, LLaMA-3.2-11H LLaVA-13b, and LLaVA-34b adopt Mistral-7B- 13B-v1.5. 15, and Nous- Hermes-2-Vi-4443 respectively. Both the number of layers and hidden dimensions increase with model scale. For vision encoders, LLaVA
Instruct-v0.2. 1.L.MA3.1,, Vicuna- -oaun--1B-vee
25

---

variants utilize a 24-layer VIT with a hidden dimension of 1024, while LLaMA-3.2-11B employs deeper 32-layer ViT (hidden dimension of 1280). All vision encoders are fine-tuned during training.
Table 7: Network architecture
LLM backbone
Vision Encoder
Model
LLM
# Layers
Hidden Dim
# Layers
Hidden Dim
LLaVA-7b
Sestral-7B-Instruct--000
32
4096
24
1024
LLaMA-3.2-111
LLaMA-3.1
40
4096
32
1280
LLaVA-138
(icuna-13b-lle
40
5120
24
1024
LLaV4-34b
Nous-Hermes-2-Yi-ee
60
7168
24
1024
E CLIP Dependency
In semantics-aware retrieval, we use CLIP to remove redundant information and select rele- vant groups before sub-goal selection. CLIP is
Table 8: Ablation study on group selection method.
widely used in semantic matching 770 M.
Method
SR
SPL
Real-time Latency
In fact, as shown in Table 8 other semantic matching models can also be used and show rea- sonable performance (e. (5.. MiniLM (64)). showing the robustness of our method. While the LLM-based selection method shows
CLIP (ours)
80.0
41.5
0.87s
MPNet [57
79.3
39.6
0.93s
MiniLM [64
80.3
40.1
0.84s
LLM
80.7
42.1
6.55s
high real-time latency because of the high computation cost, with no significant success rate improve- ment. This is because CLIP is enough to remove most of the redundant information and keep the most relevant groups. Even a little redundant information is kept, the LLM planner can exclude in the sub-goal selection period. So light- semantic matching model more reasonable.
F Open-vocabulary Evaluation
In Section we follow existing works use the HM3D dataset for comparison. However, hope emphasize that EfficientNav is not limited to HM3D and can applied to open- vocabulary datasets. Here we evaluate our method on HMBD-OVON (78), which has more target object categories. The result is shown in Table 9 and our method also shows strong performance open-vocabulary setting.
Table Evaluation on HM3D-OVON.
Method
SR
SPL
Uni-NaVid 82
41.3
21.1
FiLM-Nav 7
44.9
24.5
EfficientNav (ours)
63.5
31.6
G Implementation Details
In EfficientNav. smaller open-source planners, such as LLaVA and LLaMA3.2. We use single NVIDIA RTX A6000 GPU to deploy LLaVA-7b and LLaMA3.2-113 When using LLaVA-133 and LLaVA-34b, deploy our system on 2 and 4 NVIDIA RTX A6000 GPUs, respectively. We also deploy LLaVA-1a on a single Jetson AGX Orin. For the detection model, we use Grounding Dino (37).
H Comparison with Other Graph-based Method
Graph-based navigation maps are increasingly favored for their ability to explicitly represent the semantics and locations of visited objects. Among them, HOZ [86] proposes hierarchical object-to- zone (HOZ) graph to provide coarse-to-fine guidance for navigation robots in unseen environments, which is constructed from scene, zone, and object nodes, updated online based on real-time observa-
tions. Based on HOZ, HOZ++ [87] introduces adaptive graph structure and heterogeneous graph
26

---

fusion for the graph to further enhance object navigation performance. LROGNav [60] enhances nav- igation efficiency by integrating common- sense knowledge of object-to-room relationships extracted from large language models using both positive and negative chain-of-thought promptings, and by training multi-channel Swin-UNet with multimodal inputs. Imagine Before Go [88] introduces Self- self-supervised Generative Map (SGM) and improves exploration efficiency by learning explicit contextual relationships between objects and environments through self-supervised training. It lever- aging both episodic observations and general knowledge from Large Language Models to generate unobserved regions local semantic maps.
Our method EfficientNav improves the efficiency of LLM-based object goal navigation by navigation map caching and retrieval. Different from these works, our method focuses on the computation and management of KV cache for the navigation map description. We use discrete memory caching to decouple the context order and KV cache computation. For object grouping, we use attention-based memory clustering to reduce the impact of ignoring cross-attention. These methods enable KV cache reuse and significantly accelerate the prefilling stage of L.M planning.
I Limitations
In this section, we discuss the current limitations and potential avenues for future research.
First, in our method, we propose EfficientNav, enabling efficient zero-shot on-device ObjNAv. In fact, our memory caching and memory retrieval method also benefits the LLM inference on the cloud server by avoiding re-computation and removing redundant information. However, EfficientNav can not solve the problems of high communication latency privacy concerns that cloud serving faces. Also, in our experiment, EfficientNav achieves 6.7x real-time latency reduction compared with GPT-4 planner. However, as the inference speed of LLM can not reach that of small models, EfficientNav should be carefully used in applications that need extremely low real-time latency.
J Broader Impacts
In our paper, our method enables efficient on-device zero-shot indoor ObjNav, overcoming the tight memory constraints by designing novel memory caching and retrieval mechanisms. However, our LLM-based ObjNav works in zero-shot manner. This avoid the heavy training cost, but the pre-trained LLM may not master some highly specialized knowledge. Therefore, in high-risk areas such chemical laboratories with hazardous materials, EfficientNav should be used carefully.
27


Reviews:
[{"review_id": "arlYEsDXAH", "reviewer": "Reviewer_gaGt", "summary": "This paper presents an efficient LLM based object goal navigation approach with navigation map cashing and retrieval.", "questions": "See Weakness\n\nSection 3.3 and 3.4 is a little difficult to grasp. It would be nicer to use Figure 5 and tell how LLM semantic space and 2D map is associated.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength:\n\n- The motivation toward on device LLM based object goal navigation is valuable. \n\n- The proposed method achieves higher success rate than baseline models with relatively low latency.\n\n\nWeakness:\n\n1) The task setting is specialized: the goal object is defined only three. And it does not do vision language navigation which is tackled by its LLM baseline techniques such as MapGPT. Evaluating on object goal navigation with the same number of target object as in previous works and/or VLN would be appreciated.  \n \n2) It talks about Jetson but there is no robot hardware deployment experiment. \n\n3) I wonder how the general adaptive sparse attention approaches work on the object goal navigation task, since its not predefined so it could in theory adapt to this kind of task too. Comparisons to the model using dynamic sparse attention like these without the proposed semantics aware memory retrieval would be appreciated.\n\n- Minference\n\n- FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inference\n\n- SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inference\n\n4) It lacks some object goal navigation references that relate target and zone/room.\n\n- Hierarchical Object-to-Zone Graph for Object Navigation\n\n- Enhancing Multimodal-Input Object Goal Navigation by Leveraging Large Language Models for Inferring Room-Object Relationship Knowledge\n\n- HOZ++: Versatile Hierarchical Object-to-Zone Graph for Object Navigation", "was_revised": true, "final_justification": "While the proposed framework may not be on-the-fly deployable on real robots, most of my concerns are resolved and I found its attention based clustering interesting.", "rebuttal": "1.1 **Limited number of goal objects:** we appreciate the reviewer's attention to our experimental setup. However, we would like to clarify that our work actually **evaluates six object categories** (toilet, TV, chair, sofa, bed, and plant), strictly following the setting in the HM3D dataset (in our paper, we only show three of the goals at Line 243 as an example). In addition, to show the robustness of our method, we show the success rate of each individual object goal here. **As our method is actually open-vocabulary, we further evaluate 4 more object goals (table, door, sink, lamp) to show our generalization.** The result shows that our method achieves a high success rate on all object goals.\n| Object  | Toilet | TV    | Chair  | Sofa  | bed   | plant | table | door | sink  | lamp  |\n|---------|--------|-------|--------|-------|-------|-------|-------|------|-------|-------|\n| SR      | 72.5%  | 75.0% | 87.5%  | 90.0% | 85.0% | 70.0% | 95.0% | 100% | 72.5% | 82.5% |\n\n1.2 **MapGPT evaluation:** we also evaluate MapNav on the HM3D ObjNav dataset. According to the following table, our method achieves 14.5% SR and 6.3% SPL improvements over MapGPT, as our method removes redundant information in the prompt using semantics-aware memory retrieval. We also achieve 7.0x real-time latency reduction by using memory caching and retrieval.\n| Method        |   SR   |  SPL   | Real-time latency |\n|:-------------:|:------:|:------:|:----------------:|\n| MapGPT        |  65.5  |  35.2  |      6.08s       |\n| EfficientNav  |  80.0  |  41.5  |      0.87s       |\n\n2 **Lack of robot deployment:** We utilize a Jetson Orin as the compute device, a common choice for on-board computing hardware in robotics, and conduct experiments within the Habitat simulator. Specifically, the simulator runs on a separate computer, sending sensor data to the Jetson Orin, which hosts the LLM planner. The Jetson Orin then generates the plan and transmits it back to the separate computer, forming a closed-loop system. We would like to clarify that **simulation-based evaluation is a widely used practice** in object-goal navigation research, as evidenced by many prior works (e.g., [1, 2, 3]). Our experimental setup follows the same standard, ensuring fair comparison with existing methods while maintaining reproducibility.\n\n3 **Evaluation on sparse attention approaches:** the goals of general adaptive sparse attention and EfficientNav are different: EfficientNav clusters objects into different groups and calculates the attention of each group individually. **This structured sparse approach can decouple the group order and KV cache computation of each group, thus enabling KV reuse when we retrieve different groups in navigation planning.** However, for general adaptive sparse attention methods, **their purpose is to minimize the difference between sparse attention and full attention** and accelerate the attention calculation. But without memory retrieval, even full attention shows a low success rate, as discussed in Section 3.1. At the same time, without discrete memory caching, these methods need to recompute the KV cache of the navigation map description because of map changes, thus showing longer real-time latency.\n\nWe conduct experiments to compare these sparse attention works with our method. Our method outperforms all the baselines. We will cite all these papers and compare them with EfficientNav in our final version.\n| Method        |   SR   |  SPL   | Real-time latency |\n|:-------------:|:------:|:------:|:----------------:|\n| Minferenc     |  35.3  |  18.0  |3.88s|\n| FlexPrefill   |  36.7  |  20.1  |3.02s|\n| EfficientNav  |  80.0  |  41.5  |    0.87s       |\n\n4 **Lack of references**: we thank the reviewer for their valuable suggestion to refer to these foundational works. \n\n**HOZ** innovatively proposes a hierarchical object-to-zone (HOZ) graph to provide coarse-to-fine guidance for navigation robots in unseen environments, which is constructed from scene, zone, and object nodes, updated online based on real-time observations.\n\nBased on HOZ, **HOZ++** cleverly introduces an adaptive graph structure and heterogeneous graph fusion for the graph to further enhance object navigation performance.\n\n**LROGNav** enhances navigation efficiency by integrating common-sense knowledge of object-to-room relationships extracted from large language models using both positive and negative chain-of-thought promptings, and by training a multi-channel Swin-UNet with multimodal inputs.\n\nWe also find a paper called **Imagine Before Go**, which introduces a Self-supervised Generative Map (SGM) and improves exploration efficiency by learning explicit contextual relationships between objects and environments through self-supervised training. It leveraging both episodic observations and general knowledge from Large Language Models to generate unobserved regions of local semantic maps.\n\nOur method **EfficientNav** improves the efficiency of LLM-based object goal navigation by navigation map caching and retrieval. Different from these works, **our method focuses on the computation and management of KV cache for the navigation map description.** We use discrete memory caching to decouple the context order and KV cache computation. For object grouping, we use attention-based memory clustering to reduce the impact of ignoring cross-attention. These methods enable KV cache reuse and significantly accelerate the prefilling stage of LLM planning.\n\nWe will cite all these papers and compare them with EfficientNav in our final version.\n\n5 **Writing unclarity**: we apologize for the writing unclarity. Here we re-explain how the LLM semantic space and 2D map are associated. First, as shown in Figure 2, using detection models in the navigation process, we can construct a navigation map and add newly detected objects into the map in each navigation step.  The navigation map records the semantics and position of each object. Second, as shown in Figure 5, we use attention-based memory clustering to cluster each newly detected object into related groups. This can reduce the impact of ignoring cross-attention between different groups. Note that the clustering process is based on the LLM attention, which is not directly determined by the object position on the map. Third, also shown in Figure 5, we retrieve groups related to the current goal object, and feed the information of these groups into the LLM planner as prompt. Then, we load the KV cache of related groups calculated by discrete memory caching to avoid re-computation, and the KV cache of newly detected objects in retrieved groups is calculated. Finally, the LLM planner generates the next sub-goal.\nWe will fix our writing in the final version.\n\n[1] Shah, Dhruv, et al. \"Navigation with large language models: Semantic guesswork as a heuristic for planning.\" \n\n[2] Long, Yuxing, et al. \"Instructnav: Zero-shot system for generic instruction navigation in unexplored environment.\" \n\n[3] Wu, Pengying, et al. \"Voronav: Voronoi-based zero-shot object navigation with large language model.\""}, {"review_id": "x46zGmRVIb", "reviewer": "Reviewer_QLLB", "summary": "EfficientNav is proposed to address two major challenges in applying large language models (LLMs) to on-device Object Navigation (ObjectNav). The first challenge lies in the difficulty of deploying large LLMs on resource-constrained devices. The second issue concerns the growth of the key-value (KV) cache, which increases significantly as context information accumulates throughout the navigation process. To tackle these challenges, the authors introduce a combination of discrete memory caching and a cluster-based navigation map, allowing for more efficient estimation and control of KV cache usage. Additionally, they propose a semantic-aware memory retrieval module, which enables the system to reduce unnecessary map information by selecting only semantically relevant memory entries for reasoning and decision-making.", "questions": "1. What is the basis for clustering groups? Is it based on semantics or other metrics?\n\n2.  In the supplementary navigation examples, does each node represent only one object?\nIn Figure 11, object 7 and object 10 appear to belong to the same node, but in the trajectory, both objects appear separately. This seems contradictory.\n\n3. How is the sub-goal selected?\nSection 3.4 describes how relevant groups are selected, but since each group contains multiple objects, how is the final sub-goal selected from within a group?", "limitations": "- The paper lacks detailed architectural descriptions, particularly regarding components such as the object detection module. This omission makes it difficult to fully understand or reproduce the system pipeline. \n- The empirical evaluation is limited to indoor simulation environments. There is little analysis of how well the proposed method generalizes to more complex, real-world, or large-scale navigation scenarios.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- The proposed Attention-based Memory Clustering and Semantic-aware Memory Retrieval effectively enhance the usability of LLMs for on-device navigation tasks.\n- These two components enable the model to identify relevant features based on their relation to the current sub-goal, improving both retrieval efficiency and navigation accuracy.\n\n### Weaknesses\n- The overall architecture of the model is not fully described in the paper. And make the reader a little bit confusing. For example, it does not state which detection model is used for constructing navigation map.", "was_revised": true, "final_justification": "This paper introduces an efficient navigation model for indoor navigation and I recommend this paper for borderline accept.", "rebuttal": "1 **Model architecture confusion:** We use **grounding-dino** as our detection model. \n**While the choice of detection model impacts the semantic map's accuracy, it does not affect the core contribution or the primary problem addressed in this paper.**\nHere we choose 3 detection models to evaluate our system, and the final success rate does not change much. Note that **we mainly focus on the planning process of object navigation, which represents the primary efficiency bottleneck, rather than the perception process.** We will clearly explain this in our final version.\n| Detection Model  |   SR   |  SPL   |\n|:----------------:|:------:|:------:|\n| Grounding-dino   |  80.0  |  41.5  |\n| OWL-ViT         |  80.3  |  41.2  |\n| Detic           |  79.0  |  41.1  |\n\nHere we re-explain the working process of our navigation system. First, as shown in Figure 2, using detection models in the navigation process, we can construct a navigation map and add newly detected objects into the map in each navigation step.  The navigation map records the semantics and position of each object. Second, as shown in Figure 5, we use attention-based memory clustering to cluster each newly detected object into related groups. This can reduce the impact of ignoring cross-attention between different groups. Third, also shown in Figure 5, we retrieve groups related to the current goal object, and feed the information of these groups into the LLM planner as prompt. Then, we load the KV cache of related groups calculated by discrete memory caching to avoid re-computation, and the KV cache of newly detected objects in retrieved groups is calculated. Finally, the LLM planner generates the next sub-goal.\n\n2 **Basis for clustering groups:** As discussed in Section 3.3, we use the **LLM attention distribution** to cluster the groups. When new objects are detected, we input the information of existing groups and the newly detected objects into LLM. The information contains **both semantics and positions** of the groups/objects. And we infer a few layers of the LLM and cluster object based on the attention score between newly detected objects and existing groups. This can reduce the impact of ignoring cross-attention between groups. In conclusion, we use the attention of LLM itself to cluster the newly detected objects into different groups, both considering semantics and positions. We also evaluate clustering only based on position (cluster the new object into the nearest group) and semantics (use CLIP to calculate the similarity between new objects and existing groups), and we find our attention-based method achieves the best performance.\n| Method              |   SR   |  SPL   |\n|:-------------------:|:------:|:------:|\n| position-based      |  45.3  |  23.4  |\n| Semantic-based      |  70.8 | 34.6  |\n| **Attention-based (ours)** |  80.0  |  41.5  |\n\n3 **Node represents:** in this figure, one node **does not** represent only one object. To be specific, multiple objects related to one node only means that these objects are near the same place. It does not mean all these objects are at the center of the node. Instead, there is some distance between them, so we can visit different objects related to the same node separately. Many existing works use a similar representation manner [3, 4, 5, 7].\n\n4 **The way sub-goal is selected:** the final sub-goal is selected **by the LLM planner**. To be specific, **the sub-goal is determined in a two-step manner**: first, we select the relevant groups. Second, we select a sub-goal in the relevant groups. Both steps are important because if we directly select the final sub-goal among all groups, the planning performance will drop because of the huge amount of redundant information, as discussed in Section 3.1. \n\nFor the first step, we use CLIP to select relevant groups for better efficiency, as the task of removing redundant information is much easier than finding a specific sub-goal. For the second step, we use LLM to select the final sub-goal among the relevant groups. We also evaluate the result when using CLIP or LLM in both steps. When both are using CLIP, the accuracy drops a lot, as CLIP has a lower reasoning ability than LLM. When both are using LLM, the real-time latency becomes very long, as LLM shows higher inference latency than CLIP. And there is no significant success rate gain, as the CLIP model is enough to select relevant groups.\n| Method          |   SR   |        SPL       | Real-time Latency |\n|:---------------:|:------:|:----------------:|:-----------------:|\n| CLIP+CLIP       | 5.3  | 1.1            | 0.19s             |\n| LLM+LLM         | 80.7   | 42.1             | 6.55s            |\n| **CLIP+LLM (ours)** | 80.0   | 41.5        | 0.87s        |\n\n5 **Evaluation environment limited**: we would like to clarify that **EfficientNav is specially designed for indoor navigation.** We highlight that **indoor navigation is an important research area,** and is adopted in many applications, such as household robots and healthcare robots. **Many existing works also focus on indoor navigation** [1, 2, 3, 4, 5, 6, 7].\nAnd **simulation-based evaluation is also a widely used practice in object-goal navigation research**, as seen in many prior works (e.g., [1, 2, 3, 4, 5, 6, 7]). Our experimental setup follows the same standard, ensuring fair comparison with existing methods while maintaining reproducibility.\n\n[1] Shah, Dhruv, et al. \"Navigation with large language models: Semantic guesswork as a heuristic for planning.\" \n\n[2] Long, Yuxing, et al. \"Instructnav: Zero-shot system for generic instruction navigation in unexplored environment.\" \n\n[3] Wu, Pengying, et al. \"Voronav: Voronoi-based zero-shot object navigation with large language model.\" \n\n[4]Chen, Jiaqi, et al. \"Mapgpt: Map-guided prompting with adaptive path planning for vision-and-language navigation.\" \n\n[5]Zhang, Sixian, et al. \"Hierarchical object-to-zone graph for object navigation.\" \n\n[6]Sun, Leyuan, et al. \"Leveraging Large Language Model-based Room-Object Relationships Knowledge for Enhancing Multimodal-Input Object Goal Navigation.\" \n\n[7]Zhou, Gengze, Yicong Hong, and Qi Wu. \"Navgpt: Explicit reasoning in vision-and-language navigation with large language models.\""}, {"review_id": "0Drr7Ef7u8", "reviewer": "Reviewer_LqYc", "summary": "The authors propose discrete memory caching for object-goal navigation, whereby the map information ca split into segments and the KV cache can be more efficiently decomposed and stored. This solution mitigates the problem of constrained systems not having enough memory to store entire KV caches. Since this clustering into discrete segments may negatively impact performance due to problems with cross-attention across segments, the authors propose attention-based memory clustering.", "questions": "For semantic retrieval, do you think something like a knowledge graph or a scene graph can be used to augment or replace your current CLIP-based method?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* The authors include a nice empirical validation (Figure 6, a) that the latency incurred by loading the KV cache repeatedly in the conventional way is large.\n* Semantics-aware memory retrieval is used by leveraging a CLIP to compute semantically similar inputs, which seems reasonable.\n* Their experimental results on improving latency over the state-of-the-art are impressive.\n\nWeaknesses:\n* Some well-known related work on navigation using LLMs is missing, including the line of work on integrating scene graphs and LLMs for navigation, see, e.g., [1] below.\n[1] Rajvanshi, Abhinav, et al. \"Saynav: Grounding large language models for dynamic planning to navigation in new environments.\" Proceedings of the International Conference on Automated Planning and Scheduling. Vol. 34. 2024.", "was_revised": true, "final_justification": "The paper is very interesting and could enable low-SWaP or edge navigation in cloudless environments through the proposed KV caching. I am content with this paper being accepted.", "rebuttal": "1 Related work referring: we thank the reviewer for their valuable suggestion to refer to these foundational works. \n**SayNav** novelly uses a grounding mechanism, building a 3D scene graph of the explored environment as input to LLMs, for generating feasible and contextually appropriate high-level plans for navigation. Our method **EfficientNav** improves the efficiency of LLM-based object goal navigation by navigation map caching and retrieval. Different from SayNav, our method focuses on the computation and management of KV cache for the navigation map description. We use discrete memory caching to decouple the context order and KV cache computation, which enables KV cache reuse. And we also use attention-based memory clustering to reduce the impact of ignoring cross-attention. We will cite these papers and compare them with EfficientNav in our final version.\n\n\n2 Retrieval using knowledge graph: **a knowledge graph can be used to augment our method, while our CLIP-based method is also necessary.**  Existing works attempt to use a knowledge graph to embed prior knowledge, such as object relationships, to guide the navigation in unseen environments. In the navigation process, a knowledge graph can provide correlated objects as auxiliary information to locate the target object [1, 2]. However, **our CLIP-based method is still necessary in the following aspects**:\n\n(1) Better generalization: knowledge graph requires manually curated or statistically learned object co-occurrence priors. And the navigation performance is highly related to the built knowledge graph, limiting generalization to unseen objects/scenes. Some works combine knowledge graphs with online learning, while the graph is still based on a fixed initial graph [3].\nWhile our CLIP-based method can enable zero-shot matching after large-scale pre-training. And our navigation map is totally built from scratch, which can better adapt our method to new scenes.\n\n(2) High flexibility in similarity computation: in existing works, the knowledge graph is mainly used to evaluate the relationship between different objects. However, the CLIP-based method has higher flexibility, which can both evaluate object-object similarity and object-group similarity. Especially, the group is dynamic in our method, which further shows the flexibility of the CLIP-based method.\n\nAt the same time, **Knowledge Graph Methods have their own strength**:\n\n(1) Better efficiency: CLIP requires text encoding and similarity computation, which will introduce extra latency. While a knowledge graph only queries a precomputed graph. **However, in indoor navigation scenes, the number of groups and objects is not too large, and the latency caused by CLIP is not the bottleneck.** So we use a CLIP-based method to pursue a higher success rate. We will consider using a knowledge graph in our future work for larger-scale navigation. Thank you for your insightful comments.\n\n[1]Yang, Wei, et al. \"Visual semantic navigation using scene priors.\"\n\n[2]Xu, Nuo, et al. \"Aligning knowledge graph with visual perception for object-goal navigation.\"\n\n[3]Zhang, Sixian, et al. \"Hierarchical object-to-zone graph for object navigation.\""}, {"review_id": "lSGQKv3ryq", "reviewer": "Reviewer_hd8c", "summary": "This paper presents EfficientNav, a system for enabling efficient zero-shot object-goal navigation (ObjNav) on local devices with limited memory resources. The key challenge addressed is that existing LLM-based navigation systems rely on cloud-based large models like GPT-4, which suffer from high latency and privacy concerns. EfficientNav introduces three main contributions: (1) discrete memory caching to avoid recomputing KV cache for navigation map descriptions, (2) attention-based memory clustering to group related objects and reduce cross-attention loss, and (3) semantics-aware memory retrieval using CLIP to select relevant object groups based on navigation goals. The system achieves 11.1% improvement in success rate over GPT-4 baselines while providing 6.7× real-time latency reduction.", "questions": "Questions\n\nEffectiveness of Discrete Memory Caching: Table 4 has shown that the Discrete Memory Caching module achieves 0.4% SR improvement. However, as Section 3.2 shows, this is mainly a design to solve the memory constraints and will cause the ignorance of cross-attention between groups, so how can this module increase the SR metric, or is this just a statistical deviation? \n\nPerformance difference between discrete attention and full attention: As the authors put in Section 3.2 and 3.3, the Attention-based Memory Clustering method serves as a complement of the Discrete Memory Caching method to make up for the ignorance of cross-attention between groups. Then why will this discrete attention scheme (Discrete Memory Caching + Attention-based Memory Clustering) outperforms the full attention scheme (LLaVA Planner) by 20.2% in SR as shown in Table 4? The text descriptions in Section 4.3 does not show how this discrete attention could achieve such a performance gain.\n\nThe changes of semantic groups: Could the authors provide an example of how the semantic groups are updated as the agent is exploring the scene? And as the paper shows, the newly-added object will be added to existing groups based on the semantic similarity, so will it be a problem that some semantically-related but spatially-distant objects be clustered in a group? If yes, how will this influence the final navigation performance?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nImportant Problem Identification:\n\nThis paper addresses a critical challenge in LLM/VLM-based navigation, real-time performance, which is a common bottleneck for many navigation methods. This issue is especially pivotal for enabling the practical deployment of such methods in real-world applications.\n\nTechnical Contributions: \n\n1. Discrete Memory Caching: This module effectively addresses memory constraints by clustering navigation map information into discrete groups, independently calculating their KV cache, and efficiently reusing cached data to reduce recomputation and transfer overhead.\n\n2. Attention-based Memory Clustering: This approach leverages LLM attention mechanisms to cluster semantically related map information, thereby significantly improving the effectiveness of memory retrieval and reducing the adverse impact of ignoring cross-attention.\n\n3. Semantics-aware Memory Retrieval: By utilizing semantic similarity via the CLIP model, this module dynamically selects relevant map groups based on the navigation task, enabling efficient memory usage and significantly enhancing planning performance of smaller LLMs.\n\nThorough analysis of memory and efficiency of different LLM/VLMs: This paper clearly presents the memory requirements of different LLM/VLMs and their latency in Figure 1. It also evaluates the latency of different navigation stages in Figure 3 (b) and Figure 9. This thorough evaluation is valuable as a guidance to the design of LLM/VLM-based navigation methods.\n\nEvident performance improvement: The paper provides extensive experiments on the HM3D benchmark, demonstrating significant improvements over existing methods, notably 11.1% success rate improvements over GPT-4-based baselines and substantial latency reductions.\n\nWeaknesses\n\nAttention-Based Clustering Justification: The choice of using only 1/10 of LLM layers for attention-based clustering lacks theoretical justification. Why this specific fraction? How sensitive is performance to this hyperparameter?\n\nCLIP Dependency: The semantics-aware retrieval relies heavily on CLIP's semantic understanding. The paper doesn't analyze failure cases where CLIP's similarity scores might be misleading for navigation decisions. The contribution of CLIP to the final navigation success rates is not discussed. Is this key to the 16.7% success improvement? If not using CLIP or using other similar models, how much will the performance drop?\n\nLimited Baseline Comparisons: This paper primarily compares against naive LLM planners and GPT-4. Comparisons with other efficient LLM inference methods would be valuable.\n\nPractical deployment problems: The practical deployment complexities of integrating multiple sophisticated methods might require considerable development effort for real-world scenarios.\n\nLack of detailed ablation studies: The ablation study does not show the influence and detailed analysis of the ablation choices, such as the influence of CLIP model, the choice of 1/10 layers of LLMs and how the discrete attention scheme outperforms the full attention scheme.", "was_revised": true, "final_justification": "Author's clarifications have addressed my concerns, and I am maintaining my positive rating.", "rebuttal": "1 **Ablations for attention-based clustering:**  In practice, we find that the success rate is not very sensitive to this hyperparameter. **We evaluate different choices and find the results reasonable, except for using only 1 layer.** As the attention distribution is relatively uniform in the first layers. We set this threshold of using 1/10 layers to make a trade-off between latency and success rate.\n| Number of Layers |   SR   |  SPL   | Real-time Latency |\n|:----------------:|:------:|:------:|:-----------------:|\n| 1                | 70.0   | 34.8   | 0.82s             |\n| 2                | 78.7   | 41.0   | 0.84s             |\n| 4                | 79.7   | 41.5   | 0.87s             |\n| 6 (1/10 layer)| 80.0 | 41.5 | 0.87s         |\n| 8                | 79.3   | 40.6   | 0.87s             |\n\n2 **CLIP Dependency:** The core idea of semantics-aware retrieval is that **we need to remove redundant information before giving the environment description to the LLM planner, thus leading to success rate improvement** (Section 3.1 and 3.4). Here, we select CLIP to remove redundant information and select relevant groups because CLIP is widely used in semantic matching [1, 2, 3]. In fact, **other semantic matching models can also be used and show reasonable performance (e.g.,all-mpnet-base-v2, all-MiniLM-L6-v2), showing the robustness of our method.** We also compared our method with the position-based method (select groups according to average object distance) and the LLM-based method (remove redundant information and select final sub-goal both using LLM). For the position-based method, we evaluate both considering/not considering device memory capacity in memory retrieval, which is discussed in Line 223. Position-based method shows a low success rate as it does not consider semantics. And the LLM-based method shows high real-time latency because of the high computation cost, with no significant success rate improvement. This is because CLIP is enough to remove most of the redundant information and keep the most relevant groups. Even if a little redundant information is kept, the LLM planner can exclude them in sub-goal selection. So a light-weight semantic matching model is important.\n| Method                                      |   SR   |  SPL   | Real-time Latency | \n|:-------------------------------------------:|:------:|:------:|:-----------------:|\n| position-based (without considering device memory capacity in memory retrieval)    | 63.3   | 34.2   | 2.32s           | \n| position-based (considering device memory capacity in memory retrieval)  | 47.5   | 25.8   | 0.81s           | \n| **CLIP (ours)**                             | 80.0 | 41.5 | 0.87s      |\n| all-mpnet-base-v2                           | 79.3   | 39.6   | 0.93s         | \n| all-MiniLM-L6-v2                            | 80.3   | 40.1   | 0.84s         | \n| LLM                                         | 80.7   | 42.1   | 6.55s        | \n\n3 **Evaluation on efficient LLM inference methods:**\nOur discrete memory caching method is similar to sparse attention methods. As there is no existing sparse attention method that focuses on navigation, we compare with general adaptive sparse attention methods.\n**There is a core difference between general adaptive sparse attention and EfficientNav:** EfficientNav clusters objects into different groups and calculates the attention of each group individually. **This structured sparse approach can decouple the group order and KV cache computation of each group, thus enabling KV reuse when we retrieve different groups in navigation planning.** However, for general adaptive sparse attention methods, **their purpose is to minimize the difference between sparse attention and full attention**, and accelerate the attention calculation. But without memory retrieval, even full attention shows a low success rate, as discussed in Section 3.1. At the same time, without discrete memory caching, these methods need to recompute the KV cache of the navigation map description because of map changes, thus showing longer real-time latency.\n\nWe conduct experiments to compare these sparse attention works with our method. Our method outperforms all the baselines. We will cite all these papers and compare them with EfficientNav in our final version.\n| Method        |   SR   |  SPL   | Real-time latency |\n|:-------------:|:------:|:------:|:----------------:|\n| Minferenc     |  35.3  |  18.0  |3.88s|\n| FlexPrefill   |  36.7  |  20.1  |3.02s|\n| EfficientNav  |  80.0  |  41.5  |    0.87s       |\n\n4 **Lack of real-world deployment:** \nAlthough we did not evaluate on a real-world robot, **we deployed our model on the computation platform (Jetson Orin) that is frequently used by real-world robots**, as our method mainly solves the problem of high computation cost in prefilling by optimizing KV cache management.\nAnd we would like to clarify that **simulation-based evaluation is also a widely used practice in object-goal navigation research, as seen in many prior works (e.g., [4, 5, 6]).** Our experimental setup follows the same standard, ensuring fair comparison with existing methods while maintaining reproducibility.\n\n5 **Effectiveness of Discrete Memory Caching**: only using discrete memory caching will not affect the success rate much, and the SR improvement is just a statistical deviation.\n\n6 **Effectiveness Attention-based memory Clustering:** the planning performance is highly related to the objects selected for the planner. And **the object selection is related to (1) how to cluster objects and (2) how to select groups.**\n\n As discussed in Line 322, in LLaVA planner baseline, we also cluster objects and select groups based on position, following [4, 7] (when not using selection, the success rate is only 36.7% caused by the redundant information). However, as the selected objects have poor correlation with each other, the success rate is still low whether using full attention or discrete attention. **After attention-based memory clustering is used, objects in each selected group have high correlation, which helps the LLM to better understand the environment and reduces the impact of ignoring cross-attention at the same time, thus improving the success rate.** When using semantics-aware memory retrieval, we can select the most related groups, which further improves the success rate. And our semantic-aware memory retrieval considers memory capacity, which is not considered by [4, 7], thus showing lower real-time latency. We will add the detailed explanation in our final version.\n\n7 **The changes of groups:** We conduct object clustering based on **both semantics and position**. As discussed in Section 3.3, we use the **LLM attention distribution** to cluster the groups. When new objects are detected, we input the information of existing groups and the newly detected objects into LLM. The information contains both semantics and positions of the groups/objects. And we infer a few layers of the LLM and cluster object based on the attention score between newly detected objects and existing groups. This can reduce the difference between discrete attention and full attention. In conclusion, **we use the attention of LLM itself to cluster the newly detected objects into different groups, both considering semantics and positions.** We also evaluate clustering only based on position (cluster the new object into the nearest group) and semantics (use CLIP to calculate the similarity between new objects and existing groups), and we find our attention-based method achieves the best performance.\n| Clustering Method  |   SR   |  SPL   |\n|:------------------:|:------:|:------:|\n| position-only      | 45.3   | 23.4   |\n| semantic-only      |  70.8 | 34.6     |\n| **attention-based** | 80.0 | 41.5 |\n\nIt is true that some semantically related but spatially distant objects can be clustered in a group, e.g., objects in two different bathrooms. However, **this is reasonable as these objects are usually both relevant/irrelevant to the current task.** In the planning process, both the position and semantics of each selected object will be given to the LLM, and the LLM will analyze the environment and choose only one object as sub-goal.\n**We will add a figure to show an example of grouping updating in our final version, as adding figures in rebuttal is not allowed.**\n\n[1]Wu, Xiaoshi, et al. \"Cora: Adapting clip for open-vocabulary detection with region prompting and anchor pre-matching.\" \n\n[2]Zhang, Beichen, et al. \"Long-clip: Unlocking the long-text capability of clip.\" \n\n[3]Dorbala, Vishnu Sashank, et al. \"Clip-nav: Using clip for zero-shot vision-and-language navigation.\" \n\n[4] Shah, Dhruv, et al. \"Navigation with large language models: Semantic guesswork as a heuristic for planning.\" \n\n[5] Long, Yuxing, et al. \"Instructnav: Zero-shot system for generic instruction navigation in unexplored environment.\" \n\n[6] Wu, Pengying, et al. \"Voronav: Voronoi-based zero-shot object navigation with large language model.\" \n\n[7]Zhang, Junbo, and Kaisheng Ma. \"MG-VLN: Benchmarking Multi-Goal and Long-Horizon Vision-Language Navigation with Language Enhanced Memory Map.\""}]

Author rebuttal:

Meta-review: This paper proposes EfficientNav for on-device efficient LMM zero-shot ObjNav. Main contributions of the work are (1) discrete memory cashing to avoid recomputing KV cash (2) attention-based memory clustering and (3) semantic-aware memory retrieval to select relevant object groups based on navigation goal. Experiments show 11.1% improvement in success rate over GPT-4, while having 6.7x latency reduction. To further strengthen the paper authors should address all feedback of the reviewers, specifically improve overall writing/clarity, including more details on the overall architecture and model so it can be easy for reproduction; comparisons against baselines; practical deployment and evaluation in real world;
Decision: Accept (poster)