Title: EfficientNav: Towards On-Device Object-Goal Navigation with Navigation Map Caching and Retrieval
Venue: NeurIPS 2025
Accepted: yes
Keywords: Object-Goal Navigation

Navigation Map

Memory Caching

Memory Retrieval
Abstract: Object-goal navigation (ObjNav) tasks an agent with navigating to the location of a specific object in an unseen environment. 
Embodied agents equipped with large language models (LLMs) and online constructed navigation maps can perform ObjNav in a zero-shot manner. However, existing agents heavily rely on giant LLMs on the cloud, e.g., GPT-4, while directly switching to small LLMs, e.g., LLaMA3.2-11b, suffer from significant success rate drops due to limited model capacity for understanding complex navigation maps, which prevents deploying ObjNav on local devices.
At the same time, the long prompt introduced by the navigation map description will cause high planning latency on local devices.
In this paper, we propose EfficientNav to enable on-device efficient LLM-based zero-shot ObjNav. To help the smaller LLMs better understand the environment, we propose semantics-aware memory retrieval to prune redundant information in navigation maps.
To reduce planning latency, we propose discrete memory caching and attention-based memory clustering to efficiently save and re-use the KV cache.
Extensive experimental results demonstrate that EfficientNav
achieves 11.1\% improvement in success rate on HM3D benchmark over GPT-4-based baselines, 
and demonstrates 6.7$\times$ real-time latency reduction and 4.7$\times$ end-to-end latency reduction over GPT-4 planner. Our code is available on https://github.com/PKU-SEC-Lab/EfficientNav.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents EfficientNav, a system for enabling efficient zero-shot object-goal navigation (ObjNav) on local devices with limited memory resources. The key challenge addressed is that existing LLM-based navigation systems rely on cloud-based large models like GPT-4, which suffer from high latency and privacy concerns. EfficientNav introduces three main contributions: (1) discrete memory caching to avoid recomputing KV cache for navigation map descriptions, (2) attention-based memory clustering to group related objects and reduce cross-attention loss, and (3) semantics-aware memory retrieval using CLIP to select relevant object groups based on navigation goals. The system achieves 11.1% improvement in success rate over GPT-4 baselines while providing 6.7× real-time latency reduction.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nImportant Problem Identification:\n\nThis paper addresses a critical challenge in LLM/VLM-based navigation, real-time performance, which is a common bottleneck for many navigation methods. This issue is especially pivotal for enabling the practical deployment of such methods in real-world applications.\n\nTechnical Contributions: \n\n1. Discrete Memory Caching: This module effectively addresses memory constraints by clustering navigation map information into discrete groups, independently calculating their KV cache, and efficiently reusing cached data to reduce recomputation and transfer overhead.\n\n2. Attention-based Memory Clustering: This approach leverages LLM attention mechanisms to cluster semantically related map information, thereby significantly improving the effectiveness of memory retrieval and reducing the adverse impact of ignoring cross-attention.\n\n3. Semantics-aware Memory Retrieval: By utilizing semantic similarity via the CLIP model, this module dynamically selects relevant map groups based on the navigation task, enabling efficient memory usage and significantly enhancing planning performance of smaller LLMs.\n\nThorough analysis of memory and efficiency of different LLM/VLMs: This paper clearly presents the memory requirements of different LLM/VLMs and their latency in Figure 1. It also evaluates the latency of different navigation stages in Figure 3 (b) and Figure 9. This thorough evaluation is valuable as a guidance to the design of LLM/VLM-based navigation methods.\n\nEvident performance improvement: The paper provides extensive experiments on the HM3D benchmark, demonstrating significant improvements over existing methods, notably 11.1% success rate improvements over GPT-4-based baselines and substantial latency reductions.\n\nWeaknesses\n\nAttention-Based Clustering Justification: The choice of using only 1/10 of LLM layers for attention-based clustering lacks theoretical justification. Why this specific fraction? How sensitive is performance to this hyperparameter?\n\nCLIP Dependency: The semantics-aware retrieval relies heavily on CLIP's semantic understanding. The paper doesn't analyze failure cases where CLIP's similarity scores might be misleading for navigation decisions. The contribution of CLIP to the final navigation success rates is not discussed. Is this key to the 16.7% success improvement? If not using CLIP or using other similar models, how much will the performance drop?\n\nLimited Baseline Comparisons: This paper primarily compares against naive LLM planners and GPT-4. Comparisons with other efficient LLM inference methods would be valuable.\n\nPractical deployment problems: The practical deployment complexities of integrating multiple sophisticated methods might require considerable development effort for real-world scenarios.\n\nLack of detailed ablation studies: The ablation study does not show the influence and detailed analysis of the ablation choices, such as the influence of CLIP model, the choice of 1/10 layers of LLMs and how the discrete attention scheme outperforms the full attention scheme.\n\nQUESTIONS:\nQuestions\n\nEffectiveness of Discrete Memory Caching: Table 4 has shown that the Discrete Memory Caching module achieves 0.4% SR improvement. However, as Section 3.2 shows, this is mainly a design to solve the memory constraints and will cause the ignorance of cross-attention between groups, so how can this module increase the SR metric, or is this just a statistical deviation? \n\nPerformance difference between discrete attention and full attention: As the authors put in Section 3.2 and 3.3, the Attention-based Memory Clustering method serves as a complement of the Discrete Memory Caching method to make up for the ignorance of cross-attention between groups. Then why will this discrete attention scheme (Discrete Memory Caching + Attention-based Memory Clustering) outperforms the full attention scheme (LLaVA Planner) by 20.2% in SR as shown in Table 4? The text descriptions in Section 4.3 does not show how this discrete attention could achieve such a performance gain.\n\nThe changes of semantic groups: Could the authors provide an example of how the semantic groups are updated as the agent is exploring the scene? And as the paper shows, the newly-added object will be added to existing groups based on the semantic similarity, so will it be a problem that some semantically-related but spatially-distant objects be clustered in a group? If yes, how will this influence the final navigation performance?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

comments: SUMMARY:
The authors propose discrete memory caching for object-goal navigation, whereby the map information ca split into segments and the KV cache can be more efficiently decomposed and stored. This solution mitigates the problem of constrained systems not having enough memory to store entire KV caches. Since this clustering into discrete segments may negatively impact performance due to problems with cross-attention across segments, the authors propose attention-based memory clustering.

STRENGTHS AND WEAKNESSES:
Strengths:
* The authors include a nice empirical validation (Figure 6, a) that the latency incurred by loading the KV cache repeatedly in the conventional way is large.
* Semantics-aware memory retrieval is used by leveraging a CLIP to compute semantically similar inputs, which seems reasonable.
* Their experimental results on improving latency over the state-of-the-art are impressive.

Weaknesses:
* Some well-known related work on navigation using LLMs is missing, including the line of work on integrating scene graphs and LLMs for navigation, see, e.g., [1] below.
[1] Rajvanshi, Abhinav, et al. "Saynav: Grounding large language models for dynamic planning to navigation in new environments." Proceedings of the International Conference on Automated Planning and Scheduling. Vol. 34. 2024.

QUESTIONS:
For semantic retrieval, do you think something like a knowledge graph or a scene graph can be used to augment or replace your current CLIP-based method?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nEfficientNav is proposed to address two major challenges in applying large language models (LLMs) to on-device Object Navigation (ObjectNav). The first challenge lies in the difficulty of deploying large LLMs on resource-constrained devices. The second issue concerns the growth of the key-value (KV) cache, which increases significantly as context information accumulates throughout the navigation process. To tackle these challenges, the authors introduce a combination of discrete memory caching and a cluster-based navigation map, allowing for more efficient estimation and control of KV cache usage. Additionally, they propose a semantic-aware memory retrieval module, which enables the system to reduce unnecessary map information by selecting only semantically relevant memory entries for reasoning and decision-making.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- The proposed Attention-based Memory Clustering and Semantic-aware Memory Retrieval effectively enhance the usability of LLMs for on-device navigation tasks.\n- These two components enable the model to identify relevant features based on their relation to the current sub-goal, improving both retrieval efficiency and navigation accuracy.\n\n### Weaknesses\n- The overall architecture of the model is not fully described in the paper. And make the reader a little bit confusing. For example, it does not state which detection model is used for constructing navigation map.\n\nQUESTIONS:\n1. What is the basis for clustering groups? Is it based on semantics or other metrics?\n\n2.  In the supplementary navigation examples, does each node represent only one object?\nIn Figure 11, object 7 and object 10 appear to belong to the same node, but in the trajectory, both objects appear separately. This seems contradictory.\n\n3. How is the sub-goal selected?\nSection 3.4 describes how relevant groups are selected, but since each group contains multiple objects, how is the final sub-goal selected from within a group?\n\nLIMITATIONS:\n- The paper lacks detailed architectural descriptions, particularly regarding components such as the object detection module. This omission makes it difficult to fully understand or reproduce the system pipeline. \n- The empirical evaluation is limited to indoor simulation environments. There is little analysis of how well the proposed method generalizes to more complex, real-world, or large-scale navigation scenarios.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an efficient LLM based object goal navigation approach with navigation map cashing and retrieval.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n- The motivation toward on device LLM based object goal navigation is valuable. \n\n- The proposed method achieves higher success rate than baseline models with relatively low latency.\n\n\nWeakness:\n\n1) The task setting is specialized: the goal object is defined only three. And it does not do vision language navigation which is tackled by its LLM baseline techniques such as MapGPT. Evaluating on object goal navigation with the same number of target object as in previous works and/or VLN would be appreciated.  \n \n2) It talks about Jetson but there is no robot hardware deployment experiment. \n\n3) I wonder how the general adaptive sparse attention approaches work on the object goal navigation task, since its not predefined so it could in theory adapt to this kind of task too. Comparisons to the model using dynamic sparse attention like these without the proposed semantics aware memory retrieval would be appreciated.\n\n- Minference\n\n- FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inference\n\n- SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inference\n\n4) It lacks some object goal navigation references that relate target and zone/room.\n\n- Hierarchical Object-to-Zone Graph for Object Navigation\n\n- Enhancing Multimodal-Input Object Goal Navigation by Leveraging Large Language Models for Inferring Room-Object Relationship Knowledge\n\n- HOZ++: Versatile Hierarchical Object-to-Zone Graph for Object Navigation\n\nQUESTIONS:\nSee Weakness\n\nSection 3.3 and 3.4 is a little difficult to grasp. It would be nicer to use Figure 5 and tell how LLM semantic space and 2D map is associated.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}