Title: Dynam3D: Dynamic Layered 3D Tokens Empower VLM for Vision-and-Language Navigation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Vision-and-Language Navigation

3D Vision-Language Model
Abstract: Vision-and-Language Navigation (VLN) is a core task where embodied agents leverage their spatial mobility to navigate in 3D environments toward designated destinations based on natural language instructions. Recently, video-language large models (Video-VLMs) with strong generalization capabilities and rich commonsense knowledge have shown remarkable performance when applied to VLN tasks. However, these models still encounter the following challenges when applied to real-world 3D navigation: 1) Insufficient understanding of 3D geometry and spatial semantics; 2) Limited capacity for large-scale exploration and long-term environmental memory; 3) Poor adaptability to dynamic and changing environments.To address these limitations, we propose Dynam3D, a dynamic layered 3D representation model that leverages language-aligned, generalizable, and hierarchical 3D representations as visual input to train 3D-VLM in navigation action prediction. Given posed RGB-D images, our Dynam3D projects 2D CLIP features into 3D space and constructs multi-level 3D patch-instance-zone representations for 3D geometric and semantic understanding with a dynamic and layer-wise update strategy.  Our Dynam3D is capable of online encoding and localization of 3D instances, and dynamically updates them in changing environments to provide large-scale exploration and long-term memory capabilities for navigation. By leveraging large-scale 3D-language pretraining and task-specific adaptation, our Dynam3D sets new state-of-the-art performance on VLN benchmarks including R2R-CE, REVERIE-CE and NavRAG-CE under monocular settings. Furthermore, experiments for pre-exploration, lifelong memory, and real-world robot validate the effectiveness of practical deployment.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Dynam3D, a system designed to help robots follow natural language instructions and navigate through indoor environments using only a single RGB-D camera. Unlike previous methods that rely on video-based models, Dynam3D builds a dynamic 3D representation of the environment using a layered structure that includes small visual patches, object instances, and larger spatial zones. It updates this representation in real time, allowing the system to better understand the 3D layout, remember past observations, and adapt to changes like moving objects. The authors also introduce a 3D Vision-Language Large Model  to predict the action in a continuous environment. Experiments show that Dynam3D achieves strong results on several navigation benchmarks and works well in real-world and dynamic settings.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nThe hierarchical 3D representation enables navigation in dynamic scenes by updating only the instance-level features, allowing the algorithm to adapt its behavior in response to moving objects.\n\nThe authors applied this method to a real robot and demonstrated its performance in real-world operation.\n\n**Weaknesses**\n\nThe reviewer thinks the biggest weakness of this paper is the lack of a comprehensive ablation study. For example, while the method outperforms previous approaches in the experiments, it's unclear whether the comparison was made under the same model parameter size. In real-world applications, RGB-D cameras produce different types of noise depending on the scene, and the impact of such depth noise on navigation has not been sufficiently analyzed. Moreover, the point clouds shown in the paper appear almost perfect, so the reviewer has reservations about how well the algorithm would perform in real-world scenarios with noisy data.\n\nThe pipeline of this algorithm is highly hand-crafted, which may lead to poor robustness in handling certain edge cases, such as camera tracking errors or segmentation mistakes that occur during intermediate stages of the pipeline.\n\nQUESTIONS:\nHow would the performance be affected if the camera parameters are miscalculated or the depth data contains noise?\n\nCompared with discrete VLN settings, what are the strengths and limitations of VLN in continuous environments? It is advisable to clarify this in the paper.\n\nLIMITATIONS:\nThe authors have not adequately discussed the application scenarios of the algorithm and the requirements for data quality.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new 3D hierarchical representation structure for organizing information in vision-language action (VLA) models. The core idea is inspired to some extent by mapping-based methods, but it also departs from them in important ways. The proposed structure acts as a sliding-window-style, multi-level, dynamically updated, fused, and removed semantic memory, rather than a persistent global map. This design addresses key issues faced by video-based VLA models, namely, shallow physical understanding and inefficient memory organization, as the authors themselves note in the intro second paragraph. Technically, the paper introduces a three-tier 3D representation and its corresponding update/fusion mechanism, and demonstrates its effectiveness through both simulator-based and real-world robot experiments.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths \n\n1、This is a strong paper with a clear motivation and solid technical contributions. The proposed direction feels like a natural and promising step forward given the current state of research, bringing 3D awareness into VLA without incurring excessive computational or memory overhead.\n\n2、The technical design of this work is carefully crafted. It introduces a multi-level 3D representation structure, and incorporates dynamic mechanisms for updating, merging, and pruning tokens, which ensures computational efficiency and a certain degree of real-time capability. The technical components are well-formulated with clear mathematical definitions, allowing readers to precisely understand the authors' intentions.\n\n3、The experiments are thorough, and the comparisons are both relevant and novel. The accompanying demo, though simple, is convincing. Importantly, the insights presented in this work could inspire further research in the community, it is a meaningful exploration.\n\nWeaknesses\n\n1、My main concern lies in generalization. Higher-dimensional representations typically require more data to train effectively. Existing VLA systems like Uni-NaVid are trained on massive 2D-scale datasets (millions to tens of millions of samples), whereas this work does not appear to increase the data volume in proportion to the added complexity of 3D representations. This raises the question of whether the proposed 3D-language alignment generalizes well, especially under real-world conditions with noise and the inherent sim-to-real gap. I strongly encourage the authors to conduct more experiments specifically targeting generalization, either within simulators (e.g., cross-scene or cross-dataset setups) or on real robots using unseen environments, such as outdoor scenes.\n\n2、The second major issue is inference speed. As shown in Section 4.4, the real-world inference runs at under 2 FPS, which is somewhat slow for practical deployment. Interestingly, as noted in Section 4.1, the model's parameter size is actually smaller than other VLA baselines. This suggests that there may be unexplored opportunities—either at the model level or in engineering—to significantly improve runtime performance. I hope the authors can propose concrete and actionable ideas to reduce inference latency.\n\n3、A third point is that the robot demo leaves room for improvement. Given the introduction of 3D spatial understanding, one would expect the system to showcase unique VLN capabilities that go beyond existing 2D-based models. More spatially grounded or fine-grained instructions could have made the demo more impressive. As it stands, it feels somewhat ordinary.\n\n(Minor concern) The intent of Section 4.2 is a bit unclear. The proposed representation is described as \"ephemeral and disposable,\" yet the experiments on pre-exploration and lifelong memory imply persistent memory reuse. It would be helpful for the authors to clarify this apparent contradiction and explicitly state the purpose of these experiments.\n\n(Minor concern) Regarding the action prediction format in VLA models—have the authors considered using more advanced policy structures, such as multi-step forecasting or diffusion-based policies? Given the richness of the spatial representation, such approaches could lead to more foresightful and smoother navigation.\n\nQUESTIONS:\nI would appreciate it if the authors could primarily address the two major weaknesses:\n\nGeneralization: Please consider adding cross-dataset evaluation or using previously unseen real-world environments (e.g., outdoor scenes) to verify the robustness of 3D-language alignment.\n\nInference Speed: It would be highly valuable if the authors could propose insightful and practical ways to reduce inference latency. If the inference can be sped up noticeably, it would significantly strengthen the paper.\n\nIf possible, it would also be great to see the authors touch on the remaining concerns—about demo quality, Section 4.2 clarity, and action format—even if only in discussion or qualitative analysis.\n\nLIMITATIONS:\nYes, the limitations are discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Dynam3D, a novel framework for vision-language navigation (VLN) that introduces a dynamic hierarchical 3D scene representation. The representation integrates patch, instance, and zone tokens, addressing challenges in spatial understanding, long-term memory, and flexibility to dynamic environments. The authors combine this representation with a pretrained VLM LLaVA-Phi-3-mini, and demonstrate superior performance across VLN benchmarks (R2R-CE, REVERIE-CE, NavRAG-CE). Furthermore, Dynam3D exhibits significant results in real-world navigation tasks, including static and dynamic environments.\n\nSTRENGTHS AND WEAKNESSES:\n#### Strengths\n- The methodology is well-grounded and technically sound. The idea of dynamically updated patch-instance-zone 3D representations is both innovative and well-motivated. And Dynam3D incorporates thoughtful designs for dynamic environments, including Frustum Culling and merging discriminator.\n- The experimental results are significant, highlighting the strengths of Dynam3D in performing VLN tasks. In particular, Dynam3D shows strong performance in real-robot experiments.\n\n#### Weaknesses\n- I think the representation design is kind of overcomplicated. The pipeline for deriving various kinds of tokens involves complex procedures such as segmentation with FastSAM and rendering with g3D-LF. My concern persists despite the reported computational costs and real-time analyses. While it cannot be a major weakness to propose many dedicated and specific designs, it can make the method less appealing.\n- Given such dedicated designs, I would expect more ablation results to figure out the contributions of different parts to the performance. For example, the contributions of 3D instance tokens and panoramic 3D patch tokens are not clearly reported. On the other hand, the patch, instance, and zone tokens represent different granularities. So my concern is that would such a heterogeneity confuse the model learning (e.g., can model distinguish their token types)?\n\nQUESTIONS:\n- I think the delivered message in the experiments of pre-exploration and lifelong memory (Section 4.2) can be polished. There is no doubt that pre-exploration would improve the performance, and it cannot highlight the advantage of Dynam3D since the comparison with other methods without pre-exploration is unfair. Instead, I think the proper message here is that Dynam3D makes lifelong memory feasible and thereby achieves a similar effect to pre-exploration.\n- I would expect more details on the processing of VLN data, such as action merging, label smoothing or rebalancing. These details can be important to the final performance of VLN as demonstrated by prior works.\n- An interesting and important question is how well the merging discriminator can perform. Are the merging results accurate? Providing some simple evaluations about the merging accuracy would offer more insights.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper presents Dynam3D, a dynamic layered 3D representation framework for vision-and-language navigation that addresses limitations of existing video-based approaches through three primary contributions. First, the authors develop a hierarchical patch-instance-zone representation model that projects 2D CLIP features into 3D space and maintains multi-view consistent 3D instances using a merging discriminator trained on large-scale 3D segmentation data. Second, they implement a dynamic update mechanism employing frustum culling to remove outdated features and real-time hierarchical updates to adapt to changing environments. Third, they pass in these representations into a 3D Vision-Language Model that combines panoramic 3D patch tokens.

STRENGTHS AND WEAKNESSES:
Strengths:
* Technical innovation: Clearly identifies gaps in the current SOTA and addresses it with their method.
* Results: Demonstrates clear strength in performance in both simulated and real world setting.

Weaknesses:
* It is unclear how this method scales to larger environments - especially outdoors. 
* By extension, it is unclear to me how this method would work if frames end up being drastically different from each other.

QUESTIONS:
N/A - see weaknesses

LIMITATIONS:
N/A
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03