3D-SPATIAL MULTIMODAL MEMORY

Authors: Xueyan Zou, Yuchen Song, Ri-Zhao Qiu, Xuanbin Peng, Jianglong Ye, Sifei Liu, Xiaolong Wang
Primary area: foundation or frontier models, including LLMs
Keywords: Foundation Model, Gaussian Splatting, Large Multimodal Model, Robotics

Abstract:
We present 3D Spatial MultiModal Memory (M3), a multimodal memory system designed to retain information about medium-sized static scenes through video sources for visual perception. By integrating 3D Gaussian Splatting techniques with foundation models, M3 builds a multimodal memory capable of rendering feature representations across granularities, encompassing a wide range of knowledge. In our exploration, we identify two key challenges in previous works on feature splatting: (1) computational constraints in storing high-dimensional features for each Gaussian primitive, and (2) misalignment or information loss between distilled features and foundation model features. To address these challenges, we propose M3 with key components of principal scene components and Gaussian memory attention, enabling efficient training and inference. To validate M3, we conduct comprehensive quantitative evaluations of feature similarity and downstream tasks, as well as qualitative visualizations to highlight the pixel trace of Gaussian memory attention. Our approach encompasses a diverse range of foundation models, including vision-language models (VLMs), perception models, and large multimodal and language models (LMMs/LLMs). Furthermore, to demonstrate real-world applicability, we deploy M3’s feature field in indoor scenes on a quadruped robot. Notably, we claim that M3 is the first work to address the core compression challenges in 3D feature distillation.

Decision: accept (poster)
Reviewer scores: 8, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730381183270

thread_type:
official_review

<thread_object>
creator: reviewer_2xxj

<content>
summary:
This paper introduces 3D Spatial MultiModal Memory (M3), a video memory system aimed at retaining visual perception information over medium temporal and spatial scales. M3 employs 3D Gaussian splatting and incorporates multiple foundation models to create a multimodal memory capable of spanning visual details and encompassing a broad knowledge range. The authors address two primary limitations of prior feature splatting methods: high computational demands for storing dense features in each Gaussian primitive and misalignments with foundation model knowledge. M3 innovatively introduces principal scene components and Gaussian memory attention mechanisms to improve the efficiency of storage, training, and inference within the Gaussian Splatting Model. Evaluated across various foundation models, including vision-language and self-supervised models, M3 demonstrates robust performance, with successful deployment on a quadruped robot for grasping tasks, showcasing its practical viability in real-world applications.

strengths:
The idea is interesting and novel, and this direction is promising for diverse robotic tasks.
The method design makes sense. The figures and algorithm flowchart make the paper easy to understand.
The authors provide comprehensive coparisons in evaluation on M3 features (table 1, 2, 3, 4).

weaknesses:
My concerns mostly lie on real robot experiments. I think one of the advantage of such feature field (also claimed in paper) is for various downstream robotics manipulation tasks. While, this paper shows a simple example in grasping a duck on the table. It could make this paper much stronger if the authors can demonstrate the promising potential of this study in various practical downstream tasks.
Some example downstream tasks:
1. Geometry-based or semantic-part-based tasks using DINO or diffusion features, such as geometry-based object retrieval/grasping[1, 2], planning objects to the goal configuration[3].
2. Language-guided tasks using CLIP feature, such as like "pick up the red cube and place it on the blue book." (example references: [4, 5])
3. A multi-step task that combines different features, such as "navigate to the kitchen, locate the mug with a cat design, and bring it to the living room table." These examples would showcase how M3 can leverage different types of features for more complex robotic applications.

Some typos: for example, in figure 5, dinov->dinov2. In line 350&351, there are missing parts after `as shown in'.

[1] SparseDFF: Sparse-View Feature Distillation for One-Shot Dexterous Manipulation. ICLR 2024.

[2] Robo-ABC icon : Affordance Generalization Beyond Categories via Semantic Correspondence for Robot Manipulation. ECCV 2024.

[3] D^3Fields: Dynamic 3D Descriptor Fields for Zero-Shot Generalizable Rearrangement. CoRL 2024.

[4] Distilled Feature Fields Enable Few-Shot Language-Guided Manipulation. CoRL 2023.

[5] NaturalVLM: Leveraging Fine-grained Natural Language for Affordance-Guided Visual Manipulation. RA-L 2024.

questions:
Please see weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730601269581

thread_type:
official_review

<thread_object>
creator: reviewer_uxh3

<content>
summary:
The paper presents a method for integrating foundation model knowledge into a structural representation of the scene (here Gaussian Splatting). The approach is capable of integrating several foundation models' knowledge into Multimodal Memory. To this end, the authors propose Principal Scene Components, i.e. a method of feature reduction based on the assumption that multiple views of the scene contain redundant information. The paper evaluates the performance on a low level in feature space, and on a high level - through a downstream task, including robot deployment.

strengths:
I believe that the authors of the paper correctly identified the drawbacks of 3D feature supervision through 2D feature maps (i.e. potential loss of information with dimensionality reduction). To this end, the paper proposes a compression scheme for extracted features named principal scene components (PSC). I believe this idea to be sound and interesting. I appreciate that the idea includes a flexible parameter of similarity threshold allowing the user to decide on memory/knowledge trade-off. In principle, the method seems to be trained in a similar way to prior works, making me believe that the proposed memory module can be used interchangeably to prior works, and could be applied to an arbitrary 3DGS work. Another benefit is a significant speed-up with respect to F-3DGS. Qualitative results seem convincing. I believe the use of robot manipulation as the downstream task is a good demonstration of the capabilities of the proposed approach.

weaknesses:
While I believe the authors propose an interesting method, I see some weaknesses in the paper. Firstly, there are some drawbacks to the experimental setup:
- F-3DGS evaluated their approach through the task of novel view semantic segmentation. I see no reason why this could not be used in this paper as the experiment. This makes comparison between the two methods harder. Similarly, both F-3DGS and F-Splat do language-guided editing. Direct feature comparison is interesting and meaningful, however, we should also adhere to previously used evaluation protocols in order to facilitate fair comparisons.
- Similar to the previous point, I believe it would be beneficial to have some datasets between the proposed method and prior work in common. 
- Further, the paper claims memory efficiency as a part of the method's benefits. Therefore an ablation and comparison to prior work from the memory perspective would be useful.
- There should be a study on whether there exists a correlation between performance, training time, and memory footprint (e.g. quantitative metrics for similar training time to F-3DGS - does M3 reach the same performance or does it plateau; also a comparison between models for the same feature size, etc.).

questions:
- Do all the methods in comparison use the same underlying 3DGS model?
- Lines 48-49 mention both static scenes and time windows, could you explain that?
- Do you train reconstruction together, or do you start training with colour loss only? Alternatively, is the method sensitive to initialisation? With poor initialisation, the first iteration may be very noisy and rendered features very poor.
- Could you pinpoint the exact component that makes M3 training so much faster than F-3DGS?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730726016381

thread_type:
official_review

<thread_object>
creator: reviewer_rufb

<content>
summary:
The paper introduces MultiModal Memory (M3), an integration of 3D Gaussian Splatting (GS) techniques with features from 2D Foundation Models (FMs) that replace standard feature distillation pipelines. The authors introduce two new concepts: a) principal scene components (PSC) which reduces the dimensionality of the FM features without sacrificing quality, thus reducing computational resources for training and storing features, and b) gaussian memory attention (GMA) which proposes to learn queries to memory as gaussian attributes (and not
the FM features themselves), thus dealing with the issue of misalignment between distilled and raw features of previous methods. The proposed memory can store features from multiple FMs, thus allowing a single GS structure to reconstruct multiple different features and solve many vision-language downstream tasks. The authors compare the quality of reconstructed features vs previous approaches in 4 scenes, do ablations in another 2 robotic-related scenes, provide comparisons with downstream tasks for 1 scene, and also provide a
demonstration with a mobile robot.

strengths:
1. The introduction of Gaussian Memory Attention (GMA) is a novel and interesting way of linking Gaussian \ Splatting techniques with feature distillation. Previous works learn downsampled versions of the target features to reduce memory footprint, and decode them back with MLPs. The downsampled feature is the learned attribute in GS. With GMA, the learned attribute is a query to a memory, which can hold features from multiple foundation models. This allows a single trained GS to reconstruct features from many possible \ foundation models, instead of one GS per foundation model.

2. The introduction of Principal Scene Components (PSC) and the similarity reduction algorithm seems to be able to provide a good balance between memory footprint (#degrees stored in memory) vs. reconstruction quality, according to Table 1.

3. The authors integrate generative foundation models (Llama3v etc.) with their approach, while previous works only consider discriminative features (CLIP, DiNov2 etc.). The authors claim that the reconstructed features can be used directly as visual context to LLMs for generative tasks such as image captioning, which is I believe a great advancement of the distilled feature fields work, although they did not provide demonstrations for such applications.

weaknesses:
1. The paper misses implementation details and qualitative results for downstream task applications. How to use the reconstructed FM features to do the tasks you show in your Fig.2, namely: image captioning, language grounding and retrieval? Please share some explanation of which feature is used for what task, and how, and provide qualitative comparisons with previous works F-3DGS and F-splat.
 
2. Lacking quantitative comparisons with baselines for downstream tasks. Only Table 4 contains comparisons for 1 scene with grounding task but lacks comparison with F-Splat baseline. I believe the authors need to perform more experiments for downstream tasks for more scenes and compare with both baselines, so the final performance-efficiency tradeoff from the introduction of PSC and attention memory can be appreciated.

 3. Also, currently the paper only shows reconstructed features from their M3 method. I believe the paper would benefit from a comparison of reconstructed features between previous approaches and M3, in order to appreciate qualitatively the effect of introducing the PSC method for reducing feature dimensionality and Gaussian Memory for decoding back the original feature.

 4. There is some confusion regarding the inclusion of the temporal aspect in the memory. The authors name their structure video memory and say that it holds medium-time video clips for static scenes. However, from the method and experiments, it seems that the temporal aspect has not been a focus of the paper, as the scenes are static, and the downstream tasks are only at the image or pixel level (captioning, grounding, etc.). Do the authors mean that the video contains the multi-view collection trajectory of an agent? If that is the case, I believe it should not claim to be a video memory and model the temporal aspect, as the temporal dimension in the video is just the dimension of new views of the same static scene.

 5.  I believe the paper would benefit from better clarity when presenting the method. The authors introduce many concepts by naming them (VG, KS, V) but do not provide input domains for such concepts therefore lacking precision. For example, in line 189, the authors write for a view V ∗ ∈ V, its V ∗ = {v 1 , v 2 , ..., v n }. What is the domain of V and V*? What are the intermediate segments vi? Later, in line 200, they write F ∗ (V) = {E 1 , E 2 , ...E n } for the foundation model features of a scene and say that n is the number of views. This seems a bit contradicting to me, as before they mentioned V* as a particular scene, and n the number of multi-granularity segments of that scene. Which one is it, n is a number of scenes or a number of segments inside a scene? 

MINOR:
 - (line 250) as shown in? 
 - Table 4 is mentioned before Tables 2 and 3. Maybe rename?
 - (line 062): [...] may not be be inherently 3D-consistent [...]
 - (line 064): [...] knowledge embbeded in [...]
 - (line 080): [...] outperforms what?

questions:
1. How to use the reconstructed FM features to do the tasks you show in your Fig.2, namely: image captioning, language grounding, and retrieval? Please share some explanation of which feature is used for what task, and how, and provide qualitative comparisons with previous works F-3DGS and F-splat.

2. Can the authors provide qualitative examples comparing the reconstructed features of M3 with those from previous methods, such as F-3DGS and F-Splat? This would help assess the quality improvement gained through PSC and GMA.

3. Could the authors clarify if the term "video memory" implies the storage of multi-view data in static scenes or if there is an intention to extend this framework to dynamic scenes? If the latter, would the temporal dimension in the memory structure require additional model adjustments?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730765684651

thread_type:
official_review

<thread_object>
creator: reviewer_a6fd

<content>
summary:
The article proposes a novel approach for video memory combining structural information from Gaussian Splatting with semantic information gathered from foundation models. The main challenge in this approach is the high-dimensionality of the semantic features for encoding in Gaussian splats. This has been addressed in the literature by feature distillation, but the authors argue that this leads to inefficient misaligned distilled features. 
The authors proposed a novel attention approach to extract so-called Principle Scene Component. Experiments on a number of datasets appear to show that the proposed approach compares well in terms of fidelity and efficiency with the state-of-the-art, supporting the authors' claims.

strengths:
- The efficient integration of structural and semantic features is an important problem for AI, and the proposed method is an interesting contribution. 
- The method appears to perform well on a broad range of measures and experiments. 
- The robotic application is a nice addition.

weaknesses:
- The Gaussian Memory Attention component is a central contribution of the paper but is rather briefly discussed in the paper. It would be good to provide a clearer rationale for why this method was chosen and why it achieves the claimed result? How does it compare to standard memory mechanisms (such as in transformers), and what could be the alternative implementations. 
- Similarly, the article contains an extensive ablation study of the foundation models, but no analysis of the impact of this specific component on performance. It would be good to provide some evidence of that as an ablation study (eg, Gaussian Memory attention vs other attention implementation). This would provide valuable insight into the component's impact on overall performance. 
- The robotic experiment is interesting but it is unclear how informative it is about real-world applicability. It would be good to discuss possible challenges in scaling the approach to more complex/realistic environments and tasks, and to clarify the limitations of the current implementation as well as the future work that would be required for a real-world application.

questions:
See the questions above in the weaknesses section, in summary: 
- Can you provide a rationale for Gaussian Memory Attention component and a comparison to standard approaches? 
- Can you provide an ablation study to evidence the impact of this novel component on the system's performance? 
- Can you provide a discussion of the limitations of the approach in the robotic scenario and the outstanding questions and limitations to be overcome for real-world application.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732787900461

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response [1/2]

comment:
We thank the reviewers for the meaningful comments spanning model architecture (`a6FD`), feature representation (`RUFb`), gaussian representation (`UxH3`), and robotics (`2XXj`). ***These suggestions motivate us to extend our work to the next stage*.** 

**→ Strength:** We appreciate the reviewers recognize M3’s performance improvements in both quality (`a6FD`, `RUFb`) and efficiency (`a6FD`, `UxH3`), and value its potential for real-world robotic applications (`a6FD`,`UxH3`,`2XXj`). ***Most importantly, all the reviewers believe that the proposed idea is interesting and novel:***

**[Novelty `a6FD`, `RUFb`, `UxH3`, `2XXj`]**: The efficient integration of structural and semantic features is an important problem, and the proposed method is an interesting contribution (`a6FD`). In addition, the proposed module Gaussian Memory Attention (GMA) is novel and interesting (`RUFb`), and correctly identified the drawbacks of 3D feature supervision through 2D feature maps (`UxH3`). The proposed Principal Scene Components (PSC) and the reduction algorithm provide a good memory footprint vs. reconstruction quality (`RUFb`), and this idea is believed to be interesting and sound (`UxH3`). Meanwhile, generative foundation models are included in M3, which I believe a great advancement of the distilled feature field work (**`RUFb`**).

**→ Weakness:** Although the novelty of M3 is well recognized, all the reviewers propose constructive experimental suggestions of the proposed idea. We sincerely apologize for not uploading the rebuttal earlier, but ***we have intensively addressed the reviewer’s questions and suggestions regarding quantitative/qualitative results, real robot experiments, motivations, and writing***.

**[Downstream Tasks Performance]** While the proposed M3 outperforms previous methods in feature reconstruction, we also aim to demonstrate its effectiveness on downstream tasks. Although we have shown some downstream task performance in the initial submission, during the rebuttal, we included a more comprehensive version of grounding and retrieval evaluation. Please refer to the supplementary material for the newly proposed dataset.

In the table below, we show the downstream task performance of both F-3DGS and M3 in comparison with GT features in three datasets including Train, Playroom, and Geisel. The evaluation metrics cover both grounding and retrieval tasks, with metrics spanning IoU, AP, I2T, and T2I. We use MaskCLIP embeddings for grounding and SigLIP embeddings for retrieval. 

The table below shows the following insights:

1. With half the parameter size compared to F-3DGS, we not only achieve better results than F-3DGS by a reasonable margin (especially on AP50) but also outperform the original GT feature from MaskCLIP. This is caused by the Gaussian rendered feature usually having higher resolution as well as less noise.
2. For image retrieval, M3 performs significantly better than F-3DGS, while having relatively closer results with ground truth. Especially when we examine the top-1 results (I2T@1, T2I@1), M3 is much better than F-3DGS, this indicates that the feature map created via principal scene components does have a knowledge space more closely aligned with the original foundation model’s embedding space.

| Dataset | Method | # Param | Grounding/CLIP |  |  |  | Retrieval/SigLIP |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  |  | mIoU | cIoU | AP50 | AP60 | I2T@1 | I2T@5 | I2T@10 | T2I@1 | T2I@5 | T2I@10 |
| Train | GT | - | 25.3 | 26.3 | 14.7 | 3.3 | 81.5 | 97.3 | 100.0 | 71.0 | 89.4 | 92.1 |
| Train | F-3DGS | 61M | 24.2 | 24.3 | 16.3 | 7.1 | 2.6 | 13.2 | 28.9 | 0.0 | 2.6 | 18.4 |
| Train | M3 | **35M** | **25.4** | **26.5** | **19.6** | **12.5** | **55.2** | **84.2** | **92.1** | **52.6** | **84.2** | **92.1** |
| Playroom | GT | - | 25.6 | 24.2 | 9.6 | 3.0 | 96.5 | 100.0 | 100.0 | 62.0 | 96.5 | 100.0 |
| Playroom | F-3DGS | 61M | 23.8 | 21.4 | 11.9 | 3.0 | 79.3 | 96.6 | 96.6 | 31.0 | 79.3 | 89.7 |
| Playroom | M3 | **35M** | 23.1 | **23.1** | **11.9** | **5.9** | 72.4 | **96.6** | **100.0** | **41.3** | 65.5 | 68.9 |
| Geisel | GT | - | 19.5 | 21.4 | 5.3 | 0.0 | 100.0 | 100.0 | 100.0 | 60.0 | 85.7 | 91.4 |
| Geisel | F-3DGS | 61M | 19.0 | 20.4 | 14.1 | 1.2 | 45.7 | 94.3 | 100.0 | 0.0 | 20.0 | 34.3 |
| Geisel | M3 | **35M** | **21.8** | **23.5** | **16.5** | **11.8** | **100.0** | **100.0** | **100.0** | **71.4** | **85.7** | **94.2** |
</content>
</thread_object>

</thread>


Metareview:
metareview:
**Summary**

The paper presents 3D spatial multimodal memory (M3) which distills semantic information from a combination of 2D foundation models into 3D Gaussian splatting.  The different 2D foundations models provides features at different granularities.  Experiments compare the proposed model against two recent methods that distills 2D features into Gaussian splats (F-3DGS, F-Splat), showing that M3 is more efficient that F-3DGS and has better feature similarity than F-Splat.  

The key contribution of the work is how to compress the information from a diverse set of 2D foundation models onto 3D Gaussian without losing important information.  To do so, the paper introduces 1) "principle scene components" (PSC) which provides a compressed memory bank of 2D features, and 2) Gaussian memory attention (which provides attention between the PSC and queries).



**Strengths**
- Integration of semantic information into 3D is important [a6FD]
- Proposed method is interesting and allows for integration of many foundation models [a6FD,RUFb,UxH3,2XXj]
- Experiments show the method is effective [a6FD]
- Qualitative results are convincing [UxH3], and interesting downstream task (robot manipulation) is presented [UxH3,a6FD]


**Weaknesses**
Reviewers initially noted the following weakneses:
- Poor writing and presentation with some aspects not clearly explained
  - More explanation of Gaussian Memory Attention needed [a6FD]
  - Many implementation and experimental details missing [RUFb]
  - Various typos [RUFb,2XXj]
- Weaknesses in experiments 
  - Questions about informativeness of experiments [a6FD]
  - Missing comparisons with baselines [RUFb]
  - Missing analysis of memory, training time, and performance [UxH3]
- Limited discussion of challenges [a6FD]

Most of the above where addressed during the author response period.

In addition the AC note that the related work is overally broad and not that relevant, while missing discussion of the most relevant related work (especially slightly earlier methods than F-3DGS and F-Splat (see **Suggested Revisions** for details). 

**Recommendation**
Overall reviewers are positive on this work.  The AC also believes that the proposed method of combining many different features into Gaussian Splats can be of interest to the community and thus recommend acceptance.

**Suggested Revisions**
- Please provide more discussion of related work that actually distills / projects features / information from 2D foundation models into 3D.  The related work section on Foundation Models and Scene Graph and Video Memory discuss work that is not that relevant (e.g. L097-105).  It is recommended that more space and discussion is dedicated to more relevant related work (see suggested references below). 
- Please consistently bold best performing numbers in Table 1 and Table 2 (not just the entries where M3 is the best)
- Please use \citep for parenthetical references (e.g. "ConceptGraphs Gu et al. (2024)" => "ConceptGraphs (Gu et al. 2024)")
- Please check and fix caption for Figure 5: "Illustration of patch-level visual embedding extraction their applications"
- L360: "Table." => "Table"
- L374: "Baseline Implementation" => "Baseline Implementation."

**Suggested references**

With 3D Gaussian Splats:

[1] LangSplat: 3D Language Gaussian Splatting, Qin et al. CVPR 2024

[2] GS3LAM: Gaussian Semantic Splatting SLAM, Li et al. ACMMM 2024

[3] Semantic Gaussians: Open-Vocabulary Scene Understanding with 3D Gaussian Splatting, Guo et al. 2024

[4] CLIP-GS: CLIP-Informed Gaussian Splatting for Real-time and View-consistent 3D Semantic Understanding, Liao et al. 2024

With 3D Gaussian Splats (2 foundation models)

[5] FMGS: Foundation Model Embedded 3D Gaussian Splatting for Holistic 3D Scene Understanding, Zuo et al. IJCV 2024

With NeRF models:

[6] FeatureNeRF: Learning generalizable NeRFs by distilling foundation models [Ye et al. ICCV 2023] 

[7] Decomposing NeRF for Editing via Feature Field Distillation [Kobayashi et al. NeurIPS 2022]

[8] Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representations [Tschernezki et al. 3DV 2022]

With NeRF models (two foundation models)

[9] LERF: Language embedded radiance fields [Kerr et al. ICCV 2023], 

[10] M2DNeRF: Multi-Modal Decomposition NeRF with 3D Feature Fields [Wang et al. 2024]

With 3D point clouds:

[11] PointCLIP: Point Cloud Understanding by CLIP [Zhang et al. CVPR 2022]

[12] Image2Point: 3D Point-Cloud Understanding with 2D Image Pretrained Models [Xu et al. ECCV 2022]

[13] OpenScene: 3D Scene Understanding with Open Vocabularies [Peng et al. CVPR 2023]