3D-SPATIAL MULTIMODAL MEMORY
Authors: Xueyan Zou
Yuchen Song
Ri-Zhao Qiu
Xuanbin Peng
Jianglong Ye
Sifei Liu
Xiaolong Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Foundation Model
Gaussian Splatting
Large Multimodal Model
Robotics
Primary area: foundation or frontier models, including LLMs

Abstract: We present 3D Spatial MultiModal Memory (M3), a multimodal memory system designed to retain information about medium-sized static scenes through video sources for visual perception. By integrating 3D Gaussian Splatting techniques with foundation models, M3 builds a multimodal memory capable of rendering feature representations across granularities, encompassing a wide range of knowledge. In our exploration, we identify two key challenges in previous works on feature splatting: (1) computational constraints in storing high-dimensional features for each Gaussian primitive, and (2) misalignment or information loss between distilled features and foundation model features. To address these challenges, we propose M3 with key components of principal scene components and Gaussian memory attention, enabling efficient training and inference. To validate M3, we conduct comprehensive quantitative evaluations of feature similarity and downstream tasks, as well as qualitative visualizations to highlight the pixel trace of Gaussian memory attention. Our approach encompasses a diverse range of foundation models, including vision-language models (VLMs), perception models, and large multimodal and language models (LMMs/LLMs). Furthermore, to demonstrate real-world applicability, we deploy M3’s feature field in indoor scenes on a quadruped robot. Notably, we claim that M3 is the first work to address the core compression challenges in 3D feature distillation.

Published as a conference paper at ICLR 2025
M3: 3D-SPATIAL MULTIMODAL MEMORY
Xueyan Zou Yuchen Song Ri-Zhao Qiu Xuanbin Penglo Jianglong Ye Sifei Liu", Xiaolong Wang
o Core Contribution
UC San Diego NVVIDI
https: /ttrss////-sppcaalaame.gogoogoo io
ial
Embeddings
Memory
Render
Render
CLIP
SigLIP
PSC
PSC
PSC
Render
Ronder
Runder
SEEM
LLaMA3
LLaMAv
Embedding Space
Figure 1: Our proposed MultiModal Memory integrates Gaussian splatting with foundation models to efficiently store multimodal memory in Gaussian structure. The feature maps rendered by our approach exhibit high fidelity, preserving the strong expressive capabilities the foundation models.
ABSTRACT
We present 3D Spatial MultiModal Memory (M3), multimodal memory system designed to retain information about medium-sized static scenes through video sources for visual perception. By integrating 3D Gaussian Splatting techniques with foundation models, M3 builds a multimodal memory capable of rendering feature representations across granularities, encompassing a wide range of knowl- edge. In our exploration, we identify two key challenges in previous works on feature splatting: (1) computational constraints in storing high-dimensional fea- tures for each Gaussian primitive, and (2) misalignment or information loss be- tween distilled features and foundation model features. To address these chal- lenges, we propose M3 with key components of principal scene components and Gaussian memory attention, enabling efficient training and inference. To validate M3, conduct comprehensive quantitative evaluations of feature similarity and downstream tasks, as well as qualitative visualizations to highlight the pixel trace of Gaussian memory attention. Our approach encompasses diverse range of foundation models, including vision-language models (VLMs), perception mod- els, and large multimodal and language models (LMMs/LLMs)). Furthermore. to demonstrate real-world applicability. we deploy M3's feature field in indoor scenes on quadruped robot. Notably, we claim that M3 is the first work ad- dress the core compression challenges in 3D feature distillation.
INTRODUCTION
Human perception encompasses the world across spatial dimensions. When encountering static elements in the visual environment, individuals tend to organize and store knowledge progressively, starting from a coarse overview and refining it into finer details. For instance, we can recall our daily surroundings with varying levels of detail, ranging from high-level layout to specific part- level features. However, for larger-scale environments, our understanding tends to remain more coarse and generalized. Previous works, such NeRF (3 and 3DGS [16], have demonstrated the

---

Published as a conference paper at ICLR 2025
ability to store scene-level information the pixel level for intermediate-sctt scenes. However, these models lack the capability to retain the semantic understanding of the scene like humans.
In this study, aim to develop spatial memory system for static scenes, capable of processing static scene video clips spanning spatial horizons. The primary objective to store all human- processable information in format that is precise, efficient, and amenable future interactive queries. Our approach leverages 3D Gaussian splatting techniques and incorporates features ex- tracted from foundation models to construct scenes imbued with semantic knowledge. The selection of Gaussian splatting as our structural format was motivated by two key considerations: First, the need to address video redundancy through efficient compression, and second, the requirement for information representation. Gaussian splatting inherently provides a framework for representing the smallest units of information as Gaussian primitives as well as naturally eliminating the spatial redundancy, aligning well with the motivations.
Previous feature splatting works such as F-3DGS [51] and F-Splat (26 directly distill 2D feature maps obtained from foundation models into 3D Gaussians via differentiable rendering. We observe two key issues: First, due to the computational limitations, the feature vector dimensions in Gaussian primitives are significantly reduced compared to the original 2D feature maps (typically 16-64 versus 1024), potentially causing an information bottleneck. Second, while the original feature maps may not be inherently sD-onssit,n enforcing 3D consistency in the Gaussians can cause misalignment between the original and distilled features. Consequently, the distilled feature may not accurately capture the knowledge embedded in the foundation model.
To address these issues, we present MultiModal Memory (M3), a better integration of Gaussian splatting and multimodal foundation models that efficiently store expressive multimodal memory in a Gaussian structure, facilitating spatial queries. Specifically, we propose to store the original high-dimensional 2D feature maps in memory bank called principal scene components and use the low-dimensional principal queries from 3D Gaussians as indices. Instead of directly distilling the 2D features into 3D embeddings, we apply Gaussian memory attention between the principal scene components and principal queries to render the foundation model embeddings in 3D scene.
In this way, we combine the best of both foundation models and Gaussian splatting: preserving the high expressive ability of the original foundation model feature maps while maintaining 3D- consistent, low-dimensional Gaussian structure of the scene. Furthermore, we also design a heuristic algorithm to minimize redundancy in the memory bank by reducing the raw features from the video stream. These reduced features are referred to as Principal Scene Components. Example feature maps rendered by M3 are visualized in Fig. [
To evaluate M3, we employ a diverse set of foundation models, including vision-language models, LMM/LLMs, and perception models. We adopt both low-level metrics (e.g. PSNR) to assess the model's 's feature memorization capability and bigh-level metrics (e.g. mloU, IR, TR) to assess its per- formance on downstream perception tasks. Extensive experiments demonstrate that M3 outperforms previous works in both memorization and downstream tasks while maintaining low computational costs. Lastly, we deploy M3 quadruped robot platform for grasping, showcasing its potential for real-world generalization from single-scene, multi-scene, and long-horizon tasks.
2 RELATED WORK
Foundation Models. The field of multimodal learning has seen remarkable progress, leading to the development of diverse foundation models. In the vasion-language domain, models such as CLIP (28), Florence (40, (55,, and the recent SigLIP [47) employ ViT-style (5) transformer architec- tures to align visual and linguistic representations. For vision-specific tasks. SAM [17 29 excels in pert-level clustering, while DINO 4 24 advances self-supervised representation learning. In combines OCR and text classification for compre- hensive document analysis. The language domain has seen significant advancements in reasoning and Mistral [4
document understanding, LayoutI M
capabilities, exemplified by the LLaMA [36 37 6
15
series. While these works
have pushed language reasoning to new heights, recent studies like 33 35 7 explore mixture-of- experts approaches to enhance visual representation learning in foundation models. These devel- opments, along with advanced models such as ChatGPT (I] and Claude (2] form the backbone of modern Multimodal Large Language Models (MLLMs), paving the way for more sophisticated AI systems.
2

---

Published as a conference paper at ICLR 2025
Dertieal
A pank bansni
Grussian Spietting
Foundation Models
Caption ng
LLANN
*A doves
key
A netal try [.)
Gecunding
MutBi model Multi ginuubriity Footure Extraction
Goucsion Feoture Roid
Interactive Results
Figure 2: A scene (V) is composed of both structure (S) and knowledge (I). To model these, we leverage multiple foundation models to extract multi- multi-gramularity scene knowledge, and employ 3D Gaussian splatting to represent the spatial structure. By combining these techniques, we construct spatial multimodal memory (M3), which enables downstream applications such retrieval, cap- tioning and grounding.
3D Gaussians and Feature Field. NeRF [23 revolutionized 3D scene representation, but its im- plicit nature caused slow rendering and training. 3D Gaussian Splatting [16] (3DGS) emerged as a faster, more explicit alternative, enabling rapid training and real-time rendering. Since then, 3DGS has been enhanced: H3DGS 44 improved large-scale rendering, Mip-Splatting tackled anti-aliasing for high detail, and WildGaussians [19] addressed occlusion and appearance changes. Grendel-GS [50] enabled multi-GPU training for efficiency with larger datasets. Researchers began incorporating 3D feature fields into neural rendering pipelines, moving from NeRF- models like F3RM 32 to 3DGS- based ones. Feature 3DGS [51] added feature representations to 3DGS, leading to advancements like Feature Splatting [26 13 for language-driven scene synthesis and LEGaussians [ for open-vocabulary scene understanding. LiveScene [27) introduced interactive radiance fields, while recent work (46) focuses on improving 2D features with 3D-aware fine-tuning for better 2D-3D integration.
Scene Graph and Video Memory. Long-horizon scene understanding encompasses both spatial and temporal dimensions. For spatial modeling, scene graphs have been prominent: ConceptFu- sion [1 introduced open-set multimodal 3D mapping, ConceptGraphs [8] extended this to open- vocabulary 3D scene graphs, and Hierarchical Open- Vocabulary 3D Scene Graphs [39] applied these concepts to language-groundee navigation. Beyond Bare Queries [2]] and Open Scene Graphs [22] further demonstrated their utility in object retrieval and navigation. However, these approaches of- ten rely on heuristic edge/node construction and lack direct LMM integration via embeddings. For temporal aspects, previous works have focused on using memory bank embeddings to store infor- mation across frames. For instance, MA-LMM 9 MovieChat (34,, and Hicrarchical Memory 38 introduced various memory augmentation techniques for video understanding. Flash VStream and Streaming Long Video Understanding (25] concentrated on real-time processing of long video streams. While these temporal methods integrate better with LMMs, they face challenges such as image over-compression (representing an entire frame with a single embedding), frame re- dundancy (adjacent frames containing overlapping spatial information), and lack of explicit spatial information. Our 3D Gaussian approach bridges this gap, combining spatial precision with temporal flexibility and LMM compatibility.
48
3 METHOD
3.1 3D-SPATIAL MULTIMODAT. MEMORY (M3) PIPELINE.
real-world visual perception scene (V) consists of both structure (S) and knowledge (I). The structure of Visual Granularity (VG) can range from the fine details such as leaf shapes, to large- scale elements, such as city layouts. Concurrently, the Knowledge Space (KS) spans scales from specific information, such as leaf species (e.g. red maple leaf) to a comprehensive interpretation a view (V.). Gaussian splatting serves as a framework for
(e.g. The space needle in Seattle...) constructing scene structure with finest granularity, represented gaussian primitives, while foun- dation models provide vast world knowledge spanning various scales for scene knowledge. The organic integration of Gaussian splatting and Foundation Models infuses scene structure with multi-
3

---

Published as a conference paper at ICLR 2025
enathare paint parked
CLIPSigUP
LLaMAaN
Banderig
Rosuussin
DINOV2
Queyy V
Hattiooda boout
Foundatio Modes 0g
SEEM
LLMMANY
Dinbuo
SCOM
Principte Soone Component/ Monony Bank
Gaursitnn Hertory Atlurion
Gauasaan Pitmithom
33 Couzstrr
(a) Building the Spatial Muttimodal Memory
Altartinn
Pentixal Head
PrivichsSccre Oun gariel
Thraat
Prinsila
Siniiarity
Seftrax
Dendered Faotione
MLaMA
Livear
Orounding Hoad aEth
Pricepple QuArTy
Learned Gauastera
(b) Gaussian Memory Attention
Figure 3: Given a video sequence, we utilize foundation models (F) to extract raw features (R). These features are reduced using Algorithm[]] producing principal scene components (PSC), which are stored in a memory bank. We introduce optimizable attribute queries (q) to Gaussian primitives, and apply a Gaussian Memory Attention (Agm) mechanism to produce the final rendered features (R), which can be linked back to various heads the foundation models.
granularity knowledge, enabling the construction of full-stack Multimodal Memory of the scene with precise spatial information. To maintain efficiency while preserving the global representation of foundation model features, we compress the extracted features from foundation models into prin- cipal scene components (PSC) for each scene and learn to probe the scene via Gaussian Splatting parameters, denoted as principle query (Qp). Ultimately, leveraging the rendering capabilities of Gaussian Splatting, we can dynamically populate the GS structure with multi-granularity informa- tion spanning the entire view of the scene. Our pipeline is illustrated in Fig. 2
3.2
M3 PRELIMINARIES.
Visual Granularity (V9). Visual granularity (VG) typically represents the clustering pixel scope of an image, concept introduced in Semantic-SAM (2).. Given view V. E R (h, w denote the pixel dimensions) in the scene V, it is composed of multi-granularity segments ranging from individual pixels the full view (as illustrated the left part of Fig. ( represented by V. (V,1 V22 ...., V.m }. where V² E RP.33 is the 7th granularity of the view V.. is the number of pixels, and m denotes the total number of granularities. This multi-granullaity approach is introduced because humans naturally possess multi- muati-granularity recognition of the world for various utilities.
Knowledge Space (KSS). Different foundation models (F) focus on various aspects of knowledge. For instance, CLIP 28 and SigLIP [47] concentrate on image-level perception, while Semantic- SAM 20 emphasizes part-level visual grouping. contrast, LLaMA3/v (6) incorporates both local and global attention mechanisms. The features generated by these models occupy different knowl- edge spaces F(V.) E (KSS*, KSS "ESs,...,.SS KSS where is the total number of knowledge spaces here, emphasizing diverse aspects such as visual alignment (KSS)), semantics (KSS², reasoning (ICS"), and etc.
Principle Scene Components (PSC) and Principle Query (Qp). We extract foundation model features for each view, denoted as (V) (Ej E,,,,, E,,...E; for each foundation model (F.) and scene (V), where is the number of views. These foundation model features are represented as E; E Rllx where h, w denote the feature pixel dimensions. However, different views often contain redundant and similar features. We define the key features that construct the scene as Prin- ciple Scene Components (PSC), drawing inspiration from the terminology of Principal Component Analysis. The attribute within Gaussian representation responsible for indexing PSC is denoted as Principle Query (Qp), which is learnable parameters in each Gaussian primitive.
3.3 SPATIAL MULTIMODAL MEMORY
Build Scene Structure via 3D Gaussians, We formally define the input of M3 as a video sequence with frames, where each frame corresponds to view V.. 3D Gaussian splatting [16) is employed
4

---

Published as a conference paper at ICLR 2025
Algorithm Raw Feature (R) Similarity Reduction Algorithm
Input E glnxhxxsdd) (Raw Features), E (0, 1J (threshold), N (chonk size)
Output: PSC C R (Principle Scene Compuments)
simi stmilarityRodiuct.oo (R, 0) n + [R] Number of raw factures
I 0 [1 e of filtered indices
R
Normaliza raw features
U [0)" Daage mask, initially all ralse
for to L -1:
Ce (ÉL: [kk, (1) Current chunk
Se Ct.Rt similarity matrix chunk forj + |Ca1-1:
Ucii =0:
J S 20) Similar indices
iE =;
I 1U (kc+)) Salect Principle Components
WEJ:U; +-1
PSC (4, : PSC
fit the scene, with each view rendered by the Gaussian rasterizer. For each Gaussian primitive, the optimizable attributes include the centroid (x E IR*), rotation quaternion (r E Ra) storing the rotation and scaling matrix, opacity value (a E R and spherical harmonics (sh R²). To model the Principle Scene Components (PSC), we introduce an additional optimizable attribute: principle accommodate various foundation models. Each foundation model utilizes degrees from the Op E TR' These degrees are rendered alongside Gaus- sian parameters to produce bew-based principle queries Q with shape [H, W, 1]. Following [51],
queries (g E R') with flexible dimensionality
the colors and principle queries are rendered as:
C= Σ CoaTii
O Σ where T, = [I a5)
VEN
iEN
Here, N represents the set sorted Gaussians overlapping with the given pixel, and T, denotes the transmittance, defined as the product of opacity values of previous Gaussians overlapping the same pixel.
Extract Multi-Granularity Scene Knowledge. Upon preparing the attributes in the Gaussian primitives, we extract multi multi-granularity scene knowledge via foundation models. Different foundation models focus on different aspects of knowledge projection and granularity, as illustrated in Fig. 4 In this paper, we employ a set of foundation mod- ICLIP,SiqLIN
els F where LLaMAv is the vision-instruct version LLaMA3. For each view, we extract foundation model embeddings, formally expressed as F(V.) = E e [bx
DIOV2, LLaMA3, LLaMAv, SEEM),
SqUi
DRLDx2
We implement specific algorithms for projecting LLaMA3 language embeddings and SEEM [52] visual prompts into pixel-level features. For LLaMA3, we first use SoM [43 and Semantic-SAM to extract language descriptions for each region. The language prompt of each region is represented as T E 1f. where 4 is the number of regions extracted by Semantie-SAMs For SEEM, we utilize visual prompts corresponding to each region, with visual prompts for each image represented as O E Rll,.4, where l is the number of regions seg- mented by SEEM. We then splat the features to the pixel level, duplicating the prompts within each mask region, resulting in T and O being indexed to the dimension of [l
Figure 4: The UMAP visu- alization model embed- ding manifolds reveals dis- tinct shapes, reflecting dif- ferent focus.
After feature extraction, obtain raw features (R R(nkkxwdd)) for the full Scene with n views within each foundation model. These raw features span various granularities and knowledge spaces, providing comprehensive multimodal (vision and language) understanding of the scene. In cor- relation to 3D Gaussian Splatting, the smallest granularity component is at the pixel level, with the most level knowledge projection being the RGB color value.
5

---

Published as a conference paper at ICLR 2025
Input:
DINOV:
CLIP/SigLIP:
LLaMA3.1:
LLaMA3.2:
Leyer[1:n]
Vialon Encoder
Viaion Encoder
Language Encoder
Vision Encoder
Lenguage Encoder
Layer [n:-1]
Language Features
table
Figure 5: Illustration of patch-level visual embedding extraction their applications.
Compress Scene Knowledge to Memory. While the scene knowledge is extracted from foundation models F into raw feature space R. E R(n,Axwsd) the dimensionality is too high for storage and rendering in each scene. Previous works such as F-3DGS [5]] and F-Splat [26] address this issue through feature distillation. However, we observe two major problems with feature distillation: (a) The distilled feature experiences information loss compared the original feature due to feature compression (usually 16 to 64 dims sampling linear projection to the original dimension that is usually 1000 dims or more). (b) The upsampled feature may have misalignments with the original knowledge space of the foundation model, making it difficult be decoded by the original F. To resolve these issues, we first flatten the raw features (R) into R gixxkxwdd] and then perform simi- larity reduction on the first dimension using Algorithm| T The reduced raw feature represents principal scene component (PSC), also named as memory bank, serving the essential represen- tation of the scene. The memory bank or PSC has dimensionality R[(ad, where t depends on the similarity threshold we set. The reduction is effective due to the presence of many duplicated fea- tures in neighboring spatial pixels within view, or duplicated regions across views. We visualize the memory bank building process in Fig
Gaussian Memory Attention. Given view-based principle queries O E [H.,Wo)] that is raster- ized by gaussian primitives, and principle scene components PSC & Rll,dl, we perform Gaussian Memory Attention (Agm) ) to obtain the rendered feature in alignment with foundation models. With learnable random initialized memory projection W.. m. E RRndd, we formally define the Gaussian Memory Attention as follows:
R gm Q Softmax QQ x W. m PSC PSC.
This Gaussian memory attention links the Op with PSC and projects into the corresponding foundation model knowledge space. The attention process is depicted in Fig. b.
Scene Rendering and Deployment. Given the rendered features R for each foundation model aligning with the corresponding foundation model space, we can link back to the powerful functions of foundation models. We expect that for models like CLIP, SigLIP, SEEM, the rendered feature be directly used for vision language-based tasks such retrieval and grounding. For generative-base based models like LLaMA3, LLaMAV, we anticipate that the feature can be directly used in captioning or simple visual question answering tasks. Formally, express this X' F Faee(A) (R).
4
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets. To support extensive quantitative and qualitative evaluation, we perform experiments using several existing scene datasets 3 using a quadruped robot and a drone. Specifically, we use Garden (an outdoor scene) from Mip- NeRF360 3 Train from the Tank & Temples dataset [18, and PlayRoom well as DrJohnson from the Deephlending dataset [00. For the M3-Robot dataset, we collect images using two mobile robots. The Table-Top sequence is collected from RealSense 405D camera mounted on the end effector a Unitree ZI robot arm on Unitree B quadruped robot, where a human operator tele- operates the robot with remote obtain centripetal views of tabletop objects. Images in the Geisel sequence are collected by a tele-operated DJI Mini4-Pro drone. The collected images are processed by COLMAP 30 to obtain camera parameters and initialization.
18 10 and collected a custom robot dataset (M3-Robot)
Memory across multiple Foundation Models. The multi-modal memory mechanism allows M3 retain knowledge from many models, which differs from existing distillation- based methods that

---

Published as a conference paper at ICLR 2025
Dataset
Method
Param
Cosine|
DINOV2
L21
Cosine]
CLIP
L22
Cosincl.
SigLIP
L21
Cosine
SKEM
1.21
Cosinel
LLaMA3
L21
Cosinel
LLaMAv
L21
Train
E-Splar
61M
1.98335
0,4779
0.2500
0,7025
X7
61
0.3790
1,0108
0.3350
0.1540
0.3692
0.3528
0.1063
0.1054
0.4993
0.0150
0.6288
46.48
M3
35M
0.5321
1,681
0.3140
0,2800
0.2811
0.5096
0.1389
0.2251
0.4401
0.0253
0,7069
53.43
Garden
61M
1,9567
9.4675
0.3314
0,7443
F-3DGS
51
61M
0.2295
0.6033
0.2105
0.0945
0.2697
0.2585
0.1071
0.1422
0.4139
0.0141
0.4913
43.08
M3
35M
0.5701
1.7279
0.3168
0.2876
0.2927
0.0004
0.1839
0.3469
0.3387
0.0217
0.7233
58.04
Drjahnson
F-Splat
26
61M
0.8107
2,0333
0,6689
0.7877
0.6826
0.7744
0.4650
10.411
0.3757
0.0145
0.8184
54,82
F-30GSS
61M
0.4190
1.1279
0.2166
0.3853
0.0150
0.6669
47.35
M3
35M
0.5878
1.7553
0.3435
0.2924
0.2975
0.5366
0.2456
0.4179
0.3175
0.0226
0.7224
52.68
Playroom
F-Splat 26
61M
0.7956
1.96401
0.6458
0.7808
0.6899
0.7678
0.4745
16,873
0.3915
0.0156
0.8185
59.42
F-3DGS
61M
1.2193
0.2103
M3
35M
0.6074
1.7545
0.3260
0.2987
0.2951
0.5623
0,2560
0.4584
0.3555
0.0241
0.7288
57.38
Table 1: Feature Distance in comparison with distillation methods that use similar or higher budgets
across datasets and foundation models.
CLIP
LigLIF
Dataset
Method
#Param
mloU
cloU
AP50
AP60
I2T@
12T@5
12T10
T21@1
T2
T2110
Train
Ground Truth
25.3
26.3
14.7
3.3
81.5
97.3
100.0
71.0
89.4
92.1
F-3GGS 51
61M
24.2
24.3
16.3
7.1
2.6
13.2
28.9
0.0
2.6
18.4
M3
35M
25.4
26.5
19.6
12.5
55.2
84.2
92.1
52.6
84.2
92.1
Playroom
Ground Truth
25.6
24.2
9.6
3.0
96.5
100.0
100.0
62.0
96.5
100.0
F-DGSS S1
6IM
23.8
21.4
11.9
3.0
79.3
96.6
96.6
31.0
79.3
89.7
M3
35M
23.1
23.1
11.9
5.9
72.4
100.0
41.3
65.5
68.9
Geisel
Ground Truth
19.5
21,4
5.3
0.0
100.0
100.0
100.0
60.0
85.7
91.4
F-3DGS 51
61M
19.0
20.4
14.1
1.2
45.7
94.3
100.0
0.0
20.0
34.3
M3
35M
21.8
23,5
16.5
11.8
100.0
100.0
100,0
71.4
85,7
94.2
Table 2: Feature/RGB metrics for all foundation models and scene.
only distill a few (2-3) models. Specifically, as provided in Sec. 3.3 we employ 6 foundation models to resemble human memory of different aspects. Each model has different granularity and focus of different semantics: image-level vision-language understanding via CLIP (28) as well as SigLIP 47); pixel-level semantic understanding via SEEM [52]: sulf-supervised structural feature
via 2
24 and 1LLMM3.113222 66 for multi-modal understanding and reasoning.
In Fig. [ we provide comprehensive illustration of how extract features from foundation mod- els. The extracted features are marked in orange in alignment with language representations option- ally continued to be the input of the language Encoder.
Loss Computation. For each input image, we extract patch-level embeddings from the aforemen- tioned models. Previous methods [26] [11 compute the patch-wise distance loss on the rendered features, this not only has a high volume of GPU memory consumption that hinders parallel training for all the foundation models but also creates artifacts when downsampling the feature. In compen- sate, we use point- bastd loss, where we sample 2000 points ranging from both predict and ground truth features for distance loss computation. This largely reduces the computation overhead for training, as shown in Table. [
Low-level Evaluation Metrics. To systematically evaluate multi-modal memory, we use evaluation metrics ranging from low/pixel-leeel to high-level downstream tasks. In particular, the low-level evaluation metrics evaluate pixel-level image quality. For rendered image quality on evaluation views (views not provided in training), we use common metrics (PSNR, SSIM, and LPIPS [49] as Kerbl et al. [6]. For feature quality, we report cosine and L2 distance.
High-level Evaluation Metries. High-level evaluation metrics, different from low-level ones, focus on evaluating downstream tasks of features. For discriminative models (28 47: 52), we will report commonly used metrics such as mloU (mean Intersection over Union), cloU (complete Intersection over Union), and AP (Average Precision). For retrieval, we will use IR (Information Retrieval at rank 1) and TR @ I (Text Retrieval rank 1).
4.2 QUANTITATIVE RESULTS
Baseline Implementation For quantitative experiments, we compare M3 with two recent distillation-basedd feature GS methods (26, [1]. For fair comparisons, we train all the methods in approximately 30,000 iterations (29,993 iterations for M3 due to last-batch data loader roundoffs). The reference training features are identical for different methods. For distillation- follow F-Splat (26) to render latent feature and then decode the latent features the embedding
methods,
7

---

Published as a conference paper at ICLR 2025
Dutasel
Method
PSNRT
RGB
Time
min.
Cavine
CLIP
121
Cosine
SigLIP
1.21
Cosine
DINOV2
1.2]
Cosine]
SERM
1.2]
Cosine
LTMMA
12,
Casinet 12
LLaMAy
TaBlciop
ICLP
21:51
0.3100
0.2955
+SigLIP
21.84
0.3:00
0.3122
+DENOv2
21.79
-15
0.3101
0.2956
0.3123
0.0005
0.5161
L6057
+SEEM
21.93
-20
0.3101
0.2956
0.3123
0.005
0.5156
1,6488
0.0472
0.1013
+LLaMA3
21.97
-03
0.3:01
0.2956
0.3122
0.0055
0.5160
L,0866
0,0472
0.1012
0.3028
0.0246
+LLaMAw (AID)
21.95
445
0,3:00
0.2956
0.3122
0.0055
0.5157
1.009|
0.0472
0.1013
0.3628
0.0246
0.7262
59.92
Table 3: Ablation on the number of foundation models in M3.
Degree
# Params
Ieeation
Cosinel
L.1
CTPP
mloU
AP50
Cosini
121
SP
mloU
AP50
CasineL
DINOW
[2]
CosineL
SEEM
L21
Cosine
LLaMAT
[2.
8x6m48
14.8
0,3290
0,2000
25.4
25.3
19.6
14.6
0.2913
0.2938
0.5239
0.5277
21.8
19.4
48
0.5755
0.5645
1.7835
0.1672
0.3058
0.249
0.4504
04117
0,265
16x6-96
21.5M
30E
7k
0.3206
0.3120
0.2842
0.2500
25.3
25.7
20.6
19.0
0.2903
0.2866
0,3172
0,5227
24.3
23.2
8.1
10.3
0,5077
0.3335
1.7239
1.7513
0.1338
0.1828
0.3056
0.247
0.4504
0.4300
00261
0.0263
32x6-192
3488
30k
0.3063
0.3152
0.2735
0.2792
26.7
26.
22.8
0.2866
0.2814
0.5094
25.5
25.7
11.9
11.4
0.5318
0.5515
1.7198
1.6807
0.0972
0.1269
0.1553
0.2139
0.4101
0.446
0.0253
0.0256
64x6=2884
61.4M
7k
0.3049
0.2917
0.2734
28.1
25.4
239
23.9
0.2802
02721
0,5079
0.4957
28.5
27.8
13.5
13.5
0.5350
0,5099
1.3358
1.6810
0.055
0.1012
0.1616
0.42%
0.4348
0.0248
Table 4: Ablation on the dimensions and distillation for each foundation model.
space of reference features with multi-head MLP. For all methods, the optimization of both latent features/memory and decoders is trained from scratch for each scene.
Low-Level Results. We report the main quantitative results in Tab. 1 where the average training time and the auxiliary low-level metrics are reported. Our method, M3, outperforms F-Splat while reducing significantly compute than F-3DGS, SEEM and LLaMA3 features extraction failed on F- Splat, which we assume was mainly due to the ground truth feature extraction procedure, where duplication was performed to each segmentation to get pixel-level features.
Downstream Results. The downstream evaluation results of grounding and retrieval are shown in Table. 2 The ground truth grounding dataset of Train, Playroom, and Geisel is generated by SoM O with semantic-SAN) for mask labels and GPT4-o [] to generate the caption. Example data are shown in the Appendix. We evaluate all the images in the validation sets of the three datasets. The grounding results clearly show that M3 is better than F-3DGS with half of the parameters, the gap is non-trivial especially when looking at the AP50/AP60 columns. In addition to grounding results, we also evaluate M3 on image text retrieval, similar to grounding we use GPT4-o to generate ground truth data for three datasets. The example data are also shown in the appendix. Compared to grounding performance, M3 performs much better than F-3DGS text retrieval, the positive example is the evaluation image, and the negative pairs are generated by COCO datasets. We believe the large gap in the retrieval results is taken from Gaussian memory attention, where the rendered features are aligned with the original foundation model much better. When we find the correct embeddings in the dataset, this benefit gets enlarged.
retrieval results. For image
Ablation Results. Table. A shows the ablation of the number of foundation models involved in M3. We gradually added foundation models from simpler single-modal models to more advanced multi- modal models. While maintaining a very efficient training time, our method has independent results from different foundation models. Our implementation based on Grendel-GS, where the training procedure is efficiently paralleled. In Table. 4 the balance of memory footprint, training iterations, and performance. The table clearly shows that increasing the number degree will generally improve the performance on all metrics. While having 16 degrees for each foundation model is enough to obtain a reasonable performance. That is what the number is reported in the paper. In addition, increasing training iteration will generally increase the performance, while 1/4 of the training budget (7k) would usually get a reasonable performance.
we ablate the computation budget on training M3 in
4.3 QUALITATIVE RESULTS
M3 consistently demonstrates superior performance across diverse datasets as shown in Fig. 6 by effectively preserving fine-grained details and ensuring smooth, coherent feature representations. The method excels at retaining intricate details such as the textures of chairs and the fine features of books, highlighting its ability to capture micro-level information. This clear layering contributes to rich semantic understanding within the scenes.
Furthermore, M3 handles overlapping objects exceptionally well, as evident in the Playroom dataset, where complex arrangements rendered with accurate structural information. The outputs from various foundation models are consistently high-quality, each retaining spatial structures and se- mantic information at different granularities. This demonstrates M3's capability capture both
8

---

Published as a conference paper at ICLR 2025
Input
CLIP
LLaMA3
LLaMA
SEEM
SigLIP
Figure 6: Qualitative results across datasets using M3. The figure showcases the consistent perfor- mance of the M3 across various datasets (Garden, Playroom, Drjohnson, Table-top).
low-level spatial details and high-level semantic concepts, making it highly effective for tasks that require comprehensive scene understanding.
4.4 DEMONSTRATION RESULTS.
We also deployed M3 on quadruped robot platform to demonstrate the potential real world appli- cations of our model. In this experiment, we first tele-operate the robot to scan the table by taking a centripetal video with onboard camera. After memorizing the scene with M3, the robot is able to locate and grasp any object with text query decoded CLIP feature. With known robot pose through LiDAR, we are able to render any camera pose Te, with:
ca To Teo eo Thu x by Too To Th, x 1, Tb, x beT TT ToT w Th,, (3)
where b, and refer to camera, end effector, arm base, lidar, and world spectively, Note that to align with COLMAP coordinates, the camera pose needs modifying with COLMAR, Teo og Te.
We tested with the query "yellow bath duck" on the decoded CLIP feature, and as shown in Fig. the rubber duck is highlighted in red. The robot can then locate the 3D position of the targeted object with depth information from its depth camera and perform a grasping task.
O05 Rose
Goep Traj
Bnt
Pace
Figure 7: Real robot deployment.
Conclusion. This paper introduces M3, novel approach combining foundation models with Gaus- sian Splatting to create a spatial multimodal memory resembling human memory. M3 demonstrates superior downstream task accuracy with reduced training costs and shows practical utility when deployed on a real robot. One interesting future direction to design a reasoning module that is capable of directly operating the optimized memory bank, which leave to future study.

---

Published as a conference paper at ICLR 2025
Acknowledgement. This work supported, in part, by NSF CAREER Award IIS-2240014, and NSF CCF-2112665 (TILOS). This research project has benefitted from the Microsoft Accelerate Foundation Models Research (AFMR) grant program.
REFERENCES
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[2] Anthropic. The claude 3 model family: Opus, sonnet, haiku. Technical report, Anthropic, 2023. URL http:: anthropic.com Accessed: 2023-09-19.
[3] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip- nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF conference computer vision and patlern recognition, pp. 5470-5479, 2022..
[4] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. Proceed- ings of the IEEE/CVF international conference computer vision, pp.
2021.
[5] Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 11929, 2020.
[6] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
[7] Xiaoran Fan, Tao Ji, Changhao Jiang, Shuo Li, Senjie Jin, Sirui Song, Junke Wang, Boyang Hong, Lu Chen, Guodong Zheng, et al. Mousi: Poly-visual-expert vision-language models. arXiv preprint 4rXhv:2401.17221, 2024.
[8] Qiao Gu, Ali Kuwajerwala. Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, et al. Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 5021-5028. IEEE, 2024.
[9] Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefei Cao, Ashish Shah, Abhinav Shri- vastava, and Ser-Nam Lim. Ma-lmm: Memory- augnened large multimodal model for long- term video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13504-13514, 2024.
[10] Peter Hedman, Julien Philip, True Price, Jan-Michael Frahm, George Drettakis, and Gabriel Brostow. Deep blending for free-viewpoint image-based rendering. 2018.
[11] Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. Layoulmv3: Pre-training for document ai with unified text and image masking. In Proceedings of the 30th ACM Interna- tional Conference Multimedia, pp. 4083 4091,40911400 2022.
[12] Krishna Murthy Jatavallabhula, Alihusein Kuwajerwala, Qiao Gu, Mohd Omama, Tao Chen, Alaa Maalouf, Shuang Li, Ganesh Iyer, Soroush Saryazdi, Nikhil Keetha, et al. Conceptfusion: Open-set multimodal 3d mapping. arXiv preprint arXiv:2302.0722 2023.
[13] Mazeyu Ji, Ri-Zhao Qiu, Xueyan Zou, and Xiaolong Wang Graspsplats: Efficient manipula- tion with 3d feature splatting. arXiv preprint 2rXiv22409.02084, 2024.
[14] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
[15] Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, al. Mixtral of experts. arXiv preprint arXiv:2401 240.0008,, 2024.
10

---

Published as a conference paper at ICLR 2025
[16] Bernhard Kerbl, Georgios Kopanas, Thomas Leimktihler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 12 2023.
[17] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan Yen Lo, al. Segment anything. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 4015-4026, 2023.
[18] Arno Knapitsch, Jaesik Park, Qian-Yi Yi Zhou, and Vladlen Koltun. Tanks and temples: Bench- marking large-scale scene reconstruction. ACM Transactions on Graphics, 2017.
[19] Jonas Kulhanek, Songyou Peng, Zuzana Kukelova, Mare Pollefeys, and Torsten Sattler. Wildgaussians: 3d gaussian splatting in the wild. arXiv preprint arXiv:2407.08447, 2024.
[20] Feng Li, Hao Zhang, Peize Sun, Xueyan Zou, Shilong Liu, Jianwei Yang, Chunyuan Li, Lei Zhang, and Jianfeng Gao. Semantic-sam: Segment and recognize anything at any granularity. arXiv preprint arXiv:2307.04760 307.047777 2023.
[21] Sergey Linok, Tatiana Zemskova, Svetlana Ladanova, Roman Titkov, and Dmitry Yudin. Be- yond bare queries: Open-vocabulary object retrieval with 3d scene graph. arXiv preprint arXiv:2406.07113 2024.
[22] Joel Loo, Zhanxin Wu, and David Hsu. Open scene graphs for open world object-goal naviga- tion. arXiv preprint arXiv:2407.02477 2024.
[23] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Commu- nications of the ACM, 65(1):99-106, 2021.
[24] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafranicc, Vasil Khali- dov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint axXiv:2304.07193 2023.
[25] Rui Qian, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Shuangrui Ding, Dahua Lin, and Jiaqi Wang. Streaming long video understanding with large language models. arXiv preprint arXiv:2405. 16009, 2024.
[26] Ri-Zhao Qiu, Ge Yang, Weijia Zeng, and Xiaolong Wang. Feature splatting: Language-driven physics-based scene synthesis and editing. arXiv preprint 244 2024.
[27] Delin Qu, Qizhi Chen, Pingrui Zhang, Xianqiang Gao, Bin Zhao, Dong Wang, and Xuelong Li. Livescene: Language embedding interactive radiance fields for physical scene rendering and control. arXiv preprint arXiv:2406. 16038, 2024.
[28] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 9748-8763. PMLR, 2021.
(29) Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, al. Sam 2: Segment any- thing in images and videos. arXiv preprint 20Xiv:2408.00714 2024.
[30] Johannes I. Schonberger and Jan-Michael Frahm. Structure-fromt frommmotion revisited. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2016.
[31] Jinghuan Shang, Karl Schmeckpeper, Brandon B May, Maria Vittoria Minniti, Tarik Keleste- mur, David Watkins, and Laura Herlant. Theia: Distilling diverse vision foundation models for robot learning. arXiv preprint mrXiv:2407.20179 2024.
[32] William Shen, Ge Yang, Alan Yu, Jansen Wong, Leslie Pack Kaelbling, and Phillip Isola. Distilled feature fields enable few-shot language-guidke manipulation. arXiv preprint arXiv:2308.0793. 2023.
11

---

Published as a conference paper at ICLR 2025
[33] Jin-Chuan Shi, Miao Wang, Hao-Bin Duan, and Shao-Hua Guan. Language embedded 3d gaussians for open-vocabulary scene understanding. In Proceedings of the IEEE/CVF Confer- ence Computer Vision and Pattern Recognition, pp.
2024.
[34] Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, al. Moviechat: From dense token to sparse memory for long video understanding. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 18221-18232, 2024.
[35] Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian- 1: A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406 /6860, 2024.
[36] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Tim- othée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13997, 2023.
[37] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint WrXXi:2307.009265 2023.
[38] Yiqin Wang, Haoji Zhang, Yansong Tang, Yong Liu, Jiashi Feng. Jifeng Dai, and Xiaojie Jin. Hierarchical memory for long video qa. arXiv preprint arXiv:2407.006003 2024.
[39] Abdelrhman Werby, Chenguang Huang. Martin Büchner, Abhinav Valada, and Wolfram Bur- gard. Hierarchical open- neen-vocabulary 3d scene graphs for language-grounde robot navigation. In First Workshop on Vision-I anguage Models for Navigation and Manipulation ICRA 2024, 2024.
[40] Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing unified representation for variety of vision tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 4818-4829 2024.
[41] Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Floren- cio, Cha Zhang. Wanxiang Che, al. Layoutlmv2: Multi-modal pre-training for visually-rich document understanding. arXiv preprint arXiv:2012. 14740, 2020.
[42] Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. Layoutim: Pre-training of text and layout for document image understanding. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 1192- 1200, 2020.
[43] Jianwei Yang. Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set- of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310. 1441, 2023.
[44] Zehao Yu, Anpei Chen, Binbin Huang, Torsten Sattler, and Andreas Geiger. Mip-splatting: Alias-free 3d gaussian splatting, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19447-19456, 2024.
[45] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11433 2021.
[46] Yuanwen Yue, Anurag Das, Francis Engelmann, Siyu Tang, and Jan Eric Lenssen. Improving 2d feature representations by 3d-aware fine-tuning. arXiv preprint arXiv:2407.20229, 2024.
[47] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for lan- guage image pre-training. In Proceedings the the IEEE/CVF International Conference on Com- puter Vision, pp. 11975-11996 1986, 2023.
12

---

Published as a conference paper at ICLR 2025
[48] Haoji Zhang, Yiqin Wang, Yansong Tang, Yong Liu, Jiashi Feng, Jifeng Dai, and Xiaojie Jin. Flash-vstream: Memory-baut based real-time understanding for long video streams. arXiv preprint arXiv:2406.08085, 2024.
[49] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features as perceptual metric. In CVPR, 2018.
[50] Hexu Zhao, Haoyang Weng, Daohan Lu, Ang Li, Jinyang Li, Aurojit Panda, and Saining Xie. On scaling up 3d gaussian splatting training. arXiv preprint arXiv:2406. /8533, 2024.
[51] Shijic Zhou, Haoran Chang, Sicheng Jiang, Zhiwen Fan, Zehao Zhu, Dejia Xu, Pradyumna Chari, Suya You, Zhangyang Wang, and Achuta Kadambi. Feature 3dgs: Supercharging 3d gaussian splatting to enable distilled feature fields. In Proceedings of the IEEE/CVF Confer- Computer Vision and Pattern Recognition, pp. 21676-21688 2024.
[52] Xueyan Zou, Jianwei Yang. Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, Jianfeng Gao, and Yong Jac Lec. Segment everything everywhere all at once. Advances in Neural Information Processing Systems, 36, 2024.
13


Reviews:
[{"review_id": "3lQKxQ1qVX", "reviewer": "Reviewer_a6FD", "summary": "The article proposes a novel approach for video memory combining structural information from Gaussian Splatting with semantic information gathered from foundation models. The main challenge in this approach is the high-dimensionality of the semantic features for encoding in Gaussian splats. This has been addressed in the literature by feature distillation, but the authors argue that this leads to inefficient misaligned distilled features. \nThe authors proposed a novel attention approach to extract so-called Principle Scene Component. Experiments on a number of datasets appear to show that the proposed approach compares well in terms of fidelity and efficiency with the state-of-the-art, supporting the authors' claims.", "questions": "See the questions above in the weaknesses section, in summary: \n- Can you provide a rationale for Gaussian Memory Attention component and a comparison to standard approaches? \n- Can you provide an ablation study to evidence the impact of this novel component on the system's performance? \n- Can you provide a discussion of the limitations of the approach in the robotic scenario and the outstanding questions and limitations to be overcome for real-world application.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The efficient integration of structural and semantic features is an important problem for AI, and the proposed method is an interesting contribution. \n- The method appears to perform well on a broad range of measures and experiments. \n- The robotic application is a nice addition.", "weaknesses": "- The Gaussian Memory Attention component is a central contribution of the paper but is rather briefly discussed in the paper. It would be good to provide a clearer rationale for why this method was chosen and why it achieves the claimed result? How does it compare to standard memory mechanisms (such as in transformers), and what could be the alternative implementations. \n- Similarly, the article contains an extensive ablation study of the foundation models, but no analysis of the impact of this specific component on performance. It would be good to provide some evidence of that as an ablation study (eg, Gaussian Memory attention vs other attention implementation). This would provide valuable insight into the component's impact on overall performance. \n- The robotic experiment is interesting but it is unclear how informative it is about real-world applicability. It would be good to discuss possible challenges in scaling the approach to more complex/realistic environments and tasks, and to clarify the limitations of the current implementation as well as the future work that would be required for a real-world application.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jmkTPUy1Pa", "reviewer": "Reviewer_RUFb", "summary": "The paper introduces MultiModal Memory (M3), an integration of 3D Gaussian Splatting (GS) techniques with features from 2D Foundation Models (FMs) that replace standard feature distillation pipelines. The authors introduce two new concepts: a) principal scene components (PSC) which reduces the dimensionality of the FM features without sacrificing quality, thus reducing computational resources for training and storing features, and b) gaussian memory attention (GMA) which proposes to learn queries to memory as gaussian attributes (and not\nthe FM features themselves), thus dealing with the issue of misalignment between distilled and raw features of previous methods. The proposed memory can store features from multiple FMs, thus allowing a single GS structure to reconstruct multiple different features and solve many vision-language downstream tasks. The authors compare the quality of reconstructed features vs previous approaches in 4 scenes, do ablations in another 2 robotic-related scenes, provide comparisons with downstream tasks for 1 scene, and also provide a\ndemonstration with a mobile robot.", "questions": "1. How to use the reconstructed FM features to do the tasks you show in your Fig.2, namely: image captioning, language grounding, and retrieval? Please share some explanation of which feature is used for what task, and how, and provide qualitative comparisons with previous works F-3DGS and F-splat.\n\n2. Can the authors provide qualitative examples comparing the reconstructed features of M3 with those from previous methods, such as F-3DGS and F-Splat? This would help assess the quality improvement gained through PSC and GMA.\n\n3. Could the authors clarify if the term \"video memory\" implies the storage of multi-view data in static scenes or if there is an intention to extend this framework to dynamic scenes? If the latter, would the temporal dimension in the memory structure require additional model adjustments?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The introduction of Gaussian Memory Attention (GMA) is a novel and interesting way of linking Gaussian \\ Splatting techniques with feature distillation. Previous works learn downsampled versions of the target features to reduce memory footprint, and decode them back with MLPs. The downsampled feature is the learned attribute in GS. With GMA, the learned attribute is a query to a memory, which can hold features from multiple foundation models. This allows a single trained GS to reconstruct features from many possible \\ foundation models, instead of one GS per foundation model.\n\n2. The introduction of Principal Scene Components (PSC) and the similarity reduction algorithm seems to be able to provide a good balance between memory footprint (#degrees stored in memory) vs. reconstruction quality, according to Table 1.\n\n3. The authors integrate generative foundation models (Llama3v etc.) with their approach, while previous works only consider discriminative features (CLIP, DiNov2 etc.). The authors claim that the reconstructed features can be used directly as visual context to LLMs for generative tasks such as image captioning, which is I believe a great advancement of the distilled feature fields work, although they did not provide demonstrations for such applications.", "weaknesses": "1. The paper misses implementation details and qualitative results for downstream task applications. How to use the reconstructed FM features to do the tasks you show in your Fig.2, namely: image captioning, language grounding and retrieval? Please share some explanation of which feature is used for what task, and how, and provide qualitative comparisons with previous works F-3DGS and F-splat.\n \n2. Lacking quantitative comparisons with baselines for downstream tasks. Only Table 4 contains comparisons for 1 scene with grounding task but lacks comparison with F-Splat baseline. I believe the authors need to perform more experiments for downstream tasks for more scenes and compare with both baselines, so the final performance-efficiency tradeoff from the introduction of PSC and attention memory can be appreciated.\n\n 3. Also, currently the paper only shows reconstructed features from their M3 method. I believe the paper would benefit from a comparison of reconstructed features between previous approaches and M3, in order to appreciate qualitatively the effect of introducing the PSC method for reducing feature dimensionality and Gaussian Memory for decoding back the original feature.\n\n 4. There is some confusion regarding the inclusion of the temporal aspect in the memory. The authors name their structure video memory and say that it holds medium-time video clips for static scenes. However, from the method and experiments, it seems that the temporal aspect has not been a focus of the paper, as the scenes are static, and the downstream tasks are only at the image or pixel level (captioning, grounding, etc.). Do the authors mean that the video contains the multi-view collection trajectory of an agent? If that is the case, I believe it should not claim to be a video memory and model the temporal aspect, as the temporal dimension in the video is just the dimension of new views of the same static scene.\n\n 5.  I believe the paper would benefit from better clarity when presenting the method. The authors introduce many concepts by naming them (VG, KS, V) but do not provide input domains for such concepts therefore lacking precision. For example, in line 189, the authors write for a view V ∗ ∈ V, its V ∗ = {v 1 , v 2 , ..., v n }. What is the domain of V and V*? What are the intermediate segments vi? Later, in line 200, they write F ∗ (V) = {E 1 , E 2 , ...E n } for the foundation model features of a scene and say that n is the number of views. This seems a bit contradicting to me, as before they mentioned V* as a particular scene, and n the number of multi-granularity segments of that scene. Which one is it, n is a number of scenes or a number of segments inside a scene? \n\nMINOR:\n - (line 250) as shown in? \n - Table 4 is mentioned before Tables 2 and 3. Maybe rename?\n - (line 062): [...] may not be be inherently 3D-consistent [...]\n - (line 064): [...] knowledge embbeded in [...]\n - (line 080): [...] outperforms what?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BokjHC96ro", "reviewer": "Reviewer_UxH3", "summary": "The paper presents a method for integrating foundation model knowledge into a structural representation of the scene (here Gaussian Splatting). The approach is capable of integrating several foundation models' knowledge into Multimodal Memory. To this end, the authors propose Principal Scene Components, i.e. a method of feature reduction based on the assumption that multiple views of the scene contain redundant information. The paper evaluates the performance on a low level in feature space, and on a high level - through a downstream task, including robot deployment.", "questions": "- Do all the methods in comparison use the same underlying 3DGS model?\n- Lines 48-49 mention both static scenes and time windows, could you explain that?\n- Do you train reconstruction together, or do you start training with colour loss only? Alternatively, is the method sensitive to initialisation? With poor initialisation, the first iteration may be very noisy and rendered features very poor.\n- Could you pinpoint the exact component that makes M3 training so much faster than F-3DGS?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "I believe that the authors of the paper correctly identified the drawbacks of 3D feature supervision through 2D feature maps (i.e. potential loss of information with dimensionality reduction). To this end, the paper proposes a compression scheme for extracted features named principal scene components (PSC). I believe this idea to be sound and interesting. I appreciate that the idea includes a flexible parameter of similarity threshold allowing the user to decide on memory/knowledge trade-off. In principle, the method seems to be trained in a similar way to prior works, making me believe that the proposed memory module can be used interchangeably to prior works, and could be applied to an arbitrary 3DGS work. Another benefit is a significant speed-up with respect to F-3DGS. Qualitative results seem convincing. I believe the use of robot manipulation as the downstream task is a good demonstration of the capabilities of the proposed approach.", "weaknesses": "While I believe the authors propose an interesting method, I see some weaknesses in the paper. Firstly, there are some drawbacks to the experimental setup:\n- F-3DGS evaluated their approach through the task of novel view semantic segmentation. I see no reason why this could not be used in this paper as the experiment. This makes comparison between the two methods harder. Similarly, both F-3DGS and F-Splat do language-guided editing. Direct feature comparison is interesting and meaningful, however, we should also adhere to previously used evaluation protocols in order to facilitate fair comparisons.\n- Similar to the previous point, I believe it would be beneficial to have some datasets between the proposed method and prior work in common. \n- Further, the paper claims memory efficiency as a part of the method's benefits. Therefore an ablation and comparison to prior work from the memory perspective would be useful.\n- There should be a study on whether there exists a correlation between performance, training time, and memory footprint (e.g. quantitative metrics for similar training time to F-3DGS - does M3 reach the same performance or does it plateau; also a comparison between models for the same feature size, etc.).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gZyuMlRPa7", "reviewer": "Reviewer_2XXj", "summary": "This paper introduces 3D Spatial MultiModal Memory (M3), a video memory system aimed at retaining visual perception information over medium temporal and spatial scales. M3 employs 3D Gaussian splatting and incorporates multiple foundation models to create a multimodal memory capable of spanning visual details and encompassing a broad knowledge range. The authors address two primary limitations of prior feature splatting methods: high computational demands for storing dense features in each Gaussian primitive and misalignments with foundation model knowledge. M3 innovatively introduces principal scene components and Gaussian memory attention mechanisms to improve the efficiency of storage, training, and inference within the Gaussian Splatting Model. Evaluated across various foundation models, including vision-language and self-supervised models, M3 demonstrates robust performance, with successful deployment on a quadruped robot for grasping tasks, showcasing its practical viability in real-world applications.", "questions": "Please see weakness.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The idea is interesting and novel, and this direction is promising for diverse robotic tasks.\nThe method design makes sense. The figures and algorithm flowchart make the paper easy to understand.\nThe authors provide comprehensive coparisons in evaluation on M3 features (table 1, 2, 3, 4).", "weaknesses": "My concerns mostly lie on real robot experiments. I think one of the advantage of such feature field (also claimed in paper) is for various downstream robotics manipulation tasks. While, this paper shows a simple example in grasping a duck on the table. It could make this paper much stronger if the authors can demonstrate the promising potential of this study in various practical downstream tasks.\nSome example downstream tasks:\n1. Geometry-based or semantic-part-based tasks using DINO or diffusion features, such as geometry-based object retrieval/grasping[1, 2], planning objects to the goal configuration[3].\n2. Language-guided tasks using CLIP feature, such as like \"pick up the red cube and place it on the blue book.\" (example references: [4, 5])\n3. A multi-step task that combines different features, such as \"navigate to the kitchen, locate the mug with a cat design, and bring it to the living room table.\" These examples would showcase how M3 can leverage different types of features for more complex robotic applications.\n\nSome typos: for example, in figure 5, dinov->dinov2. In line 350&351, there are missing parts after `as shown in'.\n\n[1] SparseDFF: Sparse-View Feature Distillation for One-Shot Dexterous Manipulation. ICLR 2024.\n\n[2] Robo-ABC icon : Affordance Generalization Beyond Categories via Semantic Correspondence for Robot Manipulation. ECCV 2024.\n\n[3] D^3Fields: Dynamic 3D Descriptor Fields for Zero-Shot Generalizable Rearrangement. CoRL 2024.\n\n[4] Distilled Feature Fields Enable Few-Shot Language-Guided Manipulation. CoRL 2023.\n\n[5] NaturalVLM: Leveraging Fine-grained Natural Language for Affordance-Guided Visual Manipulation. RA-L 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)