M3: 3D-Spatial MultiModal Memory
Authors: name: Xueyan Zou; name: Yuchen Song; name: Ri-Zhao Qiu; name: Xuanbin Peng; name: Jianglong Ye; name: Sifei Liu; name: Xiaolong Wang
Published: 2025-03-20 17:59:12

Abstract: We present 3D Spatial MultiModal Memory (M3), a multimodal memory system
designed to retain information about medium-sized static scenes through video
sources for visual perception. By integrating 3D Gaussian Splatting techniques
with foundation models, M3 builds a multimodal memory capable of rendering
feature representations across granularities, encompassing a wide range of
knowledge. In our exploration, we identify two key challenges in previous works
on feature splatting: (1) computational constraints in storing high-dimensional
features for each Gaussian primitive, and (2) misalignment or information loss
between distilled features and foundation model features. To address these
challenges, we propose M3 with key components of principal scene components and
Gaussian memory attention, enabling efficient training and inference. To
validate M3, we conduct comprehensive quantitative evaluations of feature
similarity and downstream tasks, as well as qualitative visualizations to
highlight the pixel trace of Gaussian memory attention. Our approach
encompasses a diverse range of foundation models, including vision-language
models (VLMs), perception models, and large multimodal and language models
(LMMs/LLMs). Furthermore, to demonstrate real-world applicability, we deploy
M3's feature field in indoor scenes on a quadruped robot. Notably, we claim
that M3 is the first work to address the core compression challenges in 3D
feature distillation.