Masked Scene Modeling: Narrowing the Gap Between Supervised and
  Self-Supervised Learning in 3D Scene Understanding
Authors: name: Pedro Hermosilla; name: Christian Stippel; name: Leon Sick
Published: 2025-04-09 09:19:49

Abstract: Self-supervised learning has transformed 2D computer vision by enabling
models trained on large, unannotated datasets to provide versatile
off-the-shelf features that perform similarly to models trained with labels.
However, in 3D scene understanding, self-supervised methods are typically only
used as a weight initialization step for task-specific fine-tuning, limiting
their utility for general-purpose feature extraction. This paper addresses this
shortcoming by proposing a robust evaluation protocol specifically designed to
assess the quality of self-supervised features for 3D scene understanding. Our
protocol uses multi-resolution feature sampling of hierarchical models to
create rich point-level representations that capture the semantic capabilities
of the model and, hence, are suitable for evaluation with linear probing and
nearest-neighbor methods. Furthermore, we introduce the first self-supervised
model that performs similarly to supervised models when only off-the-shelf
features are used in a linear probing setup. In particular, our model is
trained natively in 3D with a novel self-supervised approach based on a Masked
Scene Modeling objective, which reconstructs deep features of masked patches in
a bottom-up manner and is specifically tailored to hierarchical 3D models. Our
experiments not only demonstrate that our method achieves competitive
performance to supervised models, but also surpasses existing self-supervised
approaches by a large margin. The model and training code can be found at our
Github repository (https://github.com/phermosilla/msm).