NeRF-MAE: Masked AutoEncoders for Self-Supervised 3D Representation
  Learning for Neural Radiance Fields
Authors: name: Muhammad Zubair Irshad; name: Sergey Zakharov; name: Vitor Guizilini; name: Adrien Gaidon; name: Zsolt Kira; name: Rares Ambrus
Published: 2024-04-01 17:59:55

Abstract: Neural fields excel in computer vision and robotics due to their ability to
understand the 3D visual world such as inferring semantics, geometry, and
dynamics. Given the capabilities of neural fields in densely representing a 3D
scene from 2D images, we ask the question: Can we scale their self-supervised
pretraining, specifically using masked autoencoders, to generate effective 3D
representations from posed RGB images. Owing to the astounding success of
extending transformers to novel data modalities, we employ standard 3D Vision
Transformers to suit the unique formulation of NeRFs. We leverage NeRF's
volumetric grid as a dense input to the transformer, contrasting it with other
3D representations such as pointclouds where the information density can be
uneven, and the representation is irregular. Due to the difficulty of applying
masked autoencoders to an implicit representation, such as NeRF, we opt for
extracting an explicit representation that canonicalizes scenes across domains
by employing the camera trajectory for sampling. Our goal is made possible by
masking random patches from NeRF's radiance and density grid and employing a
standard 3D Swin Transformer to reconstruct the masked patches. In doing so,
the model can learn the semantic and spatial structure of complete scenes. We
pretrain this representation at scale on our proposed curated posed-RGB data,
totaling over 1.8 million images. Once pretrained, the encoder is used for
effective 3D transfer learning. Our novel self-supervised pretraining for
NeRFs, NeRF-MAE, scales remarkably well and improves performance on various
challenging 3D tasks. Utilizing unlabeled posed 2D data for pretraining,
NeRF-MAE significantly outperforms self-supervised 3D pretraining and NeRF
scene understanding baselines on Front3D and ScanNet datasets with an absolute
performance improvement of over 20% AP50 and 8% AP25 for 3D object detection.