Probing the 3D Awareness of Visual Foundation Models
Authors: name: Mohamed El Banani; name: Amit Raj; name: Kevis-Kokitsi Maninis; name: Abhishek Kar; name: Yuanzhen Li; name: Michael Rubinstein; name: Deqing Sun; name: Leonidas Guibas; name: Justin Johnson; name: Varun Jampani
Published: 2024-04-12 17:58:04

Abstract: Recent advances in large-scale pretraining have yielded visual foundation
models with strong capabilities. Not only can recent models generalize to
arbitrary images for their training task, their intermediate representations
are useful for other visual tasks such as detection and segmentation. Given
that such models can classify, delineate, and localize objects in 2D, we ask
whether they also represent their 3D structure? In this work, we analyze the 3D
awareness of visual foundation models. We posit that 3D awareness implies that
representations (1) encode the 3D structure of the scene and (2) consistently
represent the surface across views. We conduct a series of experiments using
task-specific probes and zero-shot inference procedures on frozen features. Our
experiments reveal several limitations of the current models. Our code and
analysis can be found at https://github.com/mbanani/probe3d.