HaLo-NeRF: Learning Geometry-Guided Semantics for Exploring
  Unconstrained Photo Collections
Authors: name: Chen Dudai; name: Morris Alper; name: Hana Bezalel; name: Rana Hanocka; name: Itai Lang; name: Hadar Averbuch-Elor
Published: 2024-02-14 14:02:04

Abstract: Internet image collections containing photos captured by crowds of
photographers show promise for enabling digital exploration of large-scale
tourist landmarks. However, prior works focus primarily on geometric
reconstruction and visualization, neglecting the key role of language in
providing a semantic interface for navigation and fine-grained understanding.
In constrained 3D domains, recent methods have leveraged vision-and-language
models as a strong prior of 2D visual semantics. While these models display an
excellent understanding of broad visual semantics, they struggle with
unconstrained photo collections depicting such tourist landmarks, as they lack
expert knowledge of the architectural domain. In this work, we present a
localization system that connects neural representations of scenes depicting
large-scale landmarks with text describing a semantic region within the scene,
by harnessing the power of SOTA vision-and-language models with adaptations for
understanding landmark scene semantics. To bolster such models with
fine-grained knowledge, we leverage large-scale Internet data containing images
of similar landmarks along with weakly-related textual information. Our
approach is built upon the premise that images physically grounded in space can
provide a powerful supervision signal for localizing new concepts, whose
semantics may be unlocked from Internet textual metadata with large language
models. We use correspondences between views of scenes to bootstrap spatial
understanding of these semantics, providing guidance for 3D-compatible
segmentation that ultimately lifts to a volumetric scene representation. Our
results show that HaLo-NeRF can accurately localize a variety of semantic
concepts related to architectural landmarks, surpassing the results of other 3D
models as well as strong 2D segmentation baselines. Our project page is at
https://tau-vailab.github.io/HaLo-NeRF/.