Learning the 3D Fauna of the Web
Authors: name: Zizhang Li; name: Dor Litvak; name: Ruining Li; name: Yunzhi Zhang; name: Tomas Jakab; name: Christian Rupprecht; name: Shangzhe Wu; name: Andrea Vedaldi; name: Jiajun Wu
Published: 2024-01-04 18:32:48

Abstract: Learning 3D models of all animals on the Earth requires massively scaling up
existing solutions. With this ultimate goal in mind, we develop 3D-Fauna, an
approach that learns a pan-category deformable 3D animal model for more than
100 animal species jointly. One crucial bottleneck of modeling animals is the
limited availability of training data, which we overcome by simply learning
from 2D Internet images. We show that prior category-specific attempts fail to
generalize to rare species with limited training images. We address this
challenge by introducing the Semantic Bank of Skinned Models (SBSM), which
automatically discovers a small set of base animal shapes by combining
geometric inductive priors with semantic knowledge implicitly captured by an
off-the-shelf self-supervised feature extractor. To train such a model, we also
contribute a new large-scale dataset of diverse animal species. At inference
time, given a single image of any quadruped animal, our model reconstructs an
articulated 3D mesh in a feed-forward fashion within seconds.