DreamCar: Leveraging Car-specific Prior for in-the-wild 3D Car
  Reconstruction
Authors: name: Xiaobiao Du; name: Haiyang Sun; name: Ming Lu; name: Tianqing Zhu; name: Xin Yu
Published: 2024-07-24 04:13:43

Abstract: Self-driving industries usually employ professional artists to build
exquisite 3D cars. However, it is expensive to craft large-scale digital
assets. Since there are already numerous datasets available that contain a vast
number of images of cars, we focus on reconstructing high-quality 3D car models
from these datasets. However, these datasets only contain one side of cars in
the forward-moving scene. We try to use the existing generative models to
provide more supervision information, but they struggle to generalize well in
cars since they are trained on synthetic datasets not car-specific. In
addition, The reconstructed 3D car texture misaligns due to a large error in
camera pose estimation when dealing with in-the-wild images. These restrictions
make it challenging for previous methods to reconstruct complete 3D cars. To
address these problems, we propose a novel method, named DreamCar, which can
reconstruct high-quality 3D cars given a few images even a single image. To
generalize the generative model, we collect a car dataset, named Car360, with
over 5,600 vehicles. With this dataset, we make the generative model more
robust to cars. We use this generative prior specific to the car to guide its
reconstruction via Score Distillation Sampling. To further complement the
supervision information, we utilize the geometric and appearance symmetry of
cars. Finally, we propose a pose optimization method that rectifies poses to
tackle texture misalignment. Extensive experiments demonstrate that our method
significantly outperforms existing methods in reconstructing high-quality 3D
cars. https://xiaobiaodu.github.io/dreamcar-project/{Our code is
available.}