Hi3D: Pursuing High-Resolution Image-to-3D Generation with Video
  Diffusion Models
Authors: name: Haibo Yang; name: Yang Chen; name: Yingwei Pan; name: Ting Yao; name: Zhineng Chen; name: Chong-Wah Ngo; name: Tao Mei
Published: 2024-09-11 17:58:57

Abstract: Despite having tremendous progress in image-to-3D generation, existing
methods still struggle to produce multi-view consistent images with
high-resolution textures in detail, especially in the paradigm of 2D diffusion
that lacks 3D awareness. In this work, we present High-resolution Image-to-3D
model (Hi3D), a new video diffusion based paradigm that redefines a single
image to multi-view images as 3D-aware sequential image generation (i.e.,
orbital video generation). This methodology delves into the underlying temporal
consistency knowledge in video diffusion model that generalizes well to
geometry consistency across multiple views in 3D generation. Technically, Hi3D
first empowers the pre-trained video diffusion model with 3D-aware prior
(camera pose condition), yielding multi-view images with low-resolution texture
details. A 3D-aware video-to-video refiner is learnt to further scale up the
multi-view images with high-resolution texture details. Such high-resolution
multi-view images are further augmented with novel views through 3D Gaussian
Splatting, which are finally leveraged to obtain high-fidelity meshes via 3D
reconstruction. Extensive experiments on both novel view synthesis and single
view reconstruction demonstrate that our Hi3D manages to produce superior
multi-view consistency images with highly-detailed textures. Source code and
data are available at https://github.com/yanghb22-fdu/Hi3D-Official.