Isotropic3D: Image-to-3D Generation Based on a Single CLIP Embedding
Authors: name: Pengkun Liu; name: Yikai Wang; name: Fuchun Sun; name: Jiafang Li; name: Hang Xiao; name: Hongxiang Xue; name: Xinzhou Wang
Published: 2024-03-15 15:27:58

Abstract: Encouraged by the growing availability of pre-trained 2D diffusion models,
image-to-3D generation by leveraging Score Distillation Sampling (SDS) is
making remarkable progress. Most existing methods combine novel-view lifting
from 2D diffusion models which usually take the reference image as a condition
while applying hard L2 image supervision at the reference view. Yet heavily
adhering to the image is prone to corrupting the inductive knowledge of the 2D
diffusion model leading to flat or distorted 3D generation frequently. In this
work, we reexamine image-to-3D in a novel perspective and present Isotropic3D,
an image-to-3D generation pipeline that takes only an image CLIP embedding as
input. Isotropic3D allows the optimization to be isotropic w.r.t. the azimuth
angle by solely resting on the SDS loss. The core of our framework lies in a
two-stage diffusion model fine-tuning. Firstly, we fine-tune a text-to-3D
diffusion model by substituting its text encoder with an image encoder, by
which the model preliminarily acquires image-to-image capabilities. Secondly,
we perform fine-tuning using our Explicit Multi-view Attention (EMA) which
combines noisy multi-view images with the noise-free reference image as an
explicit condition. CLIP embedding is sent to the diffusion model throughout
the whole process while reference images are discarded once after fine-tuning.
As a result, with a single image CLIP embedding, Isotropic3D is capable of
generating multi-view mutually consistent images and also a 3D model with more
symmetrical and neat content, well-proportioned geometry, rich colored texture,
and less distortion compared with existing image-to-3D methods while still
preserving the similarity to the reference image to a large extent. The project
page is available at https://isotropic3d.github.io/. The code and models are
available at https://github.com/pkunliu/Isotropic3D.