HarmonyView: Harmonizing Consistency and Diversity in One-Image-to-3D
Authors: name: Sangmin Woo; name: Byeongjun Park; name: Hyojun Go; name: Jin-Young Kim; name: Changick Kim
Published: 2023-12-26 10:15:28

Abstract: Recent progress in single-image 3D generation highlights the importance of
multi-view coherency, leveraging 3D priors from large-scale diffusion models
pretrained on Internet-scale images. However, the aspect of novel-view
diversity remains underexplored within the research landscape due to the
ambiguity in converting a 2D image into 3D content, where numerous potential
shapes can emerge. Here, we aim to address this research gap by simultaneously
addressing both consistency and diversity. Yet, striking a balance between
these two aspects poses a considerable challenge due to their inherent
trade-offs. This work introduces HarmonyView, a simple yet effective diffusion
sampling technique adept at decomposing two intricate aspects in single-image
3D generation: consistency and diversity. This approach paves the way for a
more nuanced exploration of the two critical dimensions within the sampling
process. Moreover, we propose a new evaluation metric based on CLIP image and
text encoders to comprehensively assess the diversity of the generated views,
which closely aligns with human evaluators' judgments. In experiments,
HarmonyView achieves a harmonious balance, demonstrating a win-win scenario in
both consistency and diversity.