Coin3D: Controllable and Interactive 3D Assets Generation with
  Proxy-Guided Conditioning
Authors: name: Wenqi Dong; name: Bangbang Yang; name: Lin Ma; name: Xiao Liu; name: Liyuan Cui; name: Hujun Bao; name: Yuewen Ma; name: Zhaopeng Cui
Published: 2024-05-13 17:56:13

Abstract: As humans, we aspire to create media content that is both freely willed and
readily controlled. Thanks to the prominent development of generative
techniques, we now can easily utilize 2D diffusion methods to synthesize images
controlled by raw sketch or designated human poses, and even progressively
edit/regenerate local regions with masked inpainting. However, similar
workflows in 3D modeling tasks are still unavailable due to the lack of
controllability and efficiency in 3D generation. In this paper, we present a
novel controllable and interactive 3D assets modeling framework, named Coin3D.
Coin3D allows users to control the 3D generation using a coarse geometry proxy
assembled from basic shapes, and introduces an interactive generation workflow
to support seamless local part editing while delivering responsive 3D object
previewing within a few seconds. To this end, we develop several techniques,
including the 3D adapter that applies volumetric coarse shape control to the
diffusion model, proxy-bounded editing strategy for precise part editing,
progressive volume cache to support responsive preview, and volume-SDS to
ensure consistent mesh reconstruction. Extensive experiments of interactive
generation and editing on diverse shape proxies demonstrate that our method
achieves superior controllability and flexibility in the 3D assets generation
task.