Flex3D: Feed-Forward 3D Generation With Flexible Reconstruction Model
  And Input View Curation
Authors: name: Junlin Han; name: Jianyuan Wang; name: Andrea Vedaldi; name: Philip Torr; name: Filippos Kokkinos
Published: 2024-10-01 17:29:43

Abstract: Generating high-quality 3D content from text, single images, or sparse view
images remains a challenging task with broad applications.Existing methods
typically employ multi-view diffusion models to synthesize multi-view images,
followed by a feed-forward process for 3D reconstruction. However, these
approaches are often constrained by a small and fixed number of input views,
limiting their ability to capture diverse viewpoints and, even worse, leading
to suboptimal generation results if the synthesized views are of poor quality.
To address these limitations, we propose Flex3D, a novel two-stage framework
capable of leveraging an arbitrary number of high-quality input views. The
first stage consists of a candidate view generation and curation pipeline. We
employ a fine-tuned multi-view image diffusion model and a video diffusion
model to generate a pool of candidate views, enabling a rich representation of
the target 3D object. Subsequently, a view selection pipeline filters these
views based on quality and consistency, ensuring that only the high-quality and
reliable views are used for reconstruction. In the second stage, the curated
views are fed into a Flexible Reconstruction Model (FlexRM), built upon a
transformer architecture that can effectively process an arbitrary number of
inputs. FlemRM directly outputs 3D Gaussian points leveraging a tri-plane
representation, enabling efficient and detailed 3D generation. Through
extensive exploration of design and training strategies, we optimize FlexRM to
achieve superior performance in both reconstruction and generation tasks. Our
results demonstrate that Flex3D achieves state-of-the-art performance, with a
user study winning rate of over 92% in 3D generation tasks when compared to
several of the latest feed-forward 3D generative models.