Droplet3D: Commonsense Priors from Videos Facilitate 3D Generation
Authors: name: Xiaochuan Li, name: Guoguang Du, name: Runze Zhang, name: Liang Jin, name: Qi Jia, name: Lihua Lu, name: Zhenhua Guo, name: Yaqian Zhao, name: Haiyang Liu, name: Tianqi Wang, name: Changsheng Li, name: Xiaoli Gong, name: Rengang Li, name: Baoyu Fan

Scaling laws have validated the success and promise of large-data-trained
models in creative generation across text, image, and video domains. However,
this paradigm faces data scarcity in the 3D domain, as there is far less of it
available on the internet compared to the aforementioned modalities.
Fortunately, there exist adequate videos that inherently contain commonsense
priors, offering an alternative supervisory signal to mitigate the
generalization bottleneck caused by limited native 3D data. On the one hand,
videos capturing multiple views of an object or scene provide a spatial
consistency prior for 3D generation. On the other hand, the rich semantic
information contained within the videos enables the generated content to be
more faithful to the text prompts and semantically plausible. This paper
explores how to apply the video modality in 3D asset generation, spanning
datasets to models. We introduce Droplet3D-4M, the first large-scale video
dataset with multi-view level annotations, and train Droplet3D, a generative
model supporting both image and dense text input. Extensive experiments
validate the effectiveness of our approach, demonstrating its ability to
produce spatially consistent and semantically plausible content. Moreover, in
contrast to the prevailing 3D solutions, our approach exhibits the potential
for extension to scene-level applications. This indicates that the commonsense
priors from the videos significantly facilitate 3D creation. We have
open-sourced all resources including the dataset, code, technical framework,
and model weights: https://dropletx.github.io/.