One Diffusion to Generate Them All
Authors: name: Duong H. Le; name: Tuan Pham; name: Sangho Lee; name: Christopher Clark; name: Aniruddha Kembhavi; name: Stephan Mandt; name: Ranjay Krishna; name: Jiasen Lu
Published: 2024-11-25 12:11:05

Abstract: We introduce OneDiffusion, a versatile, large-scale diffusion model that
seamlessly supports bidirectional image synthesis and understanding across
diverse tasks. It enables conditional generation from inputs such as text,
depth, pose, layout, and semantic maps, while also handling tasks like image
deblurring, upscaling, and reverse processes such as depth estimation and
segmentation. Additionally, OneDiffusion allows for multi-view generation,
camera pose estimation, and instant personalization using sequential image
inputs. Our model takes a straightforward yet effective approach by treating
all tasks as frame sequences with varying noise scales during training,
allowing any frame to act as a conditioning image at inference time. Our
unified training framework removes the need for specialized architectures,
supports scalable multi-task training, and adapts smoothly to any resolution,
enhancing both generalization and scalability. Experimental results demonstrate
competitive performance across tasks in both generation and prediction such as
text-to-image, multiview generation, ID preservation, depth estimation and
camera pose estimation despite relatively small training dataset. Our code and
checkpoint are freely available at https://github.com/lehduong/OneDiffusion