OmniFlow: Any-to-Any Generation with Multi-Modal Rectified Flows
Authors: name: Shufan Li; name: Konstantinos Kallidromitis; name: Akash Gokul; name: Zichun Liao; name: Yusuke Kato; name: Kazuki Kozuka; name: Aditya Grover
Published: 2024-12-02 06:13:01

Abstract: We introduce OmniFlow, a novel generative model designed for any-to-any
generation tasks such as text-to-image, text-to-audio, and audio-to-image
synthesis. OmniFlow advances the rectified flow (RF) framework used in
text-to-image models to handle the joint distribution of multiple modalities.
It outperforms previous any-to-any models on a wide range of tasks, such as
text-to-image and text-to-audio synthesis. Our work offers three key
contributions: First, we extend RF to a multi-modal setting and introduce a
novel guidance mechanism, enabling users to flexibly control the alignment
between different modalities in the generated outputs. Second, we propose a
novel architecture that extends the text-to-image MMDiT architecture of Stable
Diffusion 3 and enables audio and text generation. The extended modules can be
efficiently pretrained individually and merged with the vanilla text-to-image
MMDiT for fine-tuning. Lastly, we conduct a comprehensive study on the design
choices of rectified flow transformers for large-scale audio and text
generation, providing valuable insights into optimizing performance across
diverse modalities. The Code will be available at
https://github.com/jacklishufan/OmniFlows.