ICML 2025

Diffuse Everything: Multimodal Diffusion Models on Arbitrary State Spaces
Kevin Rojas * 1 2 Yuchen Zhu * 1 2 Sichen Zhu 2 Felix X.-F. Ye 3 Molei Tao 1 2
Abstract
Diffusion models have demonstrated remarkable
performance in generating unimodal data across
various tasks, including image, video, and text
generation. On the contrary, the joint generation
of multimodal data through diffusion models is
still in the early stages of exploration. Existing
approaches heavily rely on external preprocess-
ing protocols, such as tokenizers and variational
autoencoders, to harmonize varied data represen-
tations into a unified, unimodal format.
This
process heavily demands the high accuracy of
encoders and decoders, which can be problem-
atic for applications with limited data. To lift
this restriction, we propose a novel framework
for building multimodal diffusion models on arbi-
trary state spaces, enabling native generation of
coupled data across different modalities. By in-
troducing an innovative decoupled noise schedule
for each modality, we enable both unconditional
and modality-conditioned generation within a sin-
gle model simultaneously. We empirically vali-
date our approach for text-image generation and
mixed-type tabular data synthesis, demonstrating
that it achieves competitive performance. Code is
available at Diffuse-Everything.
1. Introduction
Recent years have witnessed the tremendous success of
diffusion generative models in various applications. The
seminal works of continuous diffusion models on Euclidean
spaces (Sohl-Dickstein et al., 2015; Song et al., 2020; Ho
et al., 2020) have led to state-of-the-art methods for tasks
such as image generation (Dhariwal & Nichol, 2021; Bao
et al., 2023a; Karras et al., 2022; 2024b), video generation
*Equal contribution, random order by coin flip 1School of Math-
ematics, Georgia Institute of Technology, Atlanta, GA 2Machine
Learning Center, Georgia Institute of Technology, Atlanta, GA
3Department of Mathematics & Statisti...