Diffuse Everything: Multimodal Diffusion Models on Arbitrary State Spaces
Authors: Kevin Rojas
Yuchen Zhu
Sichen Zhu
Felix X-F. Ye
Molei Tao
Venue: ICML 2025 poster (2025)
Keywords: Diffusion Model
Discrete Diffusion
Multimodal
Generative Modeling
Primary area: deep_learning->generative_models_and_autoencoders

Abstract: Diffusion models have demonstrated remarkable performance in generating unimodal data across various tasks, including image, video, and text generation. On the contrary, the joint generation of multimodal data through diffusion models is still in the early stages of exploration. Existing approaches heavily rely on external preprocessing protocols, such as tokenizers and variational autoencoders, to harmonize varied data representations into a unified, unimodal format. This process heavily demands the high accuracy of encoders and decoders, which can be problematic for applications with limited data. To lift this restriction, we propose a novel framework for building multimodal diffusion models on arbitrary state spaces, enabling native generation of coupled data across different modalities. By introducing an innovative decoupled noise schedule for each modality, we enable both unconditional and modality-conditioned generation within a single model simultaneously. We empirically validate our approach for text-image generation and mixed-type tabular data synthesis, demonstrating that it achieves competitive performance. Code is available at https://github.com/KevinRojas1499/Diffuse-Everything.

Diffuse Everything:
Multimodal Diffusion Models on Arbitrary State Spaces
Kevin Rojas 12
Yuchen Zhu 12
Sichen Zhu Felix X.-E. Ye? Molei Tao
Abstract
(Ho et al., 2022; Jin et al., 2025), time series forecasting (Chen et al., 2024b; Rojas et al., 2025b) and in domains such as robotics (Chi et al., 2023) and genomics (Luo et al., 2024; Zhu et al., 2025a). Pioneering works have shown that diffusion models can also be extended to curved spaces (De Bortoli et al., 2022: Huang et al., 2022: Cheng et al., 2025; Chen & Lipman, 2024; Zhu al., 2025b), enabling high-fidelity generation of structured data on manifolds, such material configurations (Sriram al., 2024) and protein backbones (Watson et al., 2023; Yim et al., 2023), Recently, discrete diffusion models have emerged as the cornerstone for modeling categorical data with inherent discrete structures (Campbell et al., 2022; Lou et al., 2024; Campbell et al., 2024; Gat et al., 2024). Discrete diffusion models have imposed great impacts on protein sciences (Wang al., 2024b), graph generation (Xu et al., 2024; Li et al., 2025), and text generation (Nie et al., 2025a;b). In general, diffusion models have shown top performance in most scenarios with unimodal data.
Diffusion models have demonstrated remarkable performance in generating unimodal data across various tasks, including image, video, and text generation. On the contrary, the joint generation of multimodal data through diffusion models is still in the early stages of exploration. Existing approaches heavily rely on external preprocess- ing protocols, such as tokenizers and variational autoencoders, to harmonize varied data represen- tations into a unified, unimodal format. This process heavily demands the high accuracy of encoders and decoders, which be problem- atic for applications with limited data. To lift this restriction, we propose novel framework for building multimodal diffusion models on arbi- trary state spaces, enabling native generation of coupled data across different modalities. By in- troducing an innovative decoupled noise schedule for each modality, we enable both unconditional modality-conditioned and modality conditioned generation within sin- gle model simultaneously. We empirically vali- date our approach for text-image generation and mixed-type tabular data synthesis, demonstrating that it achieves competitive performance. Code is available at Diffuse-E Everussin......ng...
Generative models also demonstrated successes in multi- modal data lately. For example, conditional diffusion mod- els showed remarkable capabilities in tasks such as text-to- image generation by accurately synthesizing pictures follow- ing given instruction prompts (Ramesh et al., 2022; Chen et al., 2023; Esser et al., 2024). It's worth noting that such models still generate single-modality outputs (such as im- ages). Therefore, to jointly generate multimodal data, lever- aging only
tar-tassk-r-riiinggg conditional models is extremely computationally inefficient, as it requires combin- ing multiple independently trained models by sequentially applying them.
1. Introduction
Recent years have witnessed the tremendous success of diffusion generative models in various applications. The seminal works of continuous diffusion models on Euclidean spaces (Sohl-Dickstein et al., 2015; Song et al., 2020; Ho et al., 2020) have led to state-of-the-art methods for tasks such as image generation (Dhariwal & Nichol, 2021; Bao ct al., 2023a; Karras et al., 2022; 2024b), video generation
An alternative approach is to use a single multi- modal model that captures the joint distribution of multiple modalities. Such an approach often leads to strong performances as it al- lows information to mix across modalities (Li al., 2024a; Meta, 2024). Existing approaches of this type are mainly based on autoregressive models (AR), such Chameleon (Meta, 2024) and Unified-1O (Lu et al., 2024), where data of different modalities are represented uniformly as tokens and generated autoregressively from left to right. Apart from these, attempts have also been made to realize this idea using diffusion/flow-base methods, such as UniDif- fuser (Bao et al.. 2023b). MM-Diffusion (Ruan et al., 2023). AVDIT (Kim al., 2024), UniDisc (Swerdlow et al., 2025).
Equual contribution, random order by coin flip 'School of Math- ematics, Georgia Institute of Technology, Atlanta, GA *Machine Learning Center, Georgia Institute of Technology, Atlanta, GA 'Department of Mathematics Statisties, SUNY Albany, NY. Correspondence to: Molci Tao <mt amtao@gttchh. edu>.
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Multimodal Diffusion on Arbitrary State Spaces
Uneardionn Genevaton
POWNAIVI
poort and
PONER MI
Text Cordhtand aalrage
Text Lime Anis
The byy
KI Tlicks
AWA-II Blook
Fros Cemerntton
DIT Bioi
DIT Biocks
DV Blucks
SUT Bloces
DIT oocks
ocia
HAUR
Inage Consiloonna on Text
60 ie Goneration
Osimmage Condisoned Ferweeedd
Text Canditioned on Image Generaatoo
Image Axis
Figure 1. By injecting noise into different modalities in a decoupled fashion, we enable the unconditional and modality conditioned generation in single model. (a) Joint generation of image and text. (b) Image generation given text captions as conditions. (c) Text generation given images as conditions.
OmniFlow (Li et al., 2024a), etc. These methods generate multimodal data simultancously through iterative denoising of a randomly sampled initial noise.
Diffusion models serve as a powerful backbone for build- ing such a framework. Theorists have shown that diffusion models can be extended to a more general idea called de- noising Markov models (Benton et al., 2024; Ren al., 2025b), providing solid theoretical foundation for multi- modal extension. In addition to this, existing works such as MultiFlow (Campbell et al., 2024) and Generator Matching (Holderrieth et al., 2024) have verified the effectiveness of multimodal models in native state spaces in protein de- sign. Motivated by these successes, we propose a general framework for building multimodal diffusion models on ar- bitrary state spaces without the need for data format unifiers. Our contributions are three-fold:
A commonality among the aforementioned methods de- signed for joint multimodal generation is that they typieally rely heavily preprocessing techniques to harmonize the varied data representations into a unified format, thereby creating a single modality. One approach (taken by Chameleon (Meta, 2024), UniDisc (Swerdlow et al., 2025), etc) to cast multimodal inputs all into discrete to- tokenizers built with discrete
kens with modality or vector-quantized variational encoders (VQVAE) (Van Den Oord et al., 2017; Esser et al., 2021). An alternative route (considered by UniDiffuser (Bao et al., 2023b), Omni- Flow (Li et al., 2024a), etc) is to preprocess the multimodal data by embedding them into continuous-valiue latent vec- tors with encoders trained with variational encoders (VAE) (Kingma, 2013) or representation alignment (e.g.. CLIP (Radford et al., 2021)).
1.
We propose novel framework for building multimodal diffusion models by combining the native diffusion models designed for each data modality, and derive a unified learning objective. Under our design, learning multimodal diffusion models is as straightforward as performing joint optimization on sum of unimodal learning losses, despite requiring non-trivial proof.
For these approaches, regardless of whether discrete tokens or continuous latents are used, generation is performed in a unimodal space, and the original data modality must be recovered through decoding. Therefore, these pipelines may suffer from generation artifacts due to the limited accuracy of the decoders (Hoogeboom et al., 2024). Additionally, the requirement for tighppeeform encoder-decoder pairs can be problematic to satisfy for applications that lack abun- dant high-quality data (Zhang et al., 2023). Due to the requirement for task-specific algorithm designs, these meth- ods also cannot be conveniently extended to generate data composed of arbitrary modalities. Therefore, a natural ques- tion ask is the following:
2. We introduce decoupled noise schedules for each data modality and theoretically justify the validity of score learning under the presence of multiple time variables. We demonstrate that this design enables us to simul- taneously handle both unconditional and conditional multimodal generation in one single model. We also propose a novel guidance mechanism effective in both use cases for enhancing generation quality.
3.
We experiment with text-image generation and mixed- type tabular data synthesis, achieving competitive per- formance on both tasks with more parameter-efficient models, without relying on pre-trained models or pow- erful extra encoders. More importantly, we devise set of training strategies for the task of text-image, which is crucial for achieving success.
Can we design a principled framework to enable joint modeling of multi modall data in their native spaces without a unified representation?
2

---

Multimodal Diffusion on Arbitrary State Spaces
Fysp
X Kinege
Faod
SLU
Layerhiam
Layotform
Snaackdd Evooding
Pasch Emboldding
Table Enbeddling
Sinupical Encoding
A
6,y+
Layerhirm
MP
MLP
MLF
MM OIT Block
MMD.
2lock BM
MM-UI Blockd
Attertion
jrear
Liney
Layerhaom
Dif Block
DVT Block
Pu
Fy
Laverhion
LayerHor
By
4 4
FFN
by
MiF
MP
Uppaish-y
Uiser
6,442
ba
Fueee
MI- Bock
Block
Figure 2. Network backbone for text-image generation, motivated by MMDiT (Esser et al., 2024) and DiT (Peebles & Xie, 2023).
2. Preliminaries
2024), one considers a continuous time markov chain (CTMC) (Xt)oses? on a finite state space X as the for- ward process. The distribution of Xx represented by vector pa in the probability simplex R The dynamic of X2 can be characterized by the following equation,
In this section, we review basic concepts and formulations of common diffusion models in different state spaces.
2.1. Continuous Space Diffusion Models
For continuous diffusion models (Song et al., 2020; Ho et al., 2020), one considers a continuous time stochastic differen- tial equation (SDE) {XehosesT on the Euclidean space Idd as the forward process. The process is characterized by the following dynamic.
dpr dt
QrPts where Q, (Qr(2, v))x.eex
is a transition matrix satisfying that for any x E X, Qr x) = Eyzz x)), and for any x # y E X, Qr y) 2 We will also denote the dynamic of X₃ using the following notation,
dx,- = ((xx,t)dt + (()dW,
where f: R x R + Rd is the drift and g R + R is the diffusion coefficient. We denote pt = Law(X Law(X.). f and g are often chosen that pr is an sasple distribution. A popular pick is the time re-parametrized Ornstein Uhlenbeck process, which corresponds to the selection of f(X,, I) -,, and g() vBr, for some positive noise schedule 3r In such case, pr Rs (O, I) for reasonably large T. It can be shown that the backward process is another SDE with a different drift ( (Anderson, 1982),
X, ~ CTMC(Q.)
Q, is often chosen such that pr is simple distribution, such as uniform on X Dirac on a masked state. Common choices include uniform or masked transition matrix (Lou et al., 2024). It is known that the backward process is an- other process of the same form but with a different transition rate matrix (Kelly, 201 I), which can be described as
dX2+= = fXe, d) (t) log + g())dW,
X, CTMC(Q.)
The common practice in training is to define the score vector s(Xe,t) t) = V z log (X,) and we approximate it with a neural network ss, estimated by minimizing a variant of the following score matching loss (Vincent, 2011),
where the rate matrix Q. (2, y) y))x is defined as,
pT- Q 4), x #tyE X, PT (5)
Q,(,,z)=
min /' Eximp [Xs,(Xe,,)) ((:0))ss(00 s(X,, D)] 8
['Ex/=p
E Q x),
x= z=yc- e X.
(1)
In discrete diffusion model training, one usually defines the concrete score vector s(X,, t) = approximate it with a neural network sø (X,, t), estimated by minimizing the following a variant of the following score
2.2. Discrete Space Diffusion Models
p. (x,) yeX* and we
(vv)
For discrete diffusion models (Campbell et al., 2022; Lou et al., 2024; Ou et al., 2025; Sahoo al., 2024; Shi al.,
3

---

Multimodal Diffusion on Arbitrary State Spaces
entropy loss (Benton et al., 2024; Lou et al., 2024),
they are essentially the realization of denoising Markov modely (Benton al., 2024; Ren al., 2025b) in different situations. At a high level, diffusion models consist of a Markovian forward process that gradually injects 'noise' to transform the target data distribution into a simple distri- bution, and a backward generative process that inverts using information learned from the marginals of the forward process. This abstract view is formally summarized below.
min 0
E XI-P
Σ N
Qi(X,, ()(X).v) (s: (x by log (x())))
- -x(,,, O2 + so Xx: (0) dt.
(2)
2.3. Riemannian Diffusion Models
Consider Markov process {Xe]usesT with Xo Pdata- defined on a state space X. A Markov process can be con- veniently characterized using the notion of infinitesimal generators. Since X, is not necessarily hime-homogeneous, we instead consider the augmented process X = (X,, t) defined on the augmented space X, X' x [0, +00). Under mild regularity assumptions, X is Feller process, and its generator L can be defined as,
For Riemannian diffusion models, e.g. (De Bortoli et al., 2022; Huang et al., 2022), one considers a continuous time SDE {X2) JosesT on the manifold M as the forward process, characterized by the following dynamic,
dx;- f(X,, t)dt dW:..
where f M x R + T&M is the drift, and dw, is the manifold Brownian motion. For compact manifold M, one can pick/ f = O, and the corresponding Pr x (M) is the uniform distribution on the manifold for large T. It's proved that the backward process is another SDE on the manifold (De Bortoli et al., 2022).
( Xo = 2| f(x) t-
Cf(x)
lim
and f X. RR. is class of test function. We under- stand the generator through its decomposition L 0 + L, where L is an operator that acts on functions defined on the original space X. We can also unify the characterization of the evolution of marginals p(., t) = Law(X,) as well as the score learning objectives in terms of L. Under weak technical assumptions, p(., t) satisfies the following general form of Fokker-Plande equation,
dX, sX,, t)dt V log (X,)dd aw:M,
where V is the Riemannian gradient on M. For Rieman- nian diffusion model training. similar to the continuous case, one defines the score 8 (X,, () = V log pr (X,) and approxi- mates it with a neural network 8s, which typically requires special design to meet the requirement so (X,, 8) E M. Learning is performed through a variant of Riemannian score matching (De Bortoli et al., 2022; Huang et al., 2022),
0.p(æ, t) = p(z, t), p(x, 0) = Pdutta (x)).
where L is the adjoint operator of L. Moreover, we can define generalized explicit score matching objective (Lyu, 2012; Benton et al., 2024),
min /'" E Ex (x,,)(X.) sXx,, (xx,11) 111 dv. (3)
0
3. Methodology
JESM (3)
E t.
S
P
(Xc:t)
(4)
In this section, present the framework for constructing multimodal diffusion models on general state spaces in their native forms. We first discuss a unified perspective on dif- fusion models and then present learning algorithm for diffusion generative modeling of multiple data modalities, where each modality has an independent time variable. Such a framework allows any-to-any modality generation by one single model, which simultaneously includes the joint unconditional generation of all modalities and the condi- tional generation of subset of modalities given the rest. Finally, we discuss the Continuous- Discrret Multimodal Diffusion as an application of this framework to illustrate its importance and flexibility.
- log f. and 3 X' [0, +00) R+ Note that this is an extension of the common score matching objectives, and we could interpret JESM as a loss function that compares the 'gradient log' of p(r, t) to that of 3(x, 1), through which we learn the information of the forward marginal Pt. For example, in continuous diffusion with (() 1, L 0 f V + A, we recover the explicit score matching objective & Dayan, 2005), with (pp/3) 1 || log V log $11 Note that the explicit score matching objectives are not tractable for training purposes; we will later introduce their equivalent, trainable variants.
where
Euclidean space (Hyvärinen
3.1. Unified Perspective on Unimodal Diffusion Models
3.2. Versatile Multimodal Diffusion Models with Decoupled Times
While common unimodal diffusion models (continuous, dis- crete, Riemannian, etc) have distinct forward/backward pro- cesses, learning objectives, and score parameterizations,
Building on this unified description of unimodal diffusion models, we extend the denoising Markov model framework

---

Multimodal Diffusion on Arbitrary State Spaces
(a) Image conditioned on text. Sample generated using the caption: "The image features a sailboat sailing on large body of water, with a city skyline in the back- ground."
(b) Text conditioned on Image. We gen- erate the following caption: "The image features cityscape with a large build- ing, a bridge, and a city skyline in the background. The city is situated near the water.
(c) Joint generation of text and image. The caption corresponding this image was: "The image features large white boat with a blue roof, which allows peo- ple to look out. The boot is traveling through a body of water."
Figure Visualization of samples generated by our approach. Captions are truncated for brevity.
to a multimodal scenario. This enables us to perform genera- tive modeling of data distributions consisting of mixed-type data without requiring complicated preprocessing pipelines.
process, similar to the case of unimodal diffusion models where one performs score matching. Extending the frame- work of (Benton et al., 2024), we introduce the following generalized explicit score matching loss (GESM) for learn- ing the full marginal tt.,
We begin by formally defining the forward process in terms of generators of Markov processes. Assume that we have a data distribution Paata defined on the product state spaces X² x ... x X². For < i S. n. we have a Markov process (Xx } isseT on X', which is regular enough with a unique, easy-to-sample stationary distribution x². We pick T so that Law( (Xt)) = nt for each < n. Now can introduce the following joint forward process,
IGESM E (.
Cx: (p/) Lx lo)(p/ 1) (p/Ba)
here 8g : X x x X R is our parameterized unnormalized distribution. We have the following important properties that characterize the optimizer of IOESM
X₂- = (X), _ x7,), 0 5
ST
(X) 0 X/. X() (x)
(5)
Theorem 1. IGESM 2 O. with equality reached when Bo (ar, t) p(æ, t).
where t = (t) (t"....,,t)) t²), x = (x',...,,z) ).). We consider the i-th augmented process X²= (Xi,, (i) E X = X x [0, +oo) and we denote its generator as Lxt. We slightly abuse the notation and define the application of fx to a multivariable test function as the following,
Thm. states that the minimizer of IGESM is p(æ, t) up to multiplicative constant. In practice, we often do not di- rectly model Sg. but instead parameterize its score functions, which are invariant to multiplicative constants. For example, in continuous diffusion, one often parameterizes V log 3o. This makes the optimization of IGESM a well-defined prob- lem with a unique minimizer in terms of score learning. In practice, one can't 't evaluate IGESM as it's intractable due to the true marginals p being unavailable priori. Luckily, one can efficiently compute the following denoising and implicit variants of IGESM for learning purposes.
Cx: = lim E|/(x X +0
|X = f(x)
Ho..
1
We assume that X are independent Markov processes when conditioned on initial conditions. We want to em- phasize that the design of this forward process (5) is not only for injecting probabilistically independent 'noises' into each modality. More importantly, it allows each modality to be noised in an asynchronous way.
Theorem 2. IGESM IGDSM and IGISM are equivalent up to constants, where
To fully characterize and understand this forward process as a whole, we need to learn the full joint marginal of X t. which we denote p( xx, and should be understood the joint distribution of X X ...., X" at time (2, ....,., 1". To visualize this idea, we demonstrate the forward process with two independent time variables in Fig. 1.
LODSM
Cx Cx Bo) (ce,t) Lxi 30) 1) (Pejo/ (10
E t,P0- Pejo
C Cx130(0r:t) tx(
To 'invert' this forward process for generative modeling pur- poses, we will need to learn information from the forward
IGISM
E t.pe
L3 loee)
5

---

Multimodal Diffusion on Arbitrary State Spaces
3.3. Continuous-Disrrete Multimodal Diffusion
Zoom-ih
36.00
We consider distribution Pdata(;; 3) where IR y E X. where X is finite space. Important applications include text-image joint generation, mixed-type tabular data synthe- sis, etc. In this case, the natural choices on each state space would be continuous diffusion on Rd and discrete diffusion on X. This results in the following forward process,
32.50
29.00
16
25.50
22.00
18.50
15.00
0.2
0.3
0.6 Noise Level
0.5
0.8
dxx s(x,; Ddl + g(O)BB
Yg ~ CT) (Xo. Yo) ~ Pdmal Pdma(x, (I,Y)
Figure 4. Performance of noisy guidance on MS-COCO FID-10K. We note that using partially noised conditions results a better performance. A guidance interval of t [0.3, 0.8] was used. this enables us to design versatile conditional and uncondi- tional generative sampling algorithms by choosing different value combinations of times (t. s). We defer the detailed discussion of design choices and sampling algorithms for continuous-disrret multimodal diffusion to App. B.
Let's denote the joint marginal of (X,, Y2) as Pt, (X,, Y.). with these choices of Markov processes, Lx = ON + V 1 ( and Ly = 0, + Q.. Therefore, we can compute the generalized denoising score matching objective as, Proposition 1. For forward process (6), TGDSM is equiva- lent to the following objective,
E 5 (1))10 V [logp((21/000)1++ 2
Multimodal unconditional generation. To jointly gener- ale clean data distributed as Pdatu (z. y), we introduce a sim- ulation time variable [0, 21. and pick time parameteri- zation ( a (u), s - or(()) such as o; : [0, - [0, is continuous, non-decreasing with (0))) 0, a (T) T for 1.2. With this renerrrraaameerizationn for both time variables, we make the backward process a valid, ready-to- simulate process. This enables us to start from pure noise and generate samples from the data distribution uncondi- tionally for both modalities. We can also choose a singular time parammtterizaioon so that the simulation amounts 2- 2-stage approach for multimodal generation, where first generate one modality and sample the rest conditionally based on the generated sample.
Biilla P.
(ol3o) P. (v.lu)))
Lo 8) ((EE),
log(s))
uFu.
where 5i. .o is the learned continuous/discrete score.
Importantly, Prop. 1 shows an amazing result that score functions of multimodal joint marginal p(.,,,, st can be learned through score matching with unimodal condi- tional score for each modality. This is a non-trivial result as IGDSM suggests that, to perform multimodal generation, we need to match the score network with conditional scores of the joint distribution such as Pt, ys (To yo) in- stead of V log Pa (1r However, thanks to Bayes' rule and the design of independent noise injection per modal- ity, the two conditional scores are identical. Thus, while learning multimodal diffusion models may seem as simple as jointly optimizing a sum of unimodal diffusion model training objectives, the theoretical support for such a naive approach is grounded much deeper.
Unimodal conditional generation. This framework with decoupled time also enables conditional generation of modalties by simulating the associated backward process. We have the following simple but important observations,
Given partially noisy text Y, and its noise level sim- ulating the X backward dynamics generates a sample XT ~ Paata (
With the learned score functions, we can 'invert' the forward process for generative purposes, which is stated in Prop. 2. Proposition 2. The following process (X,,Y, has marginal distribution equals to p(z, y, - T,T s).
Given a partially noisy image X, and its noise level t. simulating the Y backward dynamics generates a sample Yr ~ Pdata (y)
rax,= = = [X,, T 1) (T - Vog v Y.)dt + t)dB,
Note that the choices of L or s in the conditioning are not restricted. When picking or as T, this equivalent to single-modality unconditional generation, as Xr or Yr are pure noise. When picking t or O, this is equivalent to conditional generation, as Xo or Yn are clean data sam- ples. More interestingly, when picking 0 < s,t < T as conditions, we generate samples based on partially noised conditions. This gives rise to the following new guidance mechanism for enhancing generation quality.
Y. ~ CTMC (X),,,))) (Xo, Yo) ~ p(z, y, T.T)
where Fi.- = -P-- 5 ()), is rate matrix with y', y entry being (x.0( (02'22QQQQ- (Qr=s) when s/ + y.
(X..))
We note that this 'backward process' is not the time reversal of the forward process in a strict sense, as it involves intro- ducing multiple time variables into the system. However,
6

---

Multimodal Diffusion on Arbitrary State Spaces
Table Results on the text to image conditional generation on MS-COCO. We mark the extra encoders leveraged by each model with the corresponding sizes and types. SR: super resolution, TE: text encoder, VAE: variational autoencoder, VE: visual encoder, VQ-GAN:
Vector Quantized GAN, VQ-VAE: vector-quantized variational autoencoder.
Model
FID
Number of Images
#Params
Extra Encoders
Models for Text-to imae generation only
DALL-EE 2 (Ramesh et al., 2022)
10.39
650M
6.5B
123M (TE) + 700M (SR)
Imagen (Saharia et al., 2022)
7.27
860M
3B
4,6B (TE) + 600M (SR)
Stable Diffusion (Rombach et al., 2022)
12.63
400M
1,45B
123M (TE) 83M (VAE)
Pix-rt-- XI.2 (Chen al., 2023)
7.32
25M
600M
123M (TE) 83M (VAE)
MMDDT-improved (Ifriqi et al., 20124)
6.79
12M
600M
123M (TE) 83M (VAF)
Models for multimodal generation and understanding
Show-o (Xie al., 2024)
9.24
35M
1.3B
115M (VE) 307M (VQ-VAE) 86M (VAE)
Transfusion (Zhou et al., 2024)
6.78
692M
Chameleon (Mela, 2024)
26.7
600M
7B
307M (VQ-GAN)
JetFormer (Tschannen et al., 2024)
20.86
1B
2.75B
Models for multimodal generation only
Versatile Diffusion (Xu et al., 2023)
11.10
400M
1.45B
123M (TE) + 83M (VAE) + 110M (TE)
UniD3 (Hu et al., 2022)
25.11
592K
600M
123M (TE) + 307M (VQ-GAN)
Our model
16.16
12M
481M
83M (VAE)
3.4. Noisy Guidance
= 0 and σ = T. The scenario of conditional generation corresponds to $ = O (a clean condition yo is given), and we choose T > σ > to improve generation quality with a partially conditioned guiding model. More interestingly, noisy guidance can even be applied to unconditional gen- eration in unsupervised way where >> 0. In this case, while is changing (since we are also generating Vs). we can still apply guidance in this process by adaptively picking a as long as a > 8. These observations indicate the power and robustness of noisy guidance as a by-product of our proposed multimodal diffusion model learning framework.
Guidance techniques have been a core component in mod- ern diffusion models for improving generation quality (e.g., Dhariwal & Nichol, 2021; Ho & Salimans, 2021; Kynkäänniemi et al., 2024; Li et al., 2024b). For contin- uous diffusion models, with strength w, the classifier-free guidance is obtained by interpolating the unconditional and conditional score functions,
wsp(xr, + (1- (Do ()
One perspective to understand the effectiveness of guidance methods is to view the unconditional score function as a conditional model with a fully-noised condition input, and the interpolation effectively serves as a correction of the conditional scores. However, the unconditional score might not be the best choice of correctors, as it causes an excessive trade-off between fidelity and diversity, resulting in a signif- icant loss in the latter (Karras et al., 2024a). This raises an interesting question about whether CFG can be improved by finding a better alternative the unconditional score in (7). In fact, within our framework, notice that
To showcase the performance of noisy guidance, evalu- ate FOK on MS- MS-COCO (Lin et al., 2014) using different values of a E [0, 1]. We find that partially denoising the caption results in an improved FID, as shown in Fig. 4. The superior performance of noisy guidance is possibly due to us- ing a partially corrupted version of the conditional model as the guiding model. This aligns closely with the idea of Autoguidance proposed in Karras al. (2024a), which uti- lizes an under-trained, smaller model as the guiding model instead of the unconditional ones. Similar to Autoguidance, noisy guidance seeks to identify and reduce the errors made by the conditional score model by measuring its difference to the partially conditioned one, boosting the generation performance. Finally, we remark that (8) only a special case applied continuous diffusion, and a similar idea can be adapted to other modalities, such as discrete diffusion guidance (Nisonoff et al., 2024; Schiff et al., 2024).
b8 (Fr, Dss 1, 8) R Vz log Pt.s (2+, ys) V log p(x,, flys. 8)
The last equality results from Bayes' theorem, and shows that our model in fact learns conditional scores at all noise levels. Leveraging this fact, we propose a new form of guidance named noisy guidance, where the unconditional score in (7) is replaced with class of conditional models with conditions noised to different levels:
4. Experiments
WSp (Zt, ye f, s) (1 (Zr, No. Vo.t, o)
o 0>5
(8)
To demonstrate the effectiveness and relevance of our frame- work for training multimodal diffusion models, we con- sider two tasks: text-image generation and mixed-type tab-
We note that the noisy guidance framework is a more gen- eral one, as it recovers the vanilla setting of CPG when
7

---

Multimodal Diffusion on Arbitrary State Spaces
Table Performance on the Trend metric percentage (%). Higher values indicate better performance. Best performance in bold.
Second best in underline.
Methods
#Parameters
Adult
Default
Shoppers
Magic
Beijing
News
GOGGLE (Liu et al., 2023)
~ 5.6M
54.71
78.06
76.10
90,53
54.06
76.81
STaSy (Kim et al., 2022)
10.3M
85.49
14.0
91.51=0015
93.39=0053
92.00.00.10
96.93
CoDi (Lee et al., 2023)
25.0M
77.51 20.08
31.59
82.22=0.11
93.47=025
92.9320.15
88.90=00
TabDDPM (Kotelnikov et al., 2023)
~ 11.7M
96.99.0225
95.1 11 =0.10
93.39=0.16
98.30
97. 97.20=000
86,84
TABSYN (Zhang et al., 2023)
10.7M
98.46220.7
97.95
97.93=0.1
98.94-031
97.76=0028
98.56=00m
TABSYN (reproduced)
10.7M
98. 20.2
95.25 #0.51
97.82=0.14
99.16-2016
94.86=0054
98.52:10.0
Our model
64K
98.75+40.9
96.00+1.23
98.24=0.13
98.85=0.42
97.42=au.1
98.57=#016
Table 3. Performance on the MLE metric. Higher values in AUC and lower values in RMSE indicate better testing performance. Best performance in bold. Second best in underline.
Methods
PParameters
Adult (AUC1)
Default (AUC1)
Shoppers (AUC1)
Magic (AUC1)
Beijing (RMSEL)
News (RMSEL)
GOGGLE (Liu al., 2023)
~ 5.6M
.77810012
.0.000000
658.10052
.65420004
1.090
8077:0000
STaSy (Kim et al., 2022)
10.3M
906:10.001
..52200006
91410.00
.00
.65620014
.871
CoDi (Lee et al., 2023)
~ 25.0M
87 40.00
.525=0.00
86.
.332#0000
.81 818==0001
1.21=0005
TabDDPM (Kotelnikov et al., 2023)
11.7M
+07
.758
918+0mm
.335.0000
.592=0011
4.86#304
TABSYN (Zhang al., 2023)
10.7M
915 =0002
.764-0.004
.20
.38=+0002
.582+0008
-861=0077
TABSYN (reproduced)
10.7M
910+amm
.755
9164tmm
.939
.655=0012
-851=0024
Our model
64K
.15100001
.764.0.002
12..00.03
941
.54310012
864410001
ular data synthesis, both of which are accomplished using the continuous- discrete Multimodal diffusion discussed in Sec. 3.3. The results for text-image generation presented in Sec. 4.1 and the results for tabular data synthesis are presented in Sec. 4.2. e also include results for combining Riemannian and discrete diffusion in App. E to demonstrate the generality of the framework.
Training& & Evaluation We train our model using a multi- stage training strategy. We kindly refer readers to App. for more details. We evaluate FID-30K on MS-COCO (Lin et al., 2014). Compared with SAM-LLaVA, MS-COCO comes with much shorter captions. To address this distribu- tion shift in caption length between training and inference time, we draw inspiration from (Ifriqi et al., 2024) and repli- cate the text to increase the caption size. We also limit the number of tokens to 40 during this evaluation. We compare our results other methods in Tab. 1. In terms of text- to-image (T2I), our methods produce similar performance compared to many other industrial-level models with larger model sizes. Notably, our model is trained on fewer sam- ples, features a significantly smaller backbone, and does not utilize extra foundation models to aid multimodal rep- resentation learning. This reflects both the efficiency and the effectiveness of our proposed approach. We also present qualitative samples in Fig. 3. For evaluation of image-to-tert (I2T) and joint generation, please kindly see App. F. 4.2. Mixed-type Tabular Data Synthesis
4.1. Text-Image Generation
Architecture We design a new score network backbone for this task based on the celebrated success of Diffusion Transformer (DiT) (Peebles & Xie, 2023) and Multimodal Diffusion Transformer (MMDiT) (Esser et al., 2024). We first process the inputted (noisy) images and texts by passing them through an MMDiT with per-modality unique time conditioning. MMDIT's remarkable strength in modeling cross-modal interaction, as well as allowing independent conditioning for each modality, makes ideal for the back- bone. The tokens then undergo unimodal DiTs for a more refined learning process. We present a comprehensive di- agram of the backbone in Fig. 2 and refer the reader to App. C for further details.
Architecture We devise score network based on DiT for this task, where both discrete and continuous tabular data are fod into the transformer after simple dimension rescaling. Our design aims at achieving early fusion of both modalities for more efficient learning. See App. D.3 for details.
Datasets We train on the SAM-LLaVA dataset introduced by Chen et al. (2023). This dataset is constructed by adding captions to the Segment Anything (SAM) (Kirillov et al., 2023) using LLaVA (Liu et al., 2024), which results in rich and diverse captions. However, it suffers from hallucinations of LLava. For example, many colored images are described as being black and white. Following (Chen et al., 2023), we tokenize each caption with length of 120 tokens.
Datasets We experiment on 6 real-world tabular datasets acquired from UCI Machine Learning Repository'. Ev- ery dataset contains columns of numerical or categorical features, associated with binary classification tasks or regres-
'httss//archiie.ccc....eee
8

---

Multimodal Diffusion on Arbitrary State Spaces
sion tasks. Detailed descriptions of datasets are in App. D.2.
has in fact been explored by many application-driven works in the literature, such as UniDiffuser (Bao et al., 2023a), MultiFlow (Campbell et al., 2024), AVDiT (Kim et al., 2024), and OmniFlow (Li et al., 2024a). While these meth- ods all leverage the multiple time variable design to achieve any-to-any generation, they mostly consider this design as a trick and do not investigate it from an algorithmic perspec- tive. Our work contributes to this literature by deriving the unified training objective and backward generative process in the presence of multiple time variables, providing theoret- ical justification for the validity of this design. To the best of our knowledge, our work is the first to formalize this idea and generalize it to an arbitrary number of modalities.
Training& & Evaluation We describe the score model archi- tecture and training settings in App. D.3. For evaluation, we follow the same setting as Zhang et al. (2023). We evaluate the Trend, Machine Learning Efficiency (MLE), Shape, Pre- cision, and Recall the generated data. We present results on Trend in Tab. 2 and MLE in Tab. 3. For more results on other metries, please see App. D.4. All experiments are repeated 20 times for robustness.
As evident from Tab. 2 and Tab. 3, our model performs either the best or second-best, with a negligible gap, among most datasets. Notably, our model has only ~ 64K parameters, which is 100 to 200 times smaller than models used in other methods. The significant reduction in model size stems from the fact that our method operates natively on the state space of mixed-type tabular data, climinating parameter-hene heavy encoders like VAF. Our newly designed transformer-based score network leverages the concept of early fusion (Meta, 2024), which also enhances parameter efficiency. This net- work learns a joint embedding between modalities starting from the first attention layer and is more parameter-efiie efffiieeet.... These results again showcase the efficiency and advantage of our proposed multimodal diffusion model framework.
A highly relevant work worth discussing in more detail UniDiffuser (Bao al., 2023b), which addresses the same text- exxt-image joint generation task considered in this paper while also utilizing decoupled time variables. A fundamen- tal difference in algorithm design is that UniDiffuser purely relies on continuous diffusion models in the CLIP latent space, which is shared by both text and images, whereas our proposed continuous disorete diffusion operates natively on the product space of Euclidean and finite-state spaces.
Diffusion Models in General State Spaces. Theoretical results have demonstrated that diffusion models can be gen- eralized to denoising Markov models, a class of generative models constructed based on the notion of Markov processes (Benton et al., 2024; Ren et al., 2025b). These works con- sidered unimodal diffusions on general state spaces with a single time variable. Our work extends the framework of de- noising Markov models by incorporating multimodal diffu- sion models the product of different state spaces, as well as multiple time variables. In the literature, there are also MultiFlow (Campbell et al., 2024) and Generator Matching (Holderrieth et al., 2024), which are multimodal extensions of flow-based methods in native state spaces. Their algo- rithm construction specifically focused on the task of protein prpuence-structurg co-generation. In contrast, we present general recipe for multimodal diffusion models that does not initially target specific tasks, despite empirically validat- ing the proposed framework on two examples: text-image generation and mix-type tabular data synthesis, leveraging our newly proposed Continuous-Discret Diffusion.
5. More Information on Related Works
Multimodal Generative Models. Various works in the lit- erature have attempted joint generation of multimodal data. Many existing methods approach the task by leveraging autoregressive models by first tokenizing multimodal data into discrete tokens and then generating them sequentially (Meta, 2024; Xie et al., 2024; Zhou et al., 2024; Tschannen et al., 2024; Lu et al., 2024; Ge et al., 2024; Wu et al., 2024; Wang et al., 2024a). Another portion of the algorithms is built on the versatile capability of iiffusiion/flowbasee methods to generate latent representations of multimodal data (Lee et al., 2023; Bao et al., 2023b; Ruan et al., 2023; Hu et al., 2022; Zhang et al., 2023; Kim et al., 2024; Chen et al., 2024a; L.i et al., 2024a; Swerdlow et al., 2025; Hayes et al., 2025; Bai et al., 2025; Shi et al., 2025). One thing in common among the aforementioned approaches is that they all extensively utilize tokenizers or encoders to produce a unimodal latent space for multimodal data, which are not modular and heavily tailored to specific applications with little theoretical support. In contrast to these works in the literature, we propose a general multimodal diffusion learn- ing framework in this paper, which is flexible for generating data on arbitrary state spaces. This is achieved by minimiz- ing the need for external, modality-speciff tokenizers and encoders, while keeping the generation in the native spaces of the targeted data.
6. Conclusions, Limitations and Future Works
We propose a novel framework for constructing multimodal diffusion models on general state spaces. We experiment on text-image and tabular data generation, and our ap- proach achieves competitive performances with a signif- icantly smaller model size. One limitation of this work is that we didn't 't explore the possibility of utilizing pretrained unimodal diffusion models as initialization of multimodal diffusion training, which could further boost training elfi- ciency. We leave this as a future direction for investigation.
Decoupled Time Variables. It's worth noting that the de- coupled time design, essential to our proposed framework,

---

Multimodal Diffusion on Arbitrary State Spaces
Acknowledgements
Chang, H., Zhang, H., Jiang, L.,, Liu, C., and Freeman, W. T. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 11315,1135, 2022.
KR, YZ and MT are grateful for partial supports by NSF Grant DMSS-147802 Cullen- Scholarship, and Emory- GT Humannty Award. FY is grateful for partial support by UAlbany- IBM research funding. This research was sup- ported in part by the Al Computing Cluster at the University at Albany.
Chen, C., Ding, H., Sisman, B., Xu, Y., Xie, O., Yao, B. Z., Tran, S. D., and Zeng, B. Diffusion models for multi- modal generative modeling. 2024a. URL https:// arxiv. org/ awaaaaas/aaa/ 2407.17571.
Impact Statement
Chen, J., Yu, J., Ge, C., Yao, L., Xie, E., Wu, Y., Wang, Z., Kwok, J., Luo, P., Lu, H., et al. Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00
This paper presents work aimed at advancing the field of Machine Learning. There are many potential societal con- sequences of our work, none of which we feel must be specifically highlighted here.
2023.
Chen, R. T. and Lipman, Y. Flow matching on general geometries. In The Twelfth International Conference on Learning Representations, 2024.
References
Anderson, B. D. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313- 326, 1982.
Chen, Y., Goldstein, M., Hua, M., Albergo, M. S., Boffi, M., and Banden-Einnden, Probabilistic forecasting with stochastic interpolants and föllmer processes. In Forty-first International Conference Machine Learn- ing, 2024b.
Bai, J.. Ye, T., Chow, W., Song, E., Chen, Q-G., Li, X., Dong. Z., Zhu, L., and YAN, S. Meissomic: Revital- izing masked generative transformers for efficient high- resolution text-to-image synthesis. In The Thirteenth International Conference on Learning Representations, 2025, URL https: http:////ppeeeeeeeeeeoee id=GJsuYHhAga.
Cheng, A., Lo, A., Lee, K. L. K., Miret, S., and Aspuru-Guzze A. Stiefel flow matching for moment- constrained structure elucidation. International Conference on Learning Representations, 2025. URL https:tt/////.ctt pectps://openreeee /opp//eeeeeeeeeeeeeeeeee/ee.eeee n forum? id-8 8 4WmbzikPP.
The Thirteenth
Bao, F., Nie, S., Xue, K., Cao, Y., Li, C., Su, H., and Zhu, All are worth words: vit backbone for diffusion models. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 22669 22679, 2023a.
Chi, C., Xu, Z., Feng, S., Cousineau, E., Du, Y., Burchfiel, B., Tedrake, R., and Song, S. Diffusion policy: Visuomo- tor policy learning via action diffusion. The International Journal of Robotics Research, pp. 02783649241223666. 2023.
Bao, F., Nie, S., Xue, K., Li, C., Pu, S., Wang, Y., Yue, G., Cao, Y., Su, H., and Zhu, J. One transformer fits all distributions in multi-modal diffusion at scale. In International Conference Machine Learning, pp. 1692- 1717. PMLR, 2023b.
De Bortoli, V., Mathieu, E., Hutchinson, M., Thornton, J., Teh, Y. W., and Doucet, A. Riemannian score-based generative modelling. Advances in Neural Information Processing Systems, 35:2406-2422 2022.
Benton, J., Shi, Y., De Bortoli, V., Deligiannidis, G., and Doucet, A. From denoising diffusions denoising markov models. Journal of the Royal Statistical Soci- ety Series B: Statistical Methodology, 86(2):286-301, 2024.
Dhariwal, P. and Nichol, A. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Campbell, A., Benton, J., De Bortoli, V., Rainforth, T., Deli- giannidis, G., and Doucet, A. A continuous time frame- work for discrete denoising models. Advances in Neural Information Processing Systems, 2222666-88299 2022.
Esser, P., Rombach, R., and Ommer, B. Taming transformers for high-resolution image synthesis. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12873-12883, 2021.
Campbell, A., Yim, J., Barzilay, R., Rainforth, T., and Jaakkola, T. Generative flows on discrete state-spaces: En- abling multimodal flows with applications to protein co- design. In International Conference on Machine Learn- ing, pp. 5453-5512. PMLR, 2024.
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F,, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024.
10

---

Multimodal Diffusion on Arbitrary State Spaces
Gat, I., Remez, T., Shaul, N., Kreuk, F., Chen, R. T., Syn- naeve, G., Adi, Y., and Lipman, Y. Discrete flow match- ing. arXiv preprint arXiv:2407.15595, 2024.
Jin, Y., Sun, Z., Li, N., Xu, K., Jiang, H., Zhuang, N., Huang, Q. Song, Y., Mu, Y., and Lin, Z. Pyramidal flow matching for efficient video generative modeling. In The Thirteenth International Conference on Learning Representations, 2025. URL https: / openreview. net fert/oormm2 wetoormmiiiiee 11:22ddd-6660 66N22RG
Ge, Y., Zhao, S., Zhu, J., Ge, Y., Yi, K., Song. L,, Li, C.,, Ding, X., and Shan, Y. Seed-x: Multimodal models with unified multi-gran -uuti-granularity comprehension and generation. arXiv preprint ar arXiv:2404. /4396, 2024.
Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the design space of diffusion based generative models. Advances in neural information processing systems, 35: 26565-26577 2022.
Hayes, T., Rao, R., Akin, H., Sofroniew, N. J., Oktay, D., Lin, Z., Verkuil, R., Tran, V. Q. Deaton, J., Wiggert, M., et al. Simulating 500 million years of evolution with a language model. Science, pp. eads0018, 2025.
Karras, T., Aittala, M., Kynkäänniemi, T., Lehtinen, J., Aila, T., and Laine, S. Guiding diffusion model with a bad version of itself. arXiv preprini 2rXiv:2406.02507, 2024a.
Ho, J. and Salimans, T. Classifier-free diffusion guid- ance. In NeurlPS Workshop on Deep Generative Mod- els and Downstream Applications, 2021. URL https: //openreview net.ilewwweeeeeeeeeee forum iid IA....EIII
Karras, T., Aittala, M., Lehtinen, J., Hellsten, J., Aila, T., and Laine, S. Analyzing and improving the training dynamics of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24174-24184, 2024b.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, (166800-8511 2020.
Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., and Fleet, J. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633-8646, 2022.
Kelly, F. P. Reversibility and stochastic networks. Cam- bridge University Press, 2011.
Kim, G., Martinez, A., Su, Y.C.,, Jou, B., Lezama, J., Gupta, A., Yu, L.. Jiang, L., Jansen, A., Walker, J. C., al. A versatile diffusion transformer with mixture of noise levels for audiovisual generation. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Holderrieth, P., Havasi, M., Yim, J., Shaul, N., Gat, I., Jaakkola, T., Karrer, B., Chen, R. T., and Lipman, Y. Generator matching: Generative modeling with arbitrary markov processes. arXiv preprint 2xXiv2410020088. 2024.
Kim, J., Lee, C., and Park, N. Stasy: Score-based tabular data synthesis. arXiv preprint arXiv:2210.04018, 2022.
Hoogeboom, E., Mensink, T., Heek, J., Lamerigts, K., Gao, R., and Salimans, T. Simpler diffusion (sid2): 1.5 fid imagenet512 with pixel-space diffusion. arXiv preprint arXiv:2410. 19324, 2024.
Kingma, D. P. Auto-encoding variational bayes. arXiv preprint arXiv: 131261111 2013.
Hu, M., Zheng, C.,, Zheng, H., Cham, T.-.,, Wang, C., Yang, Z., Tao, D., and Suganthan, P. N. Unified discrete diffu- sion for simultancous language generation. arXiv preprint arXiv:2211. 1.1822. 2022.
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4015 1026,-406,22 2023.
Huang, C.WW,, Aghajohari, M., Bose, J., Panangaden, P., and Courville, A. C. Riemannian diffusion models. Ad- vances in Neural Information Processing Systems, 35: 2750-2766 2022.
Kotelnikov, A., Baranchuk, D., Rubachev, I., and Babenko, A. Tabddpm: Modelling tabular data with diffusion mod- els. In International Conference on Machine Learning, pp. 17564 17579. PMLR, 2023.
Hyvärinen, A. and Dayan, P. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
Kynkäänniemi, T., Aittala, M., Karras, T., Laine, S., Aila, T., and Lehtinen, J. Applying guidance a limited interval improves sample and distribution quality in diffusion models. arXiv preprint wwXiv:2404.00720 2024.
Ifriqi, T. B., Astolfi, P., Hall, M., Askari-Hemmar R., Benchetrit, Y., Havasi, M., Muckley, M., Alahari, K., Romero- -oriinn,, A., Verbeek, J., et al. On improved conditioning mechanisms and pre-training strategies for diffusion models. In NewrlPS 2024- eight Confer- ence on Neural Information Processing Systems, 2024.
Lee, C., Kim, J., and Park, N. Codi: Co-evolving con- trastive diffusion models for mixed-type tabular synthesis. International Conference on Machine Learning, pp. 18940-18956. PMLR, 2023.

---

Multimodal Diffusion on Arbitrary State Spaces
Li, M., Shitole, V., Chien, E., Man, C., Wang. Z., Sridharan, S., Zhang, Y., Krishna, T., and Li, P. Layerdag: A layer- wise autoregressive diffusion model for directed acyclic graph generation. In The Thirteenth International Confer- ence on Learning Representations, 2025. URL https: / //openreview net forumm id=kam8 Emul.
Nie, S., Zhu, E., You, Z.,, Zhang, X., Ou, J., Hu, J., Zhou, J.,
Lin, Y., Wen, J.-R., and Li, C. Large language diffusion models. arXiv preprint arXiv:2502.09900 20255.
2025b.
Nisonoff, H., Xiong, J., Allenspach, S., and Listgarten, J. Unlocking guidance for discrete state-space diffusion and flow models. arXiv preprint arXiv:2406.011. 2024.
Li, S., Kallidromitis, K., Gokul, A., Liao, Z., Kato, Y., Kozuka, K., and Grover, A. Omniflow: Any-to-any gen- eration with multi-modal rectified flows. arXiv preprint arXiv:2412.01169 2024a.
Ou, J., Nie, S., Xue, K., Zhu, F., Sun, J., Li, Z., and Li, Your absorbing discrete diffusion secretly models the conditional distributions of clean data. In The Thirteenth International Conference on Learning Representations, 2025. URL https:/ /ttp///pppppeew.e/oe/peeeeeeeeep net forum? i dd 8Tanm.
Li, T., Luo, W. Chen, Z. Ma, L., and Qi, G.-J Self- guidance: Boosting flow and diffusion generation on their own, arXiv preprint 1rXiv:2412.05822 2024b.
Peebles, W. and Xie, Scalable diffusion models with transformers. In Proceedings of the IEEE/CVE Interna- tional Conference on Computer Vision, pp. 419544195544 4205,, 2023.
Lin, T., Maire, M., Belongie, S., Bourdev, L., Girshick, R., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitmick, C. Microsoft COCO: common objects in context. In ECCV, 2014.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Liu, H., Li, C., Wu, Q. and Lee, Y.J. Visual instruction tun- ing. Advances in neural information processing systems, 36, 2024.
Liu, T., Qian, Z., Berrevoets, J., and der Schaar, M. Goggle: Generative modelling for tabular data by learn- ing relational structure. In The Eleventh International Conference on Learning Representations, 2023.
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip latents. arXiv preprini arXiv:2204.06/225 1(2):3, 2022.
Ren, Y., Chen, H., Rotskoff, G. M., and Ying, L. How discrete and continuous diffusion meet: Comprehensive analysis of discrete diffusion models via stochastic inte- gral framework. arXiv preprint ar arXiv:2410.03601, 2024.
Lou, A., Meng, C., and Ermon, S. Discrete diffusion mod- eling by estimating the ratios of the data distribution. International Conference on Machine Learning, 2024.
Lu, J., Clark, C., Lee, S., Zhang, Z., Khosla, S., Marten, R., Hoiem, D., and Kembhavi, A. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 14439-26455, 2024.
Ren, Y., Chen, H., Zhu, Y., Guo, W., Chen, Y., Rotskoff, G. M., Tao, M., and Ying, Fast solvers for discrete diffusion models: Theory and applications of high-order algorithms. arXiv preprint arXiv:2502.00234 2025a.
Ren, Y., Rotskoff, G. M., and Ying, L. unified approach to analysis and design of denoising markov models. arXiv preprint arXiv:2504.011 01938, 2025b.
Luo, E., Hao, M., Wei, I.,, and Zhang, X. scdiffusion: conditional generation of high-quality single-cell data using diffusion model. Bioinformatics, 40(9):btae518. 2024.
Rojas, K., He, Y., Lai, C.-H., Takida, Y., Mitsufuji, Y., and Tao, M. guidance for discrete diffusion models. arXiv preprint arXiv:2507. 08965, 2025a.
improvements to classifier free
Lyu, S. Interpretation and generalization of score matching. arXiv preprint arXiv:1005.2629, 2012.
Rojas, K., Tian, Y., Tao, M., Nevmyvaka, Y., and Deng, W. Variational schrödinger momentum diffusion. arXiv preprint arXiv:2501. 16675, 2025b.
Meta, C. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.00 009118, 2024.
Nie, S., Zhu, F., Du, C., Pang, T., Liu, Q. Zeng, G., Lin, M., and Li, C. Scaling up masked diffusion models on text. In The Thirteenth International Conference on Learning Representations. 2025a. URL https: //openreview www.eweee/oorrreeeeeeeeee dww.wwwKOt....
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp. 10684-10695, 2022.
12

---

Multimodal Diffusion on Arbitrary State Spaces
Ruan, L., Ma, Y., Yang, H., He, H., Liu, B., Fu, J., Yuan, N. J., Jin, Q., and Guo, B. Mm-diffusion: Learning multi-modal diffusion models for joint audio and video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10219- 10228, 2023.
Van Den Oord, A., Vinyals, O., al. Neural discrete rep- resentation learning. Advances in neural information processing systems, 30, 2017.
Vincent, P. A connection between score matching and de- noising autoencoders. Neural computation, 23(7):1661 1674, 2011.
Saharia, C., Chan, W., Saxena, Li, L., Whang, J., Denton, E. L.,, Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al. Photorealistic text-to-image dif- fusion models with deep language understanding. Ad- vances in neural information processing systems, 35: 36479-36494, 2022.
Wang. X., Zhang, X., Luo, Z., Sun, Q., Cui, Y., Wang, J., Zhang, F., Wang, Y., Li, Z., Yu, Q. et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.1880 2024a.
Wang. X., Zheng, Z., Ye, F., Xue, D., Huang, S., and Gu, Q. Diffusion language models are versatile protein learners. In International Conference on Machine Learning, pp. 52309-52333 PMLR, 2024b.
Sahoo, S., Arriola, M., Schiff, Y., Gokaslan, A., Marroquin, E., Chiu, J., Rush, A., and Kuleshov, V. Simple and effective masked diffusion language models. Advances in Neural Information Processing Systems. 37:130136- 130184, 2024.
Watson, J. L., Juergens, D., Bennett, N. R., Trippe, B. L., Yim, J., Eisenach, H. E.,, Ahern, W., Borst, A. J., Ragotte, R. J., Milles, L. F., et al. De novo design of protein struc- ture and function with rfdiffusion. Nature, 620(7976): 1089-1100, 2023.
Schiff, Y., Sahoo, S. S., Phung, H., Wang, G., Boshar, S., Dalla-torre, H., de Almeida, B. P., Rush, A., Pierrot, T., and Kuleshov, V. Simple guidance mechanisms for dis- crete diffusion models. arXiv preprint 1rXiv:2412.00193, 2024.
Wu, C., Chen, X., Wu, Z., Ma, Y., Liu, X., Pan, Z., Liu, W., Xie, Z., Yu, X., Ruan, C., et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. arXiv preprint arXiv:2410. 1a..0/../880 1384,, 2024.
Shi, J., Han, K., Wang, Z., Doucet, A., and Titsias, M. Simplified and generalized masked diffusion for discrete data. Advances in neural information processing systems, 37:103131-103160 2024.
Xie, J., Mao, W., Bai, Z., Zhang, D. J., Wang. W., Lin, K. Q.. Gu, Y., Chen, Z., Yang, Z., and Shou, M. Z. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408. 12528, 2024.
Shi, Q. Bai, J., Zhao, Z., Chai, W., Yu, K., Wu, J., Song, S., Tong, Y., Li, X., Li, X., et al. Muddit: Liberating generation beyond text-to-image with unified discrete diffusion model. arXiv preprint ar0s: 23606, 2025.
Xu, X., Wang, Z., Zhang, G., Wang, K., and Shi, H. Versatile diffusion: Text, images and variations all in one diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7754-7765, 2023.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequi- librium thermodynamics. In International conference on machine learning, pp.
Xu, Z., Qiu, R., Chen, Y., Chen, H., Fan, X., Pan, M., Zeng, Z., Das, M., and Tong, H. Discrete-state continuous- time diffusion for graph generation. In The Thirty- Annual Conference on Neural Information Processing Systems, 2024.
PMLR, 2015.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A.. Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:201 1. 13456, 2020.
Ye, H., Kevin, R., and Molei, T. What exactly does guidance do in masked discrete diffusion models. arXiv preprint arXiv:2506. 10971, 2025.
Sriram, A., Miller, B. K., Chen, R. T., and Wood, B. M. Flowllm: Flow matching for material generation with large language models as base distributions. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Yim, J., Trippe, B. L., De Bortoli, V., Mathieu, E., Doucet, A., Barzilay, R., and Jaakkola, T. Se (3) diffusion model with application to protein backbone generation. In In- ternational Conference on Machine Learning, pp. 40001 40039. PMLR, 2023.
Swerdlow, A., Prabhudesai, M., Gandhi, S., Pathak, D., and Fragkiadaki,, K. Unified multimodal discrete diffusion. arXiv preprint arXiv:2503.20853 2025.
Zhang, H., Zhang, J., Srinivasan, B., Shen, Z., Qin, X., Faloutsos, C., Rangwala, H., and Karypis, G. Mixed-type tabular data synthesis with score-based diffusion in latent space, arXiv preprint arXiv:23 10.22330.996666 2023.
Tschannen, M., Pinto, A. S., and Kolesnikov, A. Jetformer: An autoregressive generative model of raw images and text. arXiv preprint arXiv:2411.1972) 2024.
13

---

Multimodal Diffusion on Arbitrary State Spaces
Zhou, C., Yu, L.,, Babu, A., Tirumala, K., Yasunaga, M., Shamis, L.,, Kahn, J., Ma, X., Zettlemoyer, L., and Levy, O. Transfusion: Predict the next token and dif- fuse images with one multi-modal model. arXiv preprint arXiv:2408. / 1039, 2024.
Zhu, S., Zhu, Y., Tao, M., and Qiu, P. Diffusion gener- ative modeling for spatially resolved gene expression inference from histology images. In The Thirteenth International Conference on Learning Representations, 2025a. URL httpa: http://ppppeeeeeee//e/ppe net/ forum? Ld-FtjLU
Zhu, Y., Chen, T., Kong, L., Theodorou, E., and Tao, M. Trivialized momentum facilitates diffusion gener- ative modeling on lie groups In The Thirteenth In- ternational Conference on Learning Representations, 2025b. URL https:://ppenreeieer net/forun id=DTatjJTDl1
14

---

Multimodal Diffusion on Arbitrary State Spaces
A. Proofs
A.1. Proof of Theorem
The following proof follows a similar idea as is presented Benton et al. (2024):
Proof. By Jensen's 's inequality, have:
log [ X ,,....,,"" x²)| Xi = z²1) S Ellog()) X xx,...,x") Xi = x²1
This implies the following inequality:
log (EIf(e),.. X x²))
Xg = x²)) log f f(x)
E [[log(f)(z* , X x²)
X x²| - log f(x)
LHS
RHS
where we denote (at ...., x²). Taking the limit as 6 + notice that the limit of RHS equals Lxi (log (zr). On the other hand, if we consider the following function g:
g(h) = EEf((" x,,,..,;""
Xi = =x"]. g(0) = f(x)
We calculate the limit of LHS as 1' + 0.
log(g(t')) log g00) (log()))(() - g(0) d'(0) f(x
lim LHS
lim --
-
1
((x)
Combining them, we have that
Cx-f(a) f(x)
2 A(oo (x)
Applying this result by choosing f B/. have that
(x/ (x) (p/ 3o (xr,,
Lxx log(v/3o)(2r)) 1) 2 0
This finishes the proof of IGESM 0. To that IGESM is minimized when B0 t) p(æ, t), note that Jensen's inequality holds the equality sign when the test function is constant. Therefore, (9) holds the equality sign for each Lx* whenever p/3g is identically constant, therefore IGESM is optimized when 3g is equivalent to up to multiplicative constant.
A.2. Proof of Theorem 2
Proof. We will start by showing the equivalence between IGDSM and IGISM, and then we will demonstrate the equivalence between IGISM and IGESM to finish the proof. We start with the definition of IGDSM.
(xool (alnnBBo)(ee (o) Lxi Bo)(Pe)o/B))(xe... (Pejo/ (x)(
IGDSM
IE PP
(x (B) (o) Lxi log(p (oo(p t) dæt (Pejo/ (xt,
E
pelo(zee
t,Pu x
E
(o t) (pe10/ () log(pr / (o) t) dx
t,po
X
15

---

Multimodal Diffusion on Arbitrary State Spaces
Then, using the properties of the adjoint Exc. we can continue to simplify the objective as,
IGDSM
E t, x
(x (ejo( (00 Pt ( t) log(peo/ (2 b)
dxe
(xx Lxx log Bo( 1) Ex: log (2 1) dxe
E t,po
(ejo 1)
Σ
B(orett) 1)
L (xx 1)
I
+ Lx log Bo Bo( ()
+ const
to Pejo
Bo(xt,
Lx,Bol(02, 1)
fxx log Bo (2
const
(e
IGISM + const
This finishes the proof of equivalence between IGDSM and IGISM To show the equivalence between IGISM and IGESM we start with the definition of IGESM.
Cxi(D/Bo) Lxi log(p//0)(x11)
IGESM
E t.aerp(-et)
(p/Ba)(zt, (x),
1//// p(zr, t)
Lx Cxi (p)
Lxi log(p/ t) dæt
(0/30)(00r.)
E Σ Bo(xx L) (p/ ((a,,t)Cxx 1og(p/ t) dx
Using again the properties of ajoint C'x. we have
IGESM = E. /
Σ
Bo t)(p/ p(x,, x) log(p/ Bo) (are, t) dare
/ place, 1)
Cxx 9xx Lx (oolp///))))) (a),, t) dact
= E. t
Bo( t))
Bi 1)
Lxi logplæz, t) + Lxi log Bo(xr,
t.
Bolwx.
Sx 1)
()) + Lxi log (o t) + const
ICISM const
This finishes the proof of equivalence of IGDSM, IGISM, IGESM up to an additive, 0-independent constant.
A.3. Proof of Proposition 1
Proof. In the following, we derive IGDSM when choosing the forward process as in (6). Recall that IGDSM is given as,
(o) (o) t) (ptol (x),, Lx 180
Ly (p (E),
IGDSM
E tepo
Ly Log(Pro/ BB)
(oad\/
Jx
Jv
Xo zo, Yo = vo). Lx and Ly are generator of the following where t = (t, xe (Xe, Yo). Pejo p(Xe,
dynamics.
dxx, f(Xe, t)dt (t)dBB
Y, ~ CTMC(Q2), (Xo. Yo) ~ Pdata (I, y)
16

---

Multimodal Diffusion on Arbitrary State Spaces
See Lem. 1 for detailed proof. We start by computing the score matching operators + x and By for X and Y respectively. Note that Lx and Ly given by the following expressions for test function h h(22, Us. 1s,
1 2o(t)ah, Cyh Σ
Lxh= Vh
h(r y, v,t, s.t,
MEX
Therefore, for the score matching operator associated with Lx, can be expressed as
Lxh
Φ (h)
Lx logh
f. Vh f-
"2011² Ah
- log 1 log h
h
h
V C.V logh) h)
g0))).
gt()) log h
Vh
g0)) log h
h 50(1 Vlog h 5 ((())2 log
Note that Vh h
Vlogh, h, V V log = A log h, therefore we derive that
1
x ( log
For the score matching operator associated with Lv, it can be expressed as,
Lyh
(h))
Lr log h
h
h(r,, y. 8)
L. ya)
log h(z,, V, t, 5))
(i yes
MEX
h(I2, t.
h(x, y, 2, s) h( yo t.
Σ
Q (v. y2)
log
h( hi Ne, t,
yEX
where the last line follows since Σ Qo(v, 0.
In this case, the Lx related term Jx IGDSM can be simplified as,
Jx (x(er Bo) (2
Lx 10)(Pt10/3)(2zz)
(Pejo/ (0)
( (zt (e,tt t)
Peju
x
t.po
50(()
V log p( Mos 6, Tos Mo) log (x Mo. 2, 8)
t.poPeejo
Moreover, since (X,, Y,) are conditionally independent given (Xo, Yo), we have that
v loggp(re,Bark
Zoz Mu)
V log (p(zz, t wo: vo) p(ys, Zo. Yo))
V log p(re, No, yo) V log p(ys, xo. yo)
V p (r)
which suggests that under our framework, the multimodal conditional score is identical to the unimodal conditional score. Using this, and set si (It, t, s) V log Ba (x ye t, s), we have
27
Jx
2 9 (t)
(t)
sp
-V Vlogp((<<(zo)
17

---

Multimodal Diffusion on Arbitrary State Spaces
Similarly, the Ly related term Ty in IGDSM can be simplified as,
Pejo
Fv = E
PP
Bo
Σ
Qv(v.M Mo.
plas, U.t.s
2o. Mo)
B0( No. t,,t,))
log p(ur, Us. t,
pla M, t, 8 Io. yo)
+ log (2 Us t, 8) (x Uo 1,
t.po
p( Vor 8
ro. vo)
(a U, 8)
2o, vo)
yEX
Bo(xs, yo t, te, V,
(0 y 8) (z ye t, 8) dæ,
E
Σ V.
Qolv. Me) (p(t,, vo.30) yo)
+ p(æe, Vo To. yo) log
to
+ const.
+ pl Mor Nar 4 Bo. Mo) p(a,, U, t, Uo)
log Bo B 6,
E
E u
Qu(y Us)
Bo (I Uss Usst. dat
to
EX
Rd
Bo
+ const
Now, we exchange variable ys and y. and thus the expression is rewritten to,
Qo w){
Bo(21. t.
p( U. t,
Fu. vo)
Bo (x. v.t,
Jy E
E
log Be vo,t,s)
dx
yEX Rd
Bempleer
Bo Vort, pl Ves xo. Mo)
+ const
E P.PooPPjo
Σ
Qu(s v)
( (,t..
plr,, y. s Fu, yo).
log Bo Bo V. v.
]
+ const
p( Us, Us, 1,8
xo, yo)
(ot Usy t,
yEX
Using again the conditional independence of (X,, Ys) given (Xo: Yo), we have that
p y, s plat, Vsr zo, yo)
ro. Yo)
p t
Zo. Mo) ru. Mo)
p(a,, t
To, Wo)P (Mor : 5 xo. yo) Fo. vo)
ply, s
p(y. Mo0
ply..
Zo. yo)
p(y.. s vo)
which indicates again that the multimodal conditional score identical to the unimodal conditional score. oy (Ire Mar L, 8)y = 3o(xt; Ur 1, 3)/Bo((r, Mar L, 8), we finally arrive at that
Set
E..u,, v) s% (Ze, No, p(u. 30). log sy (Ie, Marl, s) ]] p(y,; vo)
Jy
E L. Po Tiilla Po, 1D
+ const
VEX
Note that while the sum in Jy is over y E X, in fact when y = ys the corresponding term is constant and has no contribution the gradient of 0, therefore we can instead only summing over y E X and y * ys. recovering the presented expressions in Prop. This concludes the proof.
A.4. Proof of Proposition 2
We start by considering the function u(x,, y, (, [[,,X, Y.) Xo = I. Yo = y) as test functions. We start by computing the generator of the forward process. For notational convenience, we use L and LV denote the generators of X, and Y.. respectively.
Lemma 1 (Generator of the forward process). Given (Xe, Y.) following dynamic (6), and test function Rd x X, have that:
C = (V f(x,(( + 1 (V) ( vu+ 1 7 (t)
(10)
C u Σ u(=,,)Q..0.00 QQ_ ((r
(11)
TEX
18

---

Multimodal Diffusion on Arbitrary State Spaces
Proof. We start by computing the generator of y;:
1
L
lim As
E u Yatas) - u(x, | ,,,,,
48-40
lim
1
Σ
(u(z,0) = Y, =) - -P,, C Y, y))
20-0 As
SEX
Σ u(2, 9)
lim (P(Yo+ss= = = - y) - P(Y, 9 Y. = v))
-- As
BEX
u(æ, (B.0) u(,,)
ØEX
Similarly, for the generator of Xx,
L u lim A1-0 At
1
E
u(X++AI, V.) - u(x,, Xx =., m]
= lim
1
E
Yo), Veu(X,, X))) + ())((),,, V,(),, K.) x,=
= x, Y, = 8)
At
(f(x,, Y,), Vau(X,, Y.)) + + (t)))) v'u(Xe, x,= x, Y,
1 = (J (u) v) + 2 (1)
where we uses Ito's s lemma and the fact that the martingale vanishes under the expectation.
We now compute the generator for the backward process:
dx - F.~CTMO (Q(X,,)))) (x Fo) ~ p.T.
- - - 1) log (X,, V.)de + o(OaW,
(12)
where Vr,. = Pr-,T-s (X,,, 1, 8) a rate matrix with y', y entry being (r,a(Xe.g) as the generator of backward process X t. Y in (12) respectively.
(Qr-") We denote L. and CX
Lemma (Generator of the backward process). Given X,, Y, following (12) and test function IRd x X, we have that:
C = C = (c,tt,) = Σ u(r 0) (c.t,s) BEX
t)) + - ()VT-ttVVu(up. (V.u(z, of Va log (z, v)
1
The proof is highly similar to that of Lem. 2, and thus we omit it here avoid being repetitive. With LX LY Cx and CV being computed, we can now derive the Fokker Planck equation for both the forward process (Xe, Y.) and backward process (X,, Y.) =)), as is given in Lem. 3 and Lem. 4.
Lemma 3 (Fokker Planck Equation of Forward Process). For X1, Y, following (6), let p(1, y, L, 8) denotes the density of X2, Y,) at X, = x, Y, = y. we have that:
0.p(x,y,t,5) C p(s,,,,,,s)) 0,p(e,y,t,s) C =CX'p(z,gytt
(13)
Where _X,,, ex*,Lr represent the adjoint of LX LY and:
x cx*p(z,y,t,,) = - (f(z,t)p(a, 8)) + (1)
"x(p(,, Qap(a, (4,
Σ Qv(v.)) p(c.
SEX
19

---

Multimodal Diffusion on Arbitrary State Spaces
Lemma (Fokker Planck Equation of Backward Process). For X t: Y following (6), let p(a, y, L, 1,8) denotes the density of X, Y at X, x, Y y/ we have that:
0,,,,,,,,,)) L. (I, 0,7(,,6,68) -(.
(14)
where Cx fx represent the adjoint of Lx CL and:
p.), = v (f(x,T-t)plaa (( (,,,,,,) Σ yEX
+ + (T - t)V (p(2, s), log (z. v))
The computation central to the proof of Lem. 3 and Lem, 4 is the calculation of the adjoint operator, which can be done with standard techniques, such as integration by parts. The derivation of the Fokker Planck equations directly follows from definitions. Therefore, we also omit the proof here. With these results, we are now ready to show that (X₁, Ys) and X Y. time reversals of each other.
Lemma 5 (Time Reversal), p(x, y, t, (z.. = p T,,, s), where p is the solution to the Fokker Planck equation of the forward process in (13), p is the solution to the Fokker Planck equation of the backward process in (14).
Proof. We will prove the result by showing that p(z, U, t) = p(x, T T, satisfies the Fokker Planck equations
given (14). We start by showing the X, related equation. Substituting in p(a,, y: T 1, T *). we have that
0,plo,u,t, t, 8) = Oe(pEz y, T - -,T =)) -dp(( y, T - t,T - s) - xx xxx,,((x y. t, 1, -s) s)
where the last equality holds due to (13). On the right side of the equation, the expression can be simplified to:
p(a, = C p(a,y, t, s)
1 7 - s)
=V- - y, -,T
-T - t/
V. (f(x, -00(,,) -)) 2 - 1
-9² -17--113 1)V -1,
V - 100o:T--00((,,,TT111777 3 2 (T 1,
2
-xxiipzz, (112,,77 T,T
Therefore, we show that O,p(r.y,t, 8) = xxxxp(x, y. U.t. 8) when p(r,y, equation, we have that
v, s) = =p(r, y, T t,T - s). For the Y related
U, (t = 0, FT - t,T - t,T - 5) = -K''p(z "p(x,,,T""
- V.T - -)
Similarly, the right size of the equation, we have,
C ple
s) (,,,), p(z. -.T
=Σ (c. -, -,)-s)
NEX
Σ
(1,,4,00.0000.. -) oo - s)
VEX
= Σ
Σ
VT. -tt, -s)
JEX.
TEX.
20

---

Multimodal Diffusion on Arbitrary State Spaces
where in the last step, we use that (tx, I, (v. = E (x. L, 8) (Ū: y). We perform such a simplification since
we can only relate @ (zz t, s) and Qr-s on non-diagonal entries. Then, it holds that
P(Z.V.T t, 8) pla.i,T' t, p(z, D.T - -,
p(z,,t)
Σ
Qr-o(0.2)
DEX.Ú7y
5) - QT,-(( -,, 8)
P V, - T,T-
Σ
Qr (Ú.))() y, 1,T s) QT (1, U, 1.T 8)
8 8EX
1, -8) - QT (v. (v. ()) 1,T -8) 5) Σ QQ- (0. -
SEX
= p(z, T - - s) E Qr-s(0.1) Σ
Irr-v(v.0pp(. D. -
ØEX
DEX
Σ Qv-o(v.0)p( ū,T - --s
SEX
L p(x. V, 1,T s)
where the derivation, we use the definition of (2, t, 8) and the fact that Egex Qr= y) = O. Therefore, we have also shown that 0,,( y, 8) * x. U, 1, s) when p(ar, y. 1, 8) p(.. U,T T t, T s). Together with the fact that the initial conditions are matched by construction, i.e.,
f(. y, 0, 0) = p( y, T)
conclude the proof of the time-reversal argument, well Prop. 2.
B. Design choice for Continuous- Multimodal Diffusion Models
In this section, we go deeper into the design choices of the newly proposed continuous-diserete multimodal di diffusion models, including the choice for forward process, score parameterization, and sampling algorithms for such diffusion models.
B.1. Choice of Forward Process
We consider the following specific choice of forward process the Continuous-Disce Diffusion Model, where we choose X₁ be subjected time-rescaled Prnsten-UUhlenbee process (Song et al., 2020) and Y, to be subjected a masked discrete diffusion model (Ou et al., 2025; Sahoo et al., 2024; Shi al., 2024). Both choices effective when modeling unimodal distributions, prompting us to combine them for the design of multimodal diffusion models on their product space.
dX,, -BiXi + /23,dB Y, ~ CTMC(a, (Quakk)) (Xo, Yo) Paaua(r,) y)
(15)
To define masked discrete diffusion models, we need to introduce an additional mask token M into the state space X. Therefore, the transition rate matrix Quask is a transition matrix defined on the extended state space X = X U (M), given by
0
0
0
0
mask
:
7. ;
1
1
1
where the last row corresponds to M.
21

---

Multimodal Diffusion on Arbitrary State Spaces
Consider now the practical case, where X RRd, and Y, consists of a sequence of tokens, i.e., Ye X We represent Y, as Y, 91-.. Vi..- yn-
For pure masked discrete diffusion, it has been shown that its score has special factorization (Ou et al., 2025). This enables users to parameterize only the probability of the clean data distribution conditioned on unmasked tokens, thereby representing the score. In the following, we demonstrate that this special structure is inherited in the continuous-discrete diffusion model when the discrete part is masked discrete diffusion, which enables a more effective score parameterization, as well variance-reduced version of the training objective.
Now, consider y 1/1 ........ V......... ..... Yns and y/1 ....... Di ... Yn. where y differs from ý only the i-th position, y: MM while y: A M
Proposition 3. Let p(a, y, t, s) be the density (15) at Xt = x, Ye = ys then the discrete score has the following form
p(x: Hz, Y, = y,1,8) p(x -,,Y, = 1,
e
PP = yi 1 UM Xe,()
(16)
e
where Se (r))dr, yoM contains the unmasked tokens of y-
We begin by noting that, given the forward process defined in (15). We can solve the forward process analytically as follows:
Proof.
P(Y - A), y # M 5 1-
P(Y = 1
(17)
y= M
For any event A. We will use this fact several times in our proof. This is true since when y # M.
P(Y: y A) P(Y y Yo y, A) P(Y y A)
Our proof consists of two steps. First, demonstrate that the diffusion process be factored into conditional probability and time-dependent term. Secondly, demonstrate that can further simplify this conditional probability to contain only probabilities in terms of the clean data distribution.
Step 1: The discrete score in (16) is given by:
F(X, = Tt, Y
Ul
Yi Y/y.... Y = yn))
P(X: = Ft, Y! V1,
Vi M,
Yn = yn)
To simplify the notation, we define the following notation A we can rewrite the discrete score as:
{Y* = yk : k # i) n (X, = }. Then, using Bayes' rule,
P(Y Vil A))((A) P = M A.)((A)
P(Y: yi A c c P(Y: M A))
PY
Ax)
1-
Step 2: We now show that we can simplify A, by removing the conditioning on tokens that are masked. But firstly we do a simple calculation that will come in handy, given events A, B, one has that:
F(A BnC) P(ANBDC) PICN(C))P(CC FCCIANBRAAN C(BBC) P(BOC)
P(C))(BBCC
P(AN P(AB))(CIANB) P(B))
P(C|B)
Now, assume that is a position such that y1 = M. We denote A/= (V-: yk #
111 (X, = xx) the event given by
22

---

In Alg. I, we present the detailed algorithm for using Noisy guidance for continuous score. In Alg. 2, we present the detailed algorithm for using Noisy guidance for discrete score, whose computation not mentioned in the main text. Note that in Alg. instead of directly doing geometric average of 89 uncoodd. and 59 compute arthimetic average of the corresponding logits, then use softmax to calculate the actual guided score. Such Multimodal Diffusion on Arbitrary State Spaces
the remaining tokens. We then use the calculation above write:
P(Y y/ A,) P(Y ys A Y = M)
P(( = y4 n A (Y, MY)
P(4 n (Y, = 3
P(Y = M (1) )
P(Y) = Us A
P(Y M A
P(Y = ys A
where in the last step, used (17). The computation above shows that can remove any events relevant masked tokens, and repeating this process yields the result that the event being conditioned on only contains the unmasked part. This finishes the proof.
With Prop. 3, the score entropy in the training objective described in Prop. I can be further simplified to cross-entropy loss, as is widely discussed in the literature of masked discrete diffusion model training (Ou et al., 2025; Sahoo et al., 2024; Shi et al., 2024). Prop. 3 also implies that when designing the score network backbone, the discrete score does not need the input of time s. However, it's still dependent on the time / of the continuous modality. This a notable difference from the pure masked discrete diffusion models considered in the literature (Ou et al., 2025; Nie al., 2025a).
B.2. Noisy Guidance
cond.y is suggested in Nisonoff al. (2024),
practice is considered in Chang et al. (2022), and recently it has been shown to have theoretical advantages in Ye et al. (2025); Rojas et al. (2025a).
Algorithm Noisy Guidance for continuous score
Require: Kt, noisy image with noise level, model: sx( Mar L, 8, w), yo clean text, w Guidance Strength, a Conditioning Noise Level
Ensure: sØ Guided continuous score
1: poisy ~ (o(l(1))))
t so(a, yo y0 ot wi), sg goond, so you w))
2: $n
3: sg f- -5 + (1 w) uuncond.
cond.x
4: return sg
Algorithm 2 Noisy Guidance for discrete score
Require: 8 noisy text with noise level, model: so sa ye. s,w), 2o clean image, Guidance Strength, Conditioning Noise Level
Ensure: 8i Guided discrete score
muiss P=)o(=(zo)
uncond.y + so (x poisy yo. w.). og .ond.y + so (2o, Mo O, wa)
2: $o
uncond.y softmax(fancoo)) (auncond), b9 gonddy softmax( gcoodd)
3: 8g
4: sy softmax (w geond + (1 w) guncond)
5: return s
B.3. Samplers
For inference of cantinuous-disered multimodal diffusion models, we consider the following samplers. For conditional generation of discrete or continuous modality, see Alg. 3 and Alg. Among all the pseudo code, we set times
23

---

Multimodal Diffusion on Arbitrary State Spaces
0,1,...,NN N as the inference times and dt -1/N as the time steps, where N is the number of total inferenceediseretiizainn steps. We use r-leaping for discrete modality sampling and Heun's method for continuous modality sampling, All the inference algorithms are written with the presence of guidance and guidance intervals.
-,1,,,,,,1,,.....
Algorithm 3 Discrete Sampler with T-leaping
Algorithm 4 Continuous Sampler with Heun's method
Require: : Number of steps, w : Guidance Strength
Require: N Number of steps, w Guidance Strength
1: [a, b]: Guidance Interval, model : so (2t, Vs: i, w)
1: [a, b] : Guidance Interval, model s0(Te, Us, t, 8, s,w)
2: x : a clean image condition
2: A clean text condition
Ensure: Yo ~ Páata (-12²
Ensure: 20 ~ Pdata--ly)
3: Ve [M,..., .M]
3: 24 < 1),
4: for in times do
4: for t in times do
5: We = w if E [a, bJ else 1.
5: wi = w if E [a, b] else 1.
6: $g.50 + sa (z, Mr. 0, t, w
6:
sy + se(1,, V. wr)
7: V/ - -lepping s0 ye, t, (dt))
7:
Vold = f(z,,) t) 2g)
8: end for
8:
I - x + Voyll, 88 SM + 8g y, 1 dd, 0, wn)
9: return y/o
9: Unew = f(i,, 1) g (t)
10: Te 4- I4
(Vold Unew)
11: end for
12: return .r0
We describe one multimodal sampler for the joint generation with a certiiuoou-discced multimodal diffusion model in Alg. 5. Essentially, Alg. 5 combines r-leaping for discrete modality and Heun's method for continuous modality, each depicted in Alg. 3 and Alg. 4. However, these choices are selected without being heavily optimized to tailor to this case, and potentially, there exist much more effective and efficient samplers. For example, note that Heun's 's method is recond-order ODE sampler, while --leaping is usually considered to be first-order CTMC sampling algorithm (Ren et al., 2024). This means that the discrete score obtained at the mid-point during the inference step of Heun's method is not being used, which causes a waste of computation. A potential way to improve to replace the first-order discrete sampler with second-order variant, such as the Trap algorithm introduced in Ren al. (2025a). We leave this direction for future investigation.
Algorithm 5 Multimodal Sampler with r-leaping and Heun's Method
Require: N Number of steps, w Guidance Strength, [a, b] Guidance Interval, model: s0( yo. t, s,w)
Ensure: To, Yo Pdata
1: xı -O, I), ye + M ..... M]
2: for in times do
3: Wt = w if t [a, b] else 1.
4:
sg.50 f- so S0 Vc, t, t. we); Vold = f(x,,t) - (t)
5:
f+ 2i + vll
6:
8 + sp (i, Ye, t. + dl, 1, wi), Unew = 1 1) 2 (1)&g)
7: It + It + 1 (Vold + Vnew)dt,
Ye t i-leppppnnng Mc, t )dt)
8: end for
9: return Z0. 20
C. Experimental Details on Text- Generation
C.1. Choice of Forward Process
We consider the same forward process discussed in App. B.I, with 33 and a, given as
B, = 500 (v Batart - 1) + /v Bend)²
0, 1 - 5
(1 - 8)s B 0.00085,
Bend = 0.0120, 10-5
24

---

Multimodal Diffusion on Arbitrary State Spaces
Table 4. Hyperparameters for inference of different tasks
Parameter
text to image
image to text
joint
Number of Steps
50
50
50
Guidance Scale
5.0
1.0
5.0
Guidance Interval
[0.3, 0.8]
[0.3, 0.8]
Condition Noise Level
0.77
1.0
Early Stopping
10-5
10-5
10-3
Table 5. Model hyperparameters
Table 6. Training Hyperparameters
parameter
value
Parameter
Stage
Stage 2
Stage
patch size
Num Itr
600K
200K
140K
joint depth
EMA-3
.99999
.9999
.9999
text depth
6
Batch Size
256
512
512
image depth
Optimizer
A
Adam W
AdamW
dim text
1024
Learning Rate
2e-4
2e-4
2e-4
dim image
1024
Adam-6's
[99
[99
(99,9)
dim joint attention
1024
Weight Decay
0.03
0.03
0.03
QK RMSnorm
true
dimension per head
64
number of heads
C.2. Training Strategy
We divide our training into several stages. This is standard practice for training vision-language models. In text-to-image diffusion models, pretrained text encoder is used to achieve alignment between the text semantics and the image features. Popular choices in the literature are using CLIP or T5 as text encoders (Esser et al., 2024). However, in our use case, we require training on masked text, for which the availability of pretrained encoders is limited. For this reason, we decided not to use pretrained text encoder. This has the advantage that we don't 't rely on any pretraining, which reduces the computational requirements of our model.
Stage 1: Text-image Alignment During this stage, we train both the joint embedding and the continuous decoder. We allow noisy text to be received as input to our model, meaning that we train all possible combinations of 8 and I, but without worrying about the text prediction task. We present all training hyperparameters across all stages below.
Stage 2:
Text prediction and Image Improvement In this second stage, we freeze the joint embedding. We found that by doing so, we can simplify the training. The joint embedding now capable of generating meaningful latent representations, which can then be used predict clean text from masked tokens and latent image representation.
Stage 3: Multimodal Generation Finally, train both the image and text decoders. This is useful because the image decoder hasn't 't been trained specifically predict from the frozen joint embeddings. Training the text decoder is not necessary, but we can get some extra training time by doing so. After this stage, our model is now capable of performing all lasks.
Optional- - Stage 4: Fine-tuning on downstream Tasks When necessary, our models can be fine-tuned on downstream tasks to improve the performance.
C.3. Sampling
For sampling, we use the samplers described in Alg. 3, Alg. 4, and Alg. 5, where we do not use guidance for the discrete component. Our default values for sampling are presented in Table
25

---

Multimodal Diffusion on Arbitrary State Spaces
C.4. Hyperparamters
We include the network and training hyperparameters in table 5 and respectively, The total model contains 578M parameters and the joint embedding plus single modality is about 481M.
D. Experimental Details on Mixed-type Tabular Data Synthesis
D.1. Choice of Forward Process
We consider the same forward process discussed in App. B.I, with 8 and 0, given as
B. Astart
() Leen
0, 1 1
(1-d)s
Bstart 0.1,
Bend 20 8 8= 10
D.2. Detailed description of datasets
We evaluate our model on six tabular datasets (https://att) ptttpp://archiveee ics. l.cs.ue .uci .edd/datssets). htt...cc.....ebbaaaaateeeeeeeeee Adult, Default, Shoppers, Magic, Beijing, and News. Beijing and News datasets are designed for regression task while the other four datasets for the classification task.
Table 7. Statistics for the tabular datasets.
Dataset
#Rows
#Numerical
#Categorical
#Training
#Test
Task
Adult
48,842
9
32,561
16,281
Classification
Default
30,000
14
11
27,000
3,000
Classification
Shoppers
12,330
10
8
11,097
1,233
Classification
Magic
19,019
10
1
17,117
1,902
Classification
Beijing
41,757
37,581
4.176
Regression
News
39,644
46
2
35,679
3,965
Regression
D.3. Model architecture and training details
The embedding for every numerical feature in the data is a summation of its type embedding and scale embedding. All numerical values share the same type embedding, which is a look-up table of size number of numerical features by the hidden dimension. Each numerical value is passed through 3-layer MLP that expands single numerical value to an embedding vector with the size of the hidden dimension. Categorical features in the data are individually embedded through list of look-up embedding tables. The look-up embedding table has the size of the number of categories 1 (with extra mask token) by the hidden dimension. Then all the categorical look-up embedding tables concatenated and treated the categorical embedding for this dataset. The building block of our model is adopted from DiT (Peebles & Xie, 2023). The sinusoidal timestep is passed through 2-layer MLP before input into the DiT blocks. After adding the integer positional embedding to the embedding, numerical embeddings and categorical embeddings are concatenated and input into DiT blocks. We used 4 DiT blocks with hidden dimension = 24 and number of heads = 4. The final layer splits the output into the numerical latent and a list individual categorical latent. The latent vectors are passed into 3-layer MLPs to obtain the corresponding scores.
The noise perturbation is the variance preserving (VP) SDE. The training loss weighted summation of the matching loss for numerical features and score entropy loss for categorical features. The weighting parameter is chosen to balance the numerical and discrete loss. The optimizer is Adam W with learning rate 10-3, weight decay = 0.03, 3 = (0.9, 0.9). A linear rate warm-up scheduler is used with warmup steps 200. The training batch size 2048. We used EMA model for final evaluation. During sampling, we use Euler method for the continuous diffusion and tau-leaping for the discrete diffusion.
26

---

Multimodal Diffusion on Arbitrary State Spaces
D.4. Evaluation
We compare our model with five most recent generative models that are specifically designed to operate on tabular data: GOOGLE (Liu et al., 2023), StaSy (Kim et al., 2022), TabDDPM (Kotelnikov et al., 2023), CoDi (Lee et al., 2023), and TABSYN (Zhang al., 2023). GOOGLE is a VAE-based method while the other four are all diffusion-based methods.
Shape is a metric computed via the Kolmogorov-Smirov Test between continuous distributions and Total Variation Distance between the probabilities for categorical values, measures, and compares the column-wise density between real and synthetic data.
Trend is metric that captures pair-wise column correlation by computing Pearson correlation for numerical columns, contingency similarity for categorical columns, and contingency similarity between bucketed numerical values and categorical values.
MLE is the testing accuracy of the classification or regression task on real data after training an XGBoost Classifier or an XGBoost Regressor on the synthetic tabular data. For detailed training and optimization pipelines of MLE metric, please refer to the standardized pipeline proposed in Zhang et al. (2023).
a- e--prcisionn evaluates if the synthetic data are from the same distribution as real-world data.
3-recall quantifies whether the synthetic data can cover the entire distribution of the real data.
Table Performance on the Shape metric percentage (%). Higher values indicate better performance, Best performance bold.
Second best in underline.
Methods
#Parameters
Adult
Default
Shoppers
Magic
Beijing
News
GOOGLE
5.6M
83.03
82.98
77.67
98.10
83.07
74.68
STaSy
10.3M
88.7110.0
94.23..0.5
90.63
93.7110.13
93,2910.s
93. 1
CoDi
25. OM
78.62 1006
84.23
68. 16..0
88.4410.26
83.06100m2
67.03
TabDDPM
11.7M
98.25 2003
98, 08.4
40.13
98.99
98. a0n
21.25 1.001
TABSYN
10.7M
99.42 2006
99.15=2004
98.57
99.1220.09
98.364004 98.88=005
TABSYN (reproduced)
10.7M
99.294406
97.12 +0.09
98.36
99.02++0.1
96
98.094003
Our model
64K
99.47
99.36=009
48.50
98.96*0.16
97.94
96.80
Table 9, Performance on the a-precision metric in percentage (%).
Higher values indicate better performance. Best performance in bold.
Sccond best in underline.
Methods
#Parameters
Adult
Default
Shoppers
Magic
Beijing
News
GOOGLE
5.6M
50.68
68.89
86.95
90.88
88.81
86.41
STaSy
10.3M
82.87
90.48
89.65.205
86.5620.9
89. 19.1620.1 12
94.76:033 33
CoDi
25.0M
77.58=045
82
94. 35
2030 2036
98. 13 38
87.15 12
TabDDPM
11.7M
96.361020
97.5911036
88.55
98.59
0.00
TABSYN
10.7M
99.52.010
99.261027
99. 19_1622222
99.38 =027
98.4 18.47
96.80 06.80
TABSYN (reproduced)
10.7M
99,32=022
95.574003
99.22...1
99.21 #027
98.87 20.15
96.30+o28
Our model
64K
99.47
99.47:021
98.781042
98.75
98.49
97.47:1027
Table 10. Performance the 3-recall metric in percentage (%).
Higher values indicate better performance. Best performance in bold.
Sccond best in underline.
Methods
#Parameters
Adult
Default
Shoppers
Magic
Beijing
News
GOOGLE
5.6M
8.80
14.38
9.79
9.88
19.87
2.03
STaSy
10.3M
29.21
39
37.24
53,97
54.7920 18
39.42+032
CoDi
25.0M
9.20=a15
19
20.8240m
50.56=a31
52. 19
34.40+030
TabDDPM
11.7M
47.05
47
47.79.005
48.462042
56.92:201
0.002a00
TABSYN
10.7M
47.56
38
48.95
48. 28.03
55.84 19
45.04
TABSYN (reproduced)
10.7M
47.75
42.9540.20
47.57
47. 47.92=028
49. 49.724127
44.37
Our model
~ 64K
49.65:0026
48.29
51.25
47. 47.662033
57.44:0.00
44.58
27

---

Multimodal Diffusion on Arbitrary State Spaces
5013: Sample Distribution
Distribution Rotation Nagnitades
14
Gecoraphical Distribution of Rotation Axes
12
Cender
12
044
02
120'9
0.0
05
1.0
1.5
2.0
Notzbonn Magretude tradian ans)))
Figure 5, Visual representation of ground truth labeled Riemannian data.
S0,3) Sample Distribution
Rotation Magnitudes
1.8
el stributtion of Rotation Axes
1.4
12-
327M
1.0
Dendty
20%
0.
2.0
2.5
3.0
Pocasion Maganude iradiandi
Figure 6. Peformance of inferring discrete label based SO(3) data.
E. Additional Experiment on Riemannian-Discret Multimodal Diffusion Model
In this section, we demonstrate another application our proposed multimodal diffusion model framework by focusing on the combination of Riemannian and discrete diffusion models on the state space M X. where M is a Riemannian manifold and X is finite state space. We will introduce the method and validate it on toy example consisting of synthetic data on 50(3) x X.
E.
Multimodal Diffusion Model
We consider the setting where the target data distribution Paata( y) is defined SO(3) x X, where SO(3) and y is discrete label in X. Since (3(3) is compact manifold, choose the following the forward process,
dX, = dB, Y, ~ CTMC(Q.)
(18)
(Xo, Yo) ~ Pdata (I, y)
M is a Brownian Motion on 5O(3). Note that the stationary where Q. = o.Qmmakk is the same design choice as in (15), d.B. distribution of (18) is Haar (So3)S)(3)) δM. where Haar Haar(SO(3)) is the Haar measure on SO(3), a generalized notion of uniform distribution. Following a similar derivation as presented in the paper, we can derive its backward process,
28

---

Multimodal Diffusion on Arbitrary State Spaces
dx, = Vlo Y.T - Ye ~ CTMC 8,
-) -) +d
(19)
where the gradient V is the Riemannian gradient, and (Xt,, T 1,T 8) is defined for + D.
p(x,8,T D.T t,T 5) 5) s
((I,t,))0,11) plr,u, 1,
Axis-angle parametrization. We represent elements of 50(3) using the axis-angle parametrization. We introduce it briefly here. One can show that any element of (3)(3) can be written as exp(@K)) where:
0
0
0
K=a
0 1
+ b
0
0
0
1
0
0
0
0
0
0
0
0
0
and (a, b) E S is vector on the sphere, 0 E R4. The representation ((a, b, c). 0) is called the axis-angle representation.
Dataset of the toy problem We consider simple toy example of labeled data on SO(3), consisting of Gaussian mixtures, where each mode corresponds to a unique label. To create the problem, we write elements (a, b, c) S in spherical coordinates; in this way, only two angles need be parameterized. We then generate Gaussian mixture on the space of these angles. Additionally, dataset in Listing 1 and visualization of the axis and angles in Figure 5.
von Mises random variable for 6. We present the Python code used to generate the
As observed in Fig. 5, we have assigned labels to different geographical locations and assigned them to distinct modes on the map.
Training strategy. We train simple MLP using a similar strategy as the text text-image model. We first train a label to 50(3) model and add the discrete capabilities in second phase. To achieve this, we utilize the generalized denoising score matching loss IGDSM, described in the main paper, which is derived from the generator computed based on the chosen forward process. We find that this training strategy is generally robust.
Results. We present samples generated by our method using guidance w Figure 6, we see that our method can properly recover the data distribution. We also show the unconditional generation in Figure 7. We demonstrate that our method and training strategy generalize to other data modalities.
50:31 Semple Distribution
Distr Dustinbution Rocation Magnitudes
Geographica call Distribution
14:
50%
12
1.2
1240
04
1607
170'
60/W
60FF
100/
100
0.0
Rozatoon Magritude (adian)
Figure 7. Performance of joint generation of the labeled Ricmannian data.
29

---

Multimodal Diffusion on Arbitrary State Spaces
def sample_cont contine 903 (n samples, maanittude_kppa-999) {
NNrrthAmmrrca': ('lat': 45, "lon': -100, 'weight's 0.2), 'South_America', ('lat'; -20, 'lon's -60, 'weight': 0.15),
5
'Europe': ('lat': 50, 'lon': 10, 'weight's 0.2),
6
'Africa': ('lat': 0, 'lon': 20, 'weight': 0.15),
'Asia': ('lat': 45, "lon': 90, 'weight': 0.2),
'Australia'; ('lat'; -25, 'lon': 135, 'weight'; 0.1)
10
weights = ar.array ( [[cnt ['weight'] for cont in cont waw.ettttaeeslleee......aa continent_chh ces np. aaandmmcchoiae (len (conti nents), si size-n_samples, p-weights) rotation_vactors ne.zeros (n_samples) 3))
15
continent 1ist 1ist (continents values ()) for i in range (n samples)
continent continent_list [
theta pp/p np. np. (con ['lat'])
19
phi np. np.deg2rac (cont lon')
theta += np. random. normal I (0, 0.2)
phi np. np. random. 1mp.ramomm.nozaal4/4 (0, 0.2)
24
axis = np. ap.array([ ([
si.sin (theta) cp.cos (phi), sp.sin (theta) sih
co. (theta)
})
magnitude vonmises. rvs (magnitude kappa, Loc=np.pi/
rotation vectors oration_vectors [i] axis magnitude
return rotation_vectors, cont wwwiiinnnttticcc
Listing I. Code generating dataset
F. Additional Numerical Results for Text-Image Generation
CLIP Similarity We generate 5000 samples and evaluate the CLIP similarity between the text and image. For this evaluation, we use CLIP- CLIP--iT-larre-patchl and we limit the captions to 77 tokens. We use our sampling default values during this task. We obtain a CLIP score of 18.46 for text-to-image generation, 17.44 for image-to-text generation, and 17.57 for image-text joint generation.
Generated examples visualization We display non-cherry-pice -ickrng generated examples all three scenarios Fig. Fig. Fig. 10.
30

---

Multimodal Diffusion on Arbitrary State Spaces
image festures an old building with wall and stone brick building a ourrounded by a kesh green fill the stone wall is located near one another , and there are two trees neartyy the style of the image is black and white which gives a timeleess and classic teel the black and white color polette adds seree of emphosizing
image features large fresh produce including vanety of wateriiee fraits vegetables, arranged on top of each other on a table the hand mace trutt are dispaayed in a visually appesting manner with different mix of colors and natural texuures the fruits are well maintained and spocar for creating aesthetically pleasing and inwiing protection lisely
building sitaated
nostalga elegance to archilecturall details the lendscage
used provide elements maintain
contempurary look, help the sunilight enhance appeal and an engaging visual experfence
several people surfboards working large body surruunded by boats surtboards. fishing players soaring thouugh the water creating enjoaabee waterway to aoard wings against the sun. the style of the image a black and white photo. which adds timelass and classic feal the scene. the tlick and while color schemne elso emphasiees the baach, drawing amontion thir interact ons being nut. subjects
image
image fnatures lush green fountain middle surounded greer ery.... including flowerss and trees, park is situated front of store wall and There a bench in the center ot the grassy held the style of the image is black and while phocoo which gives it and classic and timeless appearance. the contrast benwean the vibrent green surounndngg and sandy fountein along visualy
large
the players
and the dynamic
the black and weite style appailing and artistic represertation park
the mmge foaturas a la ga graen commarcial arplare, possibly jumbo jet, flying through the sky. the airllane seen from a lov angle.
the features large, ornately natartistic designed mosque with gold celling decorative
comed desig intricate teatu details. the mosque se is adomed with decoration, includ rig lerge chardeller on the masalc, which adds to the averal grandeur the space the style of the mosque a blend of tracit onal and modam levels elements,
emphocapturing unique size. bird filled with doudss
against the clouds styte airplane flying image artistic and visually appealing. with combination the arplare 's shape and color scheme against the sky of the sumrundinn clouds creating a visuaily strinngg and evocative scene. the image likely taken with its avlation photography shovcasing the heaury and elegaroe of skylne, botanical sirp that
contemporery design. reverence splinuality reflacting history cultural nature the moeque..
strospheree the image
the imae features group of soldiers miltary lnigues driving down grasey field they wearing and
the image features group people in canoe, flosingg down river there lake. the bost flledd passangers. on riverbank, scene is captured and white giving image a vintage or fashioned appesrance. the people the rait are wearing tracit onal clorring and same of them are wear ng hooded. suggesting thet the weather trip is not to the everyday life matter the overall style the image ulend, real realetic and nostalgic atmospheree eapauring could captures the essence intsurety
waspons, which inticately designed displays vehiiees the style the image Iikey to be captured black and white which gives it classic and simeless feel. the soliers are all pasitioned in a row at slowaasing thii gud by their tearmwork and unity.... the image captures the essence a miltary parade. where they come together share their touch with each another.
people
the image festures younng boy sitting bost lush green shoreline hokinn long wooden stick. the boy is wearing a bllck si in and jear's is and he sppears to be sitting on the bench. the style of the image is black and white photoo, which gives it a diassic and timeless leel the compositon the photo adds serise mostelgis and emphesices the locus the interlabel boy fecusing king rally on the bost comteppi the sunouncinn anvironment.
the image featur es colortul scone buidings stuated along the weterfront each with a pirk door inclucing there e several buikings s stated alone beat. the buildigg have pink and white timm which adds to the vibrant and cohesive appearance of appearance the combination of minimation vibrant perspective, which gives it unique and antisic appearance monschramai e style further enha enhances the v sual appeal of the scene, capturnn the essence of the bulkings and their sur mundiigg in the atherwise manochromatic table. the photo used artist is cholce adds depch and interest
the mage lieely pair ting
the image festures harbor at the dier, with a sandy beach nearty. is ined with boats, including a row of small boass with a cannon, the lighthouse is interest was grand attention to the water water crystal clear making it visually sopea ing if hes been interacting with it. the beach supears
boats docked
image features large, brick building tall cloc lower, situated next a city street, the huiding has a unique intricate architectural design. featuring a tall steeple and a clack lover. the steepsias on is surrcunded by street aamps and bench can placed along the sicewelk. the scene outdours with sunligtt. illumissinn the iluminating surooundir ge image black white. which adds a sense of timelessness classic train the overall compositio of the image showcases the architestural beauty of the old and new, showsasing the beauty and charm of the citycappe
with rocks adding potird plants elements to scene. sky is pleasant, and the overall aomosphere here of the image, creating mix natural beauty and serenee activiiees
around
Figure 8, Visualization of texts generated conditioning on the images.
31

---

Multimodal Diffusion on Arbitrary State Spaces
the mage features long pler bridge with people, some of whom are hokding umbrellas elass the scene taves ptace in a city, with a large builcing visible in the background aundd the styla of the image is black and while which gives a classc ane timeless feal. the composition the image. with the seople and ciyscape.
image features man standing pot filed with food. whinh cooking on a stove the man is wearing a white shirt and a black hat, and he is surrounded by var warkous cooking utensils such as a spoon and bovil the scone set in kitchnn and the style of the mage is black and white giving it classic timeless tee the composttio
a large
movemment and color paletto adds sonsie nostaigla and evokes loeling the pazt. while the people and the citscapee convey sense of modern urban ltfe
actilly
image create a sense of warmth and familanity the viewer witnessing a cher cherlshed family tracittion cherashed memory..
expression
wf floeiing on large body water, as
image
imege feature es large, open, greesy with large building in backaround butding appears to be a stadium or a large event venue possibly sports arens or convertion center. the area is filld with people some of of om are walking around, while others are eiting on benches. the style of the image is black and white which gives a color
ocean with a person standing on it. the bost apoeors to be a small motorboatt possibiy speedbuat, as is desc ieed as a speed bost in the image the bost is docked a mar na, and there are several other boacs in the vieinity the style of the image a black and white photo whicch givess a clessic and timeless appearan
classic timeless feel the clack and scheme sense rosta and elegance to scene emphesicong the architecurs elements of the large building and the people in the area.
bowt and the person standiing on it, capturing the eesence a
of the image
the image features a train station with a long platform and a wann parked at the end at the train a selver and red commuter trinn, and is surroundert by yellow black str platfom. the train station appears busy and busting location. with several people waiking around and faws cars parked nearbyy the style of the image a black and white photo which gives a classic and timeless feel. the compesition.c of the image, with the train and the platforn as the main subjects
image features lush green garden with fnuntain the middle plants and bushes. the parden located near sions building which courd be house or historical site the garden is well maintained with nestly trimmd bushas and plants. creating a serone and inviting atmosphere. the style of the mage is black and white photo which adds a timeless and classic touch the the cominnation the greenery. fountzin. hamonious and visually appeeling environment.
varisty of
gray overnead
building
the image festures mountein road night. with winding road and a lit road sign. the road sky
the image is black and white photograph a city background the style with tall builiing classic ntage of image photograshty manachromatic color sel weme and the use cor trettest to emphasize the shapes, textures, and lines within the soonoo the black and white oolor paleme creates timeless and clessic simosphere, sllowing the viewer to focus on the architecural ol details, auch as the tell builing and street, without the distraction of color. style photography ofton used eapture essence a etyscape evoke
by forest a dramanic and visually sppeeling stmosphere style the imeege witistic and captures the essence the winding mountain road night. with the focus on the road. the si smouundee nding forest, and the it- up read sign. the darinnes of the sky and the contrast betwean the lit road and the surrounding darkness create captivating and visualy striking scane.
creating
char aneracteeized by
the image festures sidewalk, tike parked side, hench. there several people walking and sicycle is also present. the streel sppeers s to be lined with buidings, and there are several potted plants and mon on the sidewal, adtinng in tha cherm of the scene. the style of the mage is track and white photograph. which classic feel. the compostion the image with the paople. bike, sensee everyday urben the simple
with arick
the image depicts a busy highwey Intersaction with several cars and trucks stapped a tall booth the toll booth a large sirucaue ne with a blue and white color scheme, which stonds out against the backdrup of the sky. there are multple cars and trucks stopped at tall booth. and the traftic lights also visible in the a classic and time reminiscent dassic black and while photography. which emphasizes the contras: between light and shadow. , and the overall compasiton of the soene..
sideaak,
image black
glving image
a timeless and
style
potted
image features large, ogen. paved area tall motel pole the middle the pole is topped with a red cress, which is a symbol remembrance the area is sur surcounded by trees givir g a serer and peaceful aomosphere.... the styte the image tlack and white which adds timeless and clessic feel to the scene the black and white color scheme a emphesizes contrast surrounding drawing attention the memorial and its sigrifioanoe.
image depicts a beach scene with sandy shoreline, where e g group of people is erjoying their
time. there are several indiviuals present. including
man and boy. who are playing in the water. the
beach appears to be crowded, with many people
engeging varioss actioties such sunbathing
and socializing the image timeless feel. the
black ard
g
whis. which gives classic overall the image is nostalgic and evokes
serse lessuee relocction, as captures the essence of a typical beach day. .
Figure 9. Visualization of images generated conditioning on the text caption.
32

---

Multimodal Diffusion on Arbitrary State Spaces
the mage features small green nut brighmees shelter with door silusted in lush green fieid the hut is surruunded by a vanety of troes, huts, whith are for cammuting contes. the hut has natural envirenment the setting is designed resemble a vilage or tradtional ssian village with a peaceful and
image features row statues or adults holding up hands which are placed on dir road in a city. the statues are postioedd a way that sugpests they are a in a rew, making they easil accessiblee and intricate diplay. the stan es are made of stone and have white color scheme, which contracts with surounccnng drawing
image
the
which adds timelass and classic feal the the style the image antistic evocative. cappurig ng the essence traditionll rural lffeclyle and the simple wuthn the natural envronment.
statues lieely by traitionaal art figure. they are part a cuture event. the image eapoures res the beauty and uniqueness of the human hands and statu es showsassin their artssic significance and the skil
image festlures es snow covered field with large open covered sheber visitors sheiter is sur surrounded by ience, and there are several fatils and chairs set up outsice the shelter, the scene set in winter color, suggeoligg that the image was taken d daring the winter season. the snow covered lancscape and the sunounding trees create picturesque
inege featur es lage orange perge car per parked room, which cornected to played large air airplane. the cars are parked on red carpet and surrounded by lrees, giving the room a forest the secting the car appears to be historical velicle possibly a mu seum maintained inte the car design and style suggest thest these aniquue vehicles characterized by a
large sirplene, and well -
osgres making
from typical winter snowy onmern. the image captures res moment vinter, with people enjoying ring the cold weather st showessing the shapes and textures the snowy landscapee.
nostalgia sophisticaion, scene is hall modern style and well - Iit Hignights the vehicles a stylized and
the image features a large, ornate and anciett- looking builcing resembligg a temple or paizce. the tuiking is situnted on street and is surrounded by brick architecture very detaled intricate. suggesting that might be a aggfficant cultural landmank holds sha onlored imprrtance the of gold- colorad roof and intricane pattoms adds the build ng 's authenticity and grand appearance the image is ta ken durng the day. /ith the sun casting warm g nou en the scane.
features wall, which Incated archway the archway filled with people. some whom are walking around the area. the soine is set on sunny day with the sunlight stream ng in the sunlight illmmnasing the scano. the styln of the image a black and white photo. wrich adiss a timeless and classic feel to the scane the compostion of the beauty of sunlight.
the building
emphasizes the
archway and the creating appaaling and momorable experence.
the image festures street sign mouniee sole, which located park
the image decicts a busting outdoor market mar ket where several people walking around, possibly shroping for vegetables sceree interacting with each other. shopping and some of them are linn on the ground the meriz filed with various iters such as vegetabless. which s suggests that they contrittee to the lively atmosphere and inspire people to compete and sed. the image is in Black and write, giving it dlassic and timeless appeerance the people the image engaged disc assing mai produce. possioly discuesing examining g them fresh, purchasing homs, or simply
the sign writlen
black Inclcating
place for
individuuass
people leemn park. the sign is placed a bree. wrich ackss clessic ebement the park the style the image black and white, wh ch gives a timeinss and dlassic feal. the monoehorsult color scheme adds sersee of nostaagia and
BPGLL headhh HiE
ease Nho brnur
h ghiigghs the contrest betwein the and the humorous sicycle rider. the sign posilione park seiting creating
square
tranquiltyy
the festures man wearing blue has red hat, warking
the image depicts a busy street soane with large crowd of people gatered around. market with vai iity of temm for seling fowers. the market is situated on a brick road. giving at quaint and charning atmosphere the verdios are filldd with diferentt flower
stonstiont hh holding
book. possibly music, and appears or walking down the street. the scene sel in european city, with the man dressed casual ditthing. and the storefront is aumou indedd by several signs and which might provide information.. the oversil style the mege a black and while classic timeless
reading
inc includigg fruits
vegetables, flowers display. market buctiing activity both place and shot multipic shoppers the overall style the image a close up. tocusing on the details and the abundance and intoraction ns that take place in the world.
photograph which faall the choice black white adits composition cepth emotiin the sceree making more engaging and reistable
the image festures large. ilkaminateed christmas tree displyy. surrounded by lighis and decorations the tree is a public square and is surounded by a city skyfne v viible in the background the tree made of lights and there are multitle chr estmas trees, which waste fesst atmosphere and busting stmosphere, the image in black and white which gives
image features large statue person. possibly man, standing on tall in the shade over fores: the statue is sunounded by trees, creating ser ene and picturesque setting. the statue made of intricate carvings and spoears to be a pari of historical haidingg as it described as a staase the style of mage is artistic and evocative eapturing the beauty and composition the mythical
and limeless feel, empl wiiizing the e legaaro de of scere.
nature the statue create a visunily appealing and cxpelvittig atmosphere. invitng the siewer to sppreciate the detalls and mejesty of the scene.
surounding enwiromment
Figure 10. Visualization of text-image pairs generated jointly and unconditionally.
33


Reviews:
[{"review_id": "MGblgJnFhB", "reviewer": "Reviewer_QDhM", "summary": "In this paper, the authors focus on the problem of using diffusion to model multi-modal data domains, especially text and image data. To this end, they propose a novel approach to the noise schedule which is distinct for each modality. They justify their approach theoretically with proofs. They then evaluate their approach empirically on joint text-image generation and tabular data domains. They show quantitatively that their approach is an improvement over baselines on the tabular data.\n\n## update after rebuttal\nAlthough some of the reviewers had some concerns, I think this paper has some potential contributions. I keep my assessment of weak accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and valuable insights. \n### Regarding Generality  of the set up\n\nOur work built on the general set up of [1] which includes at least Euclidean, discrete, Riemannian and Wright-Fischer diffusions. We improve upon it by leveraging separate noise levels on different modalities which inmediately allows for combinations of modalities while leveraging joint, single-modality and conditional generation in a single model. For this reason our approach generalizes to at least all processes covered by [1]. \n### Regarding the evaluation on other domains\n\nWe must emphasize that our work focuses on introducing a general methodology for dealing with multimodal diffusions. Both the tasks of text+image and tabular data are challenging tasks that require lots of engineering and a single of these tasks is usually the study of a single work. Moreover, the audio-video example is not the best use case to deliver our point as both modalities are represented in continuous values, which in theory can be tackled with a unimodal continuous diffusion model. Therefore, we didn't attempt to do this example. \n\nTo further highlight that we are proposing a robust framework of both pratical and theoretical importance, we have included a new example regarding a toy problem that includes Riemannian + Discrete data to demonstrate the versatility of our method in other setups.  This can be found in the following annoymized repo (https://anonymous.4open.science/r/ICML-Rebuttal-FD00/Rebuttal.pdf)\n\n#### References\n\n[1] Benton, Joe, et al. \"From denoising diffusions to denoising markov models.\" Journal of the Royal Statistical Society Series B: Statistical Methodology 86.2 (2024): 286-301.", "claims_and_evidence": "Yes, to some extent. The authors justify their approach on theoretical grounds, and they show promising results on image-text and tabular data. However, it isn't clear if the results are general enough to hold to other multi-modal data domains.", "methods_and_evaluation_criteria": "The methods are sound, and the evaluation criteria are appropriate.", "theoretical_claims": "The theoretical claims appear to be correct albeit not all details were checked.", "experimental_designs_or_analyses": "While the experimental design is sound, the breadth of evaluation is narrow. The proposed approach is quite general, but the only domains that are evaluated are image/text data and tabular domains.", "relation_to_broader_scientific_literature": "Compared to prior ideas, the authors propose a novel approach to diffusion on multi-modal data, specifying a noise schedule that is separate for each modality.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "The approach is novel and shows promise, but the scope of evaluation is limited to specific domains. The argument of the paper could be strengthened with evaluations on other multi-modal domains (such as video + audio).", "other_comments_or_suggestions": "None.", "questions_for_authors": "Given the general nature of the approach the choice of evaluation datasets (images/text, tabular data) is quite constrained. Would it be possible to evaluate on other multi-modal domains (i.e. video + audio)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "juggvmEUv1", "reviewer": "Reviewer_g74r", "summary": "This paper proposes a novel diffusion-based framework for both continuous and discrete multimodal data (images and text). Specifically, for continuous image modality, the paper utilizes diffusion process as forward and backward process. For the discrete text modality, the paper uses CTMC to determine the state of text tokens. The two modalities are unified through a single diffusion process, but using different forward and backward pass formulas, with the allowance of using an asynchronous timestep. For experiments, the authors evaluate the performance of the model on trend, efficiency, shape, precision, and recall. The proposed method achieves competitive results with other models, while using a much smaller model size.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and valuable insights. \n\n### Regarding evaluation benchmarks\n\nWe'd like to emphasize that the objective of our work is to introduce a general framework for training multimodal generative models using diffusion, as opposed to being considered a task specific method.\n\nOur work is slightly different from Transfusion, Show-O, etc. These models are capable of multimodal understanding through instruction finetuning, whereas our model does not. Therefore, we do not evaluate on benchmarks that requires understanding.\n\nAs for the text-to-image generation benchmark, while GenEval, DPG-Bench are also popular choices, we found that MS-COCO is more suitable since it has been selected for evaluation for all most all models with parameters ranging from million to billion levels. We want to emphasize that one major difference between our models and most of the others on the table is that our is small while others is usually of billion order parameters. To accommodate for the parameter differences, we eventually selected MS-COCO and do not go with other larger benchmarks.\n### Regarding T2I performance\n\nWe want to first emphasize that our work focuses on training for the next generation of multimodal diffusion models for **multiple tasks** rather than a single one, which in general is much more challenging. Despite the small model size we use, when compared to other **multi task** models (with bigger size and more computational resources) like Chameleon or JetFormer, we can still achieve better results on the MS-COCO text to image task. For this reason we respectfully disagree that the text-to-image generation capability is not proved effective.\n\nRegarding the comparison to other solely T2I models listed in table such as MMDiT-improved, we want to remark that we listed these results to provide a view of top performing **single task** model in the literature. As we have explained before, a direct comparison with MMDiT-improved would be unfair due to the huge difference in task difficulty. Despite the overall more challenging problem, we still achieves a decent T2I generation quality, indicated both by the FID score and a high visual quality (see more examples in Appendix D). Therefore, we believe the achieved T2I result should be understood as a **merit** rather than a weakness.\n### Regarding Tabular performance\n\nOur approach does show superior performance in multiple tasks on tabular data synthesis than other models while using a **significantly smaller model** as shown in Table 2, which we believe should be acknowledged as a major improvement over previous works.\n### Regarding technical novelty\n\nIt is true that similar ideas have been proposed before as we have acknowledged. These works use ad-hoc concatenations of diffusion models to create powerful generative models. However, these findings are not based on solid theoretical justification as to why this is a valid thing to do. In our work, we present a general framework by extending [2] and using the per-modality generators to get valid loss functions and obtaining valid processes that preserve the distributions, confirming the findings from [1]. Additionally our framework allows not only works like [1] but exploration on other domains with other kinds of data. \n### Regarding architecture choice for text-image\n\nWe adopt this architecture of combined MM-DiT blocks and DiT blocks for the staged training strategy discussed in Appendix B.2. Our architecture design allows an effective implementation of this training strategy. To demonstrate the importance of it, as well as the modular set up of our network, we train a network composed of only MMDiT blocks and train on both tasks at once. This serves as an ablation on the architecture and training strategy. We use a network of similar size and use the same hyperparmaters as in stage 1 in Table 4. This trained model achieves an FID of 73 on the text to image MS-COCO task and a CLIP score of 9.07 on the joint generation set up. This shows that naively using all MM-DiT blocks results in terrible results and that a staged training process is required as demonstrated by our improved results. \n### Regarding use of additional text encoders\n\nMost text to image diffusion models make use of a CLIP encoder for the text. However our model also receives as input text that has been corrupted by a masking process. CLIP and other text encoders are not trained on text of this form. For this reason we decided to omit the text encoder from our  network and don't have such a baseline.\n\n#### References\n\n[1] Campbell, A., Yim, J., Barzilay, R., Rainforth, T., and Jaakkola, T. Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design. arXiv preprint arXiv:2402.04997, 2024.\n\n[2] Benton, Joe, et al. \"From denoising diffusions to denoising markov models.\" Journal of the Royal Statistical Society Series B: Statistical Methodology 86.2 (2024): 286-301.", "claims_and_evidence": "The claims in this paper, including competitive performance on text-image generation and mixed-type tabular data synthesis are supported by experiments on corresponding evaluation datasets.", "methods_and_evaluation_criteria": "The evaluation benchmark and baselines seems insufficient here. Since this work is in the same line of work with Transfusion, Show-o, JanusPro, etc., it would be useful to do evaluation on GenEval, DPG-Bench for text-to-image generation, and POPE, MME-P, MMB, SEED, GQA, MMMU, MM-Vet, etc. for image understanding.", "theoretical_claims": "I didn't check the theoretical proofs carefully.", "experimental_designs_or_analyses": "The experiments on text-image generation and mixed-type tabular data synthesis are valid, but the performance is not superior than other models.", "relation_to_broader_scientific_literature": "This work is related to unified model for multimodal generation and understanding, including Show-o, Janus series, Transfusion, etc. These prior works mainly focus on designing a unified model architecture for modality fusion, while usually apply separate training objectives for text and images. In contrast, this work focus on exploring using a unified diffusion objective for both modalities.", "essential_references_not_discussed": "Related works are well-discussed.", "other_strengths_and_weaknesses": "Major Strength: \n\n- The idea of creating a unified model that uses discrete diffusion for text and continuous diffusion for images is interesting.\n\n\nMajor Weakness:\n\n- Limited technical novelty: The overall idea of this paper is very similar with the literature for protein prediction [1], which also uses a unified diffusion model for discrete and continuous tokens. Specifically, both work uses diffusion process for continuous tokens, and CTMC for discrete tokens. Ideas including asynchronous timestep is also first proposed in that paper. In addition, the authors also uses MM-DiT block proposed in Stable Diffusion 3. Therefore, the overall novelty in this paper seems limited.\n\n- Weak result for image generation: The FID score in Table 1 is worse than most baselines, including MMDiT-improved, which has been trained on the same number of images and with comparable trainable parameters. Therefore, the the text-to-image generation capability is not proved effective.\n\n[1] Campbell, A., Yim, J., Barzilay, R., Rainforth, T., and Jaakkola, T. Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design. arXiv preprint arXiv:2402.04997, 2024.", "other_comments_or_suggestions": "NA", "questions_for_authors": "Some questions about missing ablation experiments: \n\n- Why not apply MM-DiT for all blocks? Ablation experiments on model architecture would be helpful. \n\n- The paper didn't use text encoder, but encode the text into learnable embeddings directly. Could the authors provide some comparison experiments about using and not using text encoder?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZhUzxAMojc", "reviewer": "Reviewer_R8im", "summary": "- The paper introduces a framework for multimodal diffusion models on arbitrary state spaces via independent noise schedules for each modality.\n- In particular, the paper proposes a theoretically grounded framework for multimodal diffusion of continuous as well as discrete state spaces.\n- After training, the diffusion model can be used for unconditional joint generation of multiple modalities or a single modality conditioned on another one.\n- An evaluation for text-to-image generation using a multi-modal DiT (MMDiT) architecture and mixed-tabular data synthesis shows competetitive performance compared to baselines despite smaller model sizes.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and valuable insights. \n### Regarding T2I performance\n\nWe want to emphasize that our work focuses on training for the next generation of multimodal diffusion models for **multiple tasks** rather than a single one, which in general is much more challenging. Despite the smaller model, when compared to other **multi task** models like Chameleon or JetFormer (which are commerical-grade), we can still achieve better results on the MS-COCO text to image task.\n\nThe comparison to T2I models such as MMDiT-improved and PixArt-alpha is included to provide a view of **single task** SOTA models. A direct comparison is unfair due to the huge difference in task difficulty. Despite this, we achieve a decent T2I generation quality, indicated both by the FID score and a high visual quality (samples in Appendix D).\n### Regarding VAEs\n\nWe apologize about the confusion. We agree that in image diffusion models VAEs are used to get a compressed latent space. We were trying to state that in the case of multimodal models, encoders are used to set data into the same space with techniques such as VQ-VAE or VQ-GAN. For example, images can be tokenized, changing from a continuous state space to a discrete state space, where we get compatibility with the text modality. In our work we would like to present an approach that can treat each data modality in its native state space. \n\n### Comparison to UniDiffuser\n\nWe thank the reviewer for pointing us to this reference, we will include it in the revision. UniDiffuser uses the similar idea of introducing separate noise levels for different modalities. However they encode the continuous image and discrete text into continuous embeddings. They then train a continuous diffusion model to generate the latent emmbeddings. After using proper decoders they can recover the image and text. This technique is of the kind defined in the previous point, where both modalities are moved into the same state space and the generation is performed there.\n\nDifferent from UniDiffuser, we propose treating each modality in its native space. We treat images as a continuous object and text as a discrete one. We do so by leveraging advances in continuous and discrete diffusion and introducing a novel framework that allows combining different diffusion models together.\n### Regarding errors on Zoom-in plot\n\nWe sincerely apologize for the confusion and thank you for catching this. Indeed, the colors of the zoom-in are incorrect, there should be a change of colors with the $w=3$ and $w=5$ curves. We will correct this in further revisions. \n### Regarding use of extra encoders\n\nHere we meant the encoders to change data from one space to another as described above. As well as encoders like CLIP which are usually used in T2I diffusion models. Which encode text into a latent vector which is used to condition the diffusion model. Instead, we directly apply a learnable embedding to the text (with nn.Embedding) and learn its own hidden representations. \n\nWe apologize since we had a typo in our table, we did not train in the pixel space. We used the usual stable diffusion VAE to train the continuous component in the latent space (for dimension compression). As expressed in other parts of our paper (as you described), we trained in the latent space for images. \n### Regarding tabular evaluation metrics\n \nThe metrics used in tabular data experiments are described in Appendix C.3. For better readability, we will also add more descriptions of these metrics in the main text in the updated version.\n### Regarding improvement on tabular data\n\nWe believe that the reduction in model size is due to two major reasons:\n1) We operate on the native state space for mixed type tabular data and don't use any complicated encoders for embedding each modality. This reduces the model size as encoders, such as VAE, can be parameter-heavy. \n2) Based on point 1, we designed a new score network based on transformers that directly takes a mixed-type tabular data point as input. Our score network adopts the idea of early-fusion (also discussed and adopted in Chameleon), which learns a joint embedding between modalities starting from the first attention layer. We suspect that early-fusion approaches are more effective and parameter-efficient. \nWe will add more discussion related to this in the updated version for better readability.\n\n### Regarding Section 3.1\n\nSection 3.1 discusses a statistical and theoretical framework of diffusion models by drawing connections to generalized score-matching objectives. While the connection is proposed in prior work (which we have clearly indicated), we leverage this perspective to derive a framework for multi-modal diffusion models of both practical and theoretical importance. For ease of understanding and notation consistency, we kept this content in Section 3.\n\n### Regarding reference to MM-DiT architecture plot \n\nWe will gladly add the citation as described.", "claims_and_evidence": "All claims are supported by convincing evidence except for:\n- The authors claim that on text-to-image generation they \"achieve similar performance as commercial-grade models using a small model without leveraging powerful extra encoders\" (lines 100 ff., right column).\n  - The perfomance compared to MMDiT-improved with the same number of training images and a similar-sized model or PixArt-alpha XL/2 with also a similar-sized model but double the number of training images is significantly worse.\n  - It is unclear how the proposed method does not use extra encoders.", "methods_and_evaluation_criteria": "The proposed methods make sense except for:\n- The paper claims that it does not make use of any extra encoders (also blank cell in table 1).\n  - I do not understand how this is done. Does it mean that diffusion of images is done on pixel level? But then the authors mention \"noisy latents\" in line 310, right column and the MMDiT architecture is not built for pixel-space diffusion.  Appendix B.2 mentions a joint embedding and continuous decoder trained for text-image alignment but this whole architecture remains unclear.\n\nThe used evaluation criteria follow standard setups and make sense.", "theoretical_claims": "I did not check the proofs of all theoretical claims provided in Appendix A.", "experimental_designs_or_analyses": "All experimental designs and analyses seem to be valid.", "relation_to_broader_scientific_literature": "There has been a lot of research on diffusion models for continuous spaces in recent years. Recently, diffusion models for discrete spaces gained attention. There have papers proposing unified perspectives like denoising Markov models [1] as mentioned by this paper in lines 149, right column. This paper focuses on the formal derivation of multimodal diffusion with individual noise levels for each modality. Experiments for text-to-image generation leverage the MMDiT architecture introduced by previous work StableDiffusion 3 [2].\n\n- [1] From Denoising Diffusions to Denoising Markov Models. ournal of the Royal Statistical Society Series B: Statistical Methodology. 2024\n- [2] Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. ICML 2024", "essential_references_not_discussed": "There is an essential related work that has not been discussed:\n- UniDiffuser [1] proposes a multi-modal transformer-based diffusion model with independent noise levels / timesteps for each modality with applications to text-to-image, image-to-text, and joint text- & image-generation as well as discussion of enabling classifier-free guidance. Therefore, the core idea of this paper is not novel anymore.\n\n[1] One Transformer Fits All Distributions in Multi-Modal Diffusion at Scale. ICML 2024", "other_strengths_and_weaknesses": "Strengths:\n- The paper introduction motivates the advantages of continuous and discrete diffusion and the potential of multi-modal diffusion well.\n- The proposed noisy guidance in section 3.4. renders an additional interesting technical contribution.\n- The flexibility of the framework enabling unconditional multimodal generation as well as conditional single-modality generation given other modalities is elegant.\n- The experimental results for tabular data synthesis using a much smaller model than baselines are promissing.\n\nWeaknesses:\n- Lack of clarity:\n  - The introduction states that VAEs are mainly used for mapping different modalities into a single modality for diffusion. However, the main motivation of using VAEs is to have a compressed latent space for more efficient diffusion model training as introduced by the original latent diffusion paper.\n  - In figure 3, the zoom-in seems to contradict with the rest of the plot.\n  - As stated above, it remains unclear how the proposed method does not make use of any extra encoders.\n  - A short description of the metrics used for evaluation of tabular data synthesis would be helpful.\n  - It is unclear why the proposed approach is better than all baselines for tabular data synthesis despite using a so much (100 to 200 times) smaller model. What are the key differences?\n- Section 3.1. is not very clear about whether the unified perspective is from prior work, in which case it should be part of preliminaries, or whether it is something novel.\n- The used MMDiT architecture as outlined in figure 2 is largely inspired by StableDiffusion 3 and therefore it should be cited in the figure and the architecture description (first paragraph of section 4.1).", "other_comments_or_suggestions": "No other comments or suggestions", "questions_for_authors": "1. Could you comment on the main differences compared to UniDiffuser (see review section Essential References Not Discussed)?\n  - Since this work basically proposed the same approach, a discussion of the key differences is necessary to understand the novelty of this paper.\n2. How exactly does the used model not rely on extra encoders? Could you please specify the architecture (joint embedding and continuous decoder mentioned in appendix) and how all components are connected?\n  - This is essential to resolve the lack of clarity w.r.t. one of the key differences compared to baselines.\n3. It is unclear why the proposed approach is better than all baselines for tabular data synthesis despite using a so much (100 to 200 times) smaller model. What are the key differences?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors provide a method and derivations for performing joint diffusion on multiple modalities. The paper received mixed scores after the first review round. The reviewers highlighted the general paradigm and the technical contributions in the framework as positive. Concerns were raised regarding novelty of the individual contributions, clarity and result quality. While most concerns could be addressed in the rebuttal, concerns about novelty remained after discussion with the authors.
Specifically, there is a previous work (One Transformer Fits All Distributions in Multi-Modal Diffusion at Scale. ICML 2024), which the authors missed in their paper and which is a strongly related and crucial work. During the rebuttal, the authors promised to add and discuss it to the draft, and also were able to outline how their method significantly contributes on top of it. In the end, the discussion convinced two out of three reviewers to vote for accepting the paper.

I will follow with a (weak) accept recommendation. I agree that there is novelty in this work and that the technical and theoretical framework can be valuable for future work. At the same time, the overall novelty is slightly dampened by the mentioned publication.
If the paper gets accepted in the end, the authors should add an in-depth discussion of the mentioned previous work to the paper.
Decision: Accept (poster)