A Unification of Discrete, Gaussian, and Simplicial Diffusion
Authors: Nuria Alina Chandra
Yucen Lily Li
Alan Nawzad Amin
Alex Ali
Joshua Rollins
Sebastian W. Ober
Aniruddh Raghu
Andrew Gordon Wilson
Venue: ICLR 2026 Poster (2026)
Keywords: discrete diffusion
simplicial diffusion
gaussian diffusion
generative models
proteins
dna
Primary area: generative models
TL;DR: We unify three domains of diffusion for discrete data.

Abstract: To model discrete sequences such as DNA, proteins, and language using diffusion, practitioners must choose between three major methods: diffusion in discrete space, Gaussian diffusion in Euclidean space, or diffusion on the simplex. Despite their shared goal, these models have disparate algorithms, theoretical structures, and tradeoffs: discrete diffusion has the most natural domain, Gaussian diffusion has more mature algorithms, and diffusion on the simplex in principle combines the strengths of the other two but in practice suffers from a numerically unstable stochastic processes. Ideally we could see each of these models as instances of the same underlying framework, and enable practitioners to switch between models for downstream applications.  However previous theories have only considered connections in special cases. Here we build a theory unifying all three methods of discrete diffusion as different parameterizations of the same underlying process: the Wright-Fisher population genetics model. In particular, we find simplicial and Gaussian diffusion as two large-population limits. Our theory formally connects the likelihoods and hyperparameters of these models and leverages decades of mathematical genetics literature to unlock stable simplicial diffusion. Finally, we relieve the practitioner of balancing model trade-offs by demonstrating it is possible to train a single model that can perform diffusion in any of these three domains at test time. Our experiments show that Wright-Fisher simplicial diffusion is more stable and outperforms previous simplicial diffusion models on conditional DNA generation. We also show that we can train models on multiple domains at once that are competitive with models trained on any individual domain.

Published as a conference paper ICLR 2025
A UNIFICATION OF DISCRETE, GAUSSIAN, AND SIMPLICIAL DIFFUSION
Nuria Alina Chandra" * New York University
Yucen Lily Li New York University
Alan N. Amin New York University
Alex Ali New York University
Joshua Rollins CUNY
Sebastian W. Ober
Aniruddh Raghu BigHat Biosciences
Andrew Gordon Wilson New York University
BigHat Biosciences
ABSTRACT
To model discrete sequences such as DNA, proteins, and language using diffusion, practitioners must choose between three major methods: diffusion in discrete space, Gaussian diffusion in Euclidean space, or diffusion on the simplex. Despite their shared goal, these models have disparate algorithms, theoretical structures, and tradeoffs: discrete diffusion has the most natural domain, Gaussian diffusion has more mature algorithms, and diffusion on the simplex in principle combines the strengths of the other two but in practice suffers from a numerically unstable stochastic processes. Ideally we could see each of these models as instances of the same underlying framework, and enable practitioners to switch between models for downstream applications. However previous theories have only considered connections in special cases. Here we build theory unifying all three methods of discrete diffusion as different parameterizations of the same underlying process: find simplicial and
the Wright- Fisher population genetics model. In particular, Gaussian diffusion as two arge-population limits. Our theory formally connects the likelihoods and hyperparameters of these models and leverages decades of mathematical genetics literature to unlock stable simplicial diffusion. Finally, we relieve the practitioner of balancing model trade-offss by demonstrating is possible to train a single model that can perform diffusion in any of these three domains at test time. Our experiments show that Fright-Fisher simplicial diffusion is more stable and outperforms previous simplicial diffusion models on conditional DNA generation. We also show that we can train models on multiple domains once that are competitive with models trained on any individual domain.
INTRODUCTION
To generate high quality sequences conditioned on desired properties. practitioners build diffusion models of language, DNA, and proteins (Sahoo et al. 2024 Sarkar al. 2024 Alamdari al. 2023 Li et al. 2024). These models corrupt each letter in a sequence the "forward" process and train a model to reverse that corruption the "backward" process. model which has been trained to de-noise can be used for high-quality conditional generation (Wang et al. 2024b), for
optimization (Gruver et al. 2023), and myriad other downstream tasks (Luo al. 2022 2025).
Baron et
practitioner has three main choices of forward process (Fig. lb). each with their own strengths:
1. Discrete: occurs in the most natural domain (Campbell et al. 2022).
2. Gaussian: has more mature sampling and training procedures (Dieleman al. 2022
3. Simplicial: in theory inherits the continuous algorithms of Gaussian diffusion while in natural space, but in practice suffers from numerical instability (Avdeyev et al. 2023).
Unfortunately, there is little theoretical infrastructure to compare these models, and thus practitioners have little tacit knowledge to rely on when selecting or designing a model. This gap in understanding is particularly evident in two basic comparison problems which have yet to be solved. First, despite
$ Equal contribution.

---

Published as a conference paper ICLR 2025
Discrete (5 (==1)
Population size (C - 0b
Wright-Fisher
A-A-A-A-C-
Population size 5
B-BB-B-A-A--..
C-B-B-B-A--..
Mutation & Reproduction
B
Gaussian
Simplicial
+
4
B
C B
Reproduction rate
(b)
Figure 1: Discrete, Gaussian, and Simplicial diffusion for discrete data are unified by Wright- Fisher diffusion. (a) Wright-Fisher diffusion with population size ( = 6, showing mutation and reproduction processes across generations. (b) The three diffusion methods emerge different limits of Wright-Fisher: discrete diffusion corresponds to = 1, while Gaussian and simplicial diffusion arise oo with zero and non-zero reproduction rates.
models from the three frameworks achieving similar likelihood values, there is a belief that the "continuous-spae" likelihood is not directly comparable with discrete-space likelihood" (Avdeyev et al. 2023). Second, forward processes in each of these models are specified by hyperparameters with vastly different interpretations. It is unclear how to qualitatively compare the assumptions embedded into each set of hyperparameters across models.
Here we address these theoretical and practical challenges by unifying these streams with a process from human population genetics the Wright-Fisher (WF) model. Our contributions are as follows:
We formally prove all three methods are instances of WF (Fig. D. In particular discrete diffusion corresponds to the WF model with population size of and simplicial and Gaussian diffusion correspond to large population limits with and without reproduction.
We use this connection to answer the two comparison questions above. Surprisingly, we show that likelihoods can only be compared in some cases, depending seemingly inconsequential parameterization choice introduced for only discrete diffusion models in Austin al. 1 which call the hollow parameterization.
We apply our theory to explain and solve the instability of simplicial diffusion by leveraging decades of mathematical genetics literature. We show that this stable simplicial diffusion superior in conditional generation of DNA.
We leverage our theory to show that particular parameterization choice the sufficient- statistic parameterization allows one to train single model that can perform diffusion on all three domains at test time"] We show in experiment that models trained this way are competitive with models trained single domains. This removes the necessity for the practitioner to choose particular model before training.
Our code is available altttpe://gtthub..mmmmmmee///
ffusion
RELATED WORK
We discuss past unification theories and attempts at stable simplicial diffusion. App. we discuss related works in classical diffusion theory, and parameterizations of diffusion models.
Theories unifying discrete and continuous diffusion Winkler et al. (2024) indirectly used a result from [toone] 1963) to connect the special case of -ieedmmesi,, unbiased discrete diffusion to
Of independent interest, it also explains the root of the noted invariiivarinnce" of masking diffusion and extends this property every diffusion model. We discuss this in App. D

---

Published as a conference paper at ICLR 2025
one-dimensional Gaussian diffusion. They use this observation to heuristically argue. conjecture, the convergence of the backwards processes as well. Sahoo et al. (2025) suggested that by taking Gaussian diffusion and applying argmax, one recovers discrete diffusion. F They used this insight to answer the loss comparison problem by proving that the BLOO of discrete diffusion is always superior to that of continuous diffusion. Unfortunately, this is based in App. B): by applying argmax to Gaussian diffusion one does not get a Markov process, a property which was crucial to their proof of the loss comparison question. In our approach, we build mathematically rigorous foundation to compare these models.
mathematical error (details
Stable simplicial diffusion models Richemond et al. (2022) and Avdeyev et al. (2023) suggest diffusion on simplex using two processes used in finance the "Cox-Ingersoll Ross process", and its normalization onto the simplex, the "Jacobi process" and [Benton et al. (024)) suggest "Wright-Fisher diffusion" in toy experiment. However these models struggle from numerical instability. One solution to this instability is to essentially perform Gaussian diffusion (see App. A))
Another is to build wow-matching models on the simplex (Stark al. 2024 Tang et al. 2025 Davis et al. 2024] Eijkelboom et al. 2024). However these sacrifice the ability to straightforwardly calculate likelihood and access many diffusion algorithms, such as classifier guidance.
BACKGROUND AND MOTIVATION
First we describe diffusion models for discrete data and the challenges unifying the frameworks.
3.1 DIFFUSION MODELS FOR DISCRETE DATA
We consider modelling distribution p(20) over discrete space of size B, and will extend to sequences of discrete objects below. Our model will begin with a distribution that is easy to sample from, q(x1), and then applies stochastic process parametrized by from time to O. This produces a trajectory (=( and we hope to pick 0 so that go(xr) ~ p(xo)).
Markov processes To generate training data to fit (o(())))oo), we take samples we ~ p(x0) and evolve it according to a Markov process to get a trajectory p((x)))=o)- We can train qo on these trajectories by optimizing a negative ELBO
- log go (xo) $ Ep((z)) log yo (x(()))))))
(((=)
(1)
= --E (p((s log go (0(()+))_o==1)
+ (L(p(z|(o))l((z)))
p(()))) 21)
The time dilation function To make the second term of Eqn. I small we need p(r) (mo) x g(x1),
Conveniently, applying a Markov process to Vy usually leads to p( converging
stationary
distribution p() as t - x0, a good choice for q(z1). However our is on the interval [0. 1]. so
we compress [0, o0) into [0, 1]: we pick an increasing "time dialation" function T [0, [0. 20)
and simulate 24 so that it has had the Markov process applied to it for time time 7.. In particular, if Ti very large, p()))=0)) & p(.tas))
q(x1). 7 is a more convenient parametrization for our presentation than equivalent functions 3, tr 0t exp(-T)) in other works (Shi et al. 2024).
Picking Tı very large, the second term of the ELBO can be made arbitrarily small, so we leave it out of the presentation below.
Matching forward and backward flow qe is usually parameterized to take I,, t and predict the zo that generated Iz, that is, approximate p(ro Ie. t); we represent this prediction ăo = qe (z |2,, t) as vector of probabilities over the B tokens Σg To,6 1. Some rearrangement then allows one to rewrite the first term of Eqn. as an expectation of term L that can be interpreted as the divergence between the "infinitesimal flow" forward p and backward go at 24:
E Ep(e. [(xe, L, 20, Øo).
Thus getting a stochastic estimate of the ELBO has 3 steps: (I) Sample noisy x, by simulating the Markov process for time Te, (2) Predict do-noised wy with go ( xe, t), and (3) Estimate the ELBO by computing the particular form of L.
'Interestingly, Stone 1963) also wrote discrete diffusion as the function of an underlying Gaussian diffusion. However the function from Stone (1963) was path- dete-ddeeedent time-dilation rather than argmax.
3

---

Published as a conference paper ICLR 2025
Moving to multiple dimensions To model sequences of discrete objects Zo -.. simply apply the forward process to each position zij independently. "Sample noisy x;," remains the same, repeated for every d. The "infinitesimal flow" for each position is also independent: the "Estimate ELBO" step also remains the same, repeated for every and then summed across all d. Therefore, in the "Predict de-noised zo" step will predict i go (= (r, for each d.
20 we
3.2 CHALLENGES COMPARING DOMAINS FOR DISCRETE DIFFUSION
Comparing diffusion models A practitioner much choose forward process which will determine how they train their diffusion model. For discrete diffusion, the forward process is mutation defined with rate matrix L; the form for L was derived in Campbell et al. (2022). This gives Alg. ! where Ep is the indicator vector for the token Tu, (2) A log A X2 À + X2 is the KL divergence Σ. = and ñ. is the derivative of T5. For Gaussian diffusion, the forward process is Brownian motion on embeddings emb(20) E IR"; the form for was derived in Ho et al. (2020). This gives Alg. [ Now, how can a practitioner compare how well each model fits its data, and how can they leverage their expert knowledge when designing their forward process? Unfortunately there is little infrastructure for answering these questions.
between two Poisson distributions, ((
Algorithm ELBO for discrete diffusion
Algorithm 2 ELBO for Gaussian diffusion
1: Sample f Unif(0,
1: Sample f Unif(0, 1)
2: Sample noisy 2:;
2: Sample noisy 2;:
3: Sample 26 Cateaoriiaa *
3: Set 2i emmb e
4: Predict de-noised Zo:
4: Predict de-noised To:
5: Predict žg qe (2), t)
5: Predict žg qo(xol
6: Estimate ELBO:
6: Estimate ELBO:
7: (bb) e (1/6"
7: L 5e 1-e-27y [mmb(3a) emb(#o))1
8: L Σ Co AD (w ( (()()))
8:
Likelihood comparison We would like to compare the ELBOs L] ] of discrete and Gaussian diffusion, but the later infinity due singularity t becomes smal Practitioners must therefore choose a minimum tmin 4 Formally this is equivalent to estimating an ELBO for p(z instead of log p(2o)- However, p(=.t....)
continuous density, fundamentally a different object than the probability of discrete object p(ro). Paradoxically, the values of the ELBOs of the two models are often close suggesting they may nevertheless be formally comparable.
Hyperparameter comparison Discrete and Gaussian diffusion models are specified by hyperparame- ters C and emb with vastly different interpretations: a matrix whose entry Lo describes the rate which bn mutates to b2, versus an embedding function emb that takes the alphabet into Euclidean space R' for some r (we write cmb(#o) as shorthand for Σ, (b))..
Stability of simplex diffusion Below we'll also discuss simplicial diffusion which in principle combines the combines the strengths of discrete and Gaussian diffusion. In practice, it numerically unstable and slow as Sample noisy involves "sampling from Jacobi diffusion processes [which] is more expensive than commonly used SDEs", and Estimate ELBO involves caleulation which "at very small É tends to become very large and cause numerical issues" (Avdeyev et al. 2023).
Practical unification Currently, practitioners must commit to go (0o 1,, trained on these three modalities before training, restricting their access to downstream algorithms. Ideally they could avoid making this choice.
"To see this, note at initialization [[emb(zo) emb(ūo))l" is roughly constant, and for the classical choice = log(1 - t). the square error in Alg. 2 weighted by g22 so the loss is 2 22 f.t t-=dt = oc; a different choice of 7, only acts
and therefore cannot make the loss finite.
*Some discrete diffusion models also have a singularity at 0*,, requiring one to specify a Imin (Campbell models, including masking.
al.
2022 Lou et al. (003). This is not the case for
partially explaining its popularity (Amin et al. 2025 Shi et al. 2024).
4

---

Published as a conference paper ICLR 2025
4
UNIFYING DISCRETE AND GAUSSIAN DIFFUSION
To build the infrastructure for comparing domains for discrete diffusion, we unify discrete and Gaussian diffusion broader framework. Our results lead to better understanding of loss and hyperparameter comparisons. In the following section we extend our framework to simplicial diffusion.
4.1 UNIFICATION RESULT
Our idea represent each dimension of sequence with Ç copies to get sequence of sequences.
ex. for ( = 4, 2g = A CIC T is represented as AAAA CCCC CCCC TTTT.
Then each letter in each sequence is evolved by the mutation matrix L. When Ç we get discrete diffusion and we show that as Ç - oo we get Gaussian diffusion. Below we discuss the case where Ty is single letter / token, which can naturally be extended to multi-dimensional diffusion model.
T. on the simplex We will ultimately arrive at Gaussian limit in Euclidean space, but we first represent De on the simplex. Above, Ie
Algorithm 3 ELBO for ( discrete diffusion
was one of B tokens; now it's one of B
1: Sample ( ~ 1n)
quences of B tokens 2, = xt I...y (C)
It
2: Sample noisy 25;
3: Sample J, ~ MMltinooiial(, ))
can be generated by sampling each it
Categorical Tqj e..) In App. E.1 we note
4: Predict de-noised Zo:
that the loss and target p(zo ft,t) do not depend on the order of s4. Therefore we can represent
5: Predict Ty = go( (2,
Estimate ELBO:
7: ((b) (6' ( (1183
normalized vector of counts Fr,6 = #lb in x///C.
App. E. we derive the ELBO, giving Alg. 3 difterences to dis-
8: L Σ Co (w (100(T))
crete diffusion in Alg. are in blue.
Gaussian limit as Ç-00 The main idea of
our proof below that as (- oc, trajectories converge quickly to स., the stationary distribution of L. and behave like Gaussians near म because of the central limit theorem (Fig.2). 2 As S 00 we will zoom further and further into the neighbourhood of i where the diffusion occurs we move from diffiusion the simplex to diffusion in Euclidean space. Our proof extends previous results in one-dimension (Stone 1963), but uses more modern machinery; interestingly, we see that in the multi-dimmenional case, the relevant Gaussian diffusion occurs in subspace determined by the first eigenspace of C.
Theorem 4.1. (Formal statement and proof in App. F.2 Call O > - ーA1 > -x2 > the eigenvalues
of C and P1 the projection onto the left eigenspace corresponding to \1. Without loss of generality, assume \1
For each Ç pick time dilation of log (Se-2re 2r. Ç (+ 1) and rescale Z
VC - (c 1) 2T (xt - Define the embedding into (Rrnn((A)) Q )1(Q1 1/2
where Q1 diag(a) 1/2 Pidiag((F)) 1/2 and is any isometry from Im(Q1) Brank(
When Ç = we get discrete diffusion: Ti Ti and of is only linearly transformed (Ie F) //
When ( + Ld, we get Gaussian diffusion in the first eigenspace:
Only the first eigenspace has signal: the component of a KerQ1 becomes independent of æo.
The paths (Q ())).(1)1)) converge in distribution to paths from Gaussian diffusion with time dilation Tt and embedding amb(zo) Q1 (To/ /
The ELBO in Alg. converges to the ELBO in Alg.
'This analogous asymptotic methods zoom into a point bounded space to get limit its unbounded tangent plane (ex. chapter 20 va der Vaart (1998) TThis assumption is for convenience. Rescale gew and new Ai7s get the same diffusion.
5

---

Published as a conference paper at ICLR 2025
Proof idea: As Ç - x0, by the law of large num- bers, Jt approaches z e which itself goes to the stationary distribution of L. We can there- fore decompose
Ft 20
signal
noise
Figure 2: Discrete diffusion with a large pop-
The "noise" term is Z2 ET, Since is an average of Ç samples, by the central limit the- orem, it is approximately Gaussian with scale (-1/2 and independent of xp. The "signal" term
ulation converges to Gaussian diffusion. With <= 1000, we show example trajectories (Z))e that converge to approximate Gaussians near therefore is what allows us to predict zz.
The only relevant behaviour is that of the slowest-decaying eigenspaces of L: the top eigen-space represents the convergence to मे and cancels with -,, the next one is P1 with eigenvalue 1, and all others vanish quickly. Therefore the signal is approximately e- Pio.. This means
1 N(.. Σ) for some E. VC
- P1ōo+
Finally, choosing the right scaling and oH gives us Gaussian diffusion. Most of the formal proof involves checking regularity conditions.
4.2 APPLICATION: UNDERSTANDING COMPARISONS OF LOSSES AND HYPERPARAMETERS
Loss comparison Thm. 4.1 suggests that there virtually no difference to training discrete diffusion model with 10 100 and training Gaus- sian diffusion with Alg. 2 on a computer, suggesting their ELBOs are comparable. Yet the limiting Gaussian ELBO is infinite! Fig. 2 suggests why: paths from E have two phases, a nearly deterministic phase at low t (Fig 2 lefft, and then random phase (Fig right). Diffusion models reversing these paths should therefore go through a random phase, until p(zo æ2, becomes obvious, and then trace deterministic path back to Co. However, at initialization, To is "never obvious" to the neural network ga(ang 1), leading to mismatches to the deterministic paths in samples (Fig. 3 "Random"). As Ç gets larger, the paths get more deterministic, causing the singularity in the limit.
Random
Hollow
Figure 3: The hollow parameterization leads to realistic reverse path samples. Ç = 300.
The practical solution is simple weight the output of the neural network
by the evidence for each Tp- (0 x,, ttttpppprrr Zg. (2x) where p(az, zo, t) "automatically handles" deciding when is obvious (Fig. "Hollow"). This was suggested in the appendix Austtn al. (202T) as way to improve discrete diffusion models, but becomes important here as wav to build new Gaussian diffusion models with formally comparable likelihoods? Amin et al. (2025) showed that in higher dimensions this becomes equivalent to using the "hollow" predictor go(00 Fr, t))cp(>) ag, (=g((, ", -. t) where x; is all positions except d. In App. F.3 we formally prove that the hollow parametrization removes the singularity of the ELBO.
Hyperparameter comparison Thm. 4. gives us formula for emb determined by the slowest-decaying directions in L. App. E.4 also shows that every emb can be induced from some L. Remarkably. this connection accommodates embeddings in different dimensions R:: is determined by the dimension of the dominant eigenspace of L. In Fig. 4 we show emb for the BLOSUM stochastic processes for amino acids, and see it clusters similar amino acids together. The practical implications are (1) one can sanity-check their designed L by plotting its induced embeddings, and (2) discrete diffusion offers a richer design space, as one can specify all the interacting eigenspaces of L rather than just the dominant one, emb.
Hydrophabic
Charged
Aromatic
Small
Other
Figure 4: emb of amino acids from BLOSUM L. emb(20) from Thm. 4.1 for L from Amin et al. (025)).
7 Note the hollow parametrization specific to discrete data where there are only finitely many possible x 'This does not require change of architecture: the network can take in but must learn to disregard x

---

Published as a conference paper ICLR 2025
UNIFYING SIMPLICIAL DIFFUSION
Now we add simplicial diffusion to our unification of discrete and Gaussian diffusion. Proving the equivalence of the forward process: we add reproduction to our population of letters and simply refer to the well known result of Kimura (1955) from mathematical genetics. We also derive new results on the limit of the ELBO and explore our connection with theory of mathematical geneties; this will allow us to address the instabilities that plague simplicial diffusion models.
5.1 UNIFICATION RESULTS
The Wright-Fisher model We now allow our population of reproduce. The population is swapped with a new generation at rate ( (that is, a new generation occurs at ÅT ~ (xp(1)/C and at each generation create Ç "children" which pick a parent uniformly at random. Between generations, individuals also mutate according L (Fig. lay. We now ask what happens when
8.
The limit of p(())))) Kimura (1955) was the first to derive the (- 00 limit of the stochastic process. Unlike the mutation-only case which zooms in F. this limiting distribution has paths that travel throughout the simplex (Fi..Ib). This limit, often itself called "Wright-Fisse Fiight-FFiher diffusion" is
exactly the forward process in simplicial diffusion (Avdeyev et al.] 2023). Details are in App. [E.5.1 One biologically reasonable assumption past works make is parent-independent mutation rate matrix, that is, L v (1 7) for stationary distribution ग and mutation rate vi O. This does not restrict the design space of simplicial diffusion, which is specified by an intensity parameter ei and stationary distribution म. so we make the same assumption.
The limit of the ELBO We derive the limit of the discrete diffusion ELBO. Remarkably, we get an objective that matches "score functions" 8 like that heuristically derived in Avdeyev et al. (202)). also mathematically equivalent the expression in Eqn 27 of Benton et al. (2024) with two differences: (1) we avoid taking the derivative of the neural network, and (2) their expression differs by an unknown constant from the ELBO, while ours directly comparable with ELBOs from other models.
Theorem 5.1. (Proof in App. E.5.2 As Ç - 3D, the discrete diffusion objective in Alg. converges to the quantity in line of Alg. 4
The main idea of the proof is an application of Taylor expansion and Stirling's approximation: the main challenge is handling of behaviour at the boundaries of the simplex and regularity conditions.
5.2 APPLICATION: FAST AND STABLE SIMPLICIAL DIFFUSION
We have unified simplicial diffusion with discrete and Gaussian diffusion, in particular allowing like- lihood comparison, which will be crucial in the following section. Our unification also immediately suggests a connection to the mathematical genetics literature. We now apply the solutions from that literature to improve simplicial diffusion models. Many of the formulas are standard but long save their statement and experimental validation to App.
Algorithm 4 ELBO for simplicial diffusion. Our changes to Avddeev et al. (2023) are coloured.
1: Sample f Unif(0, 1)
2: Sample noisy 2y;
A(w, T)) with Alg. 5 if < 0.05, use Alg. 6 4: Sample 7, Dirichlet(yfrr man)
3: Sample
5: Predict de-noised Zo:
6: Predict žg = qe( 2o mz,
7: Estimate ELBO:
8: Compute 5(57 No. t) = Vz, log p(,; |x0, b) with Eqn.
9: L
ft ( No. t) T æo. 1] ( (this is an ELBO); if < 0.05, use Eqn.
Sampling noisy zt Avdeyev et al. (023) samples 2, by costly and approximate simulation from stochastic differential equation (SDE). Instead, the suggestively titled paper "Exact simulation of the -rihht-Fisher diffusion" (Jenkins and Spano 2017) gives a fast exact formula for the marginals ra. The algorithm samples V. from a Dirichlet that is centred at the stationary mutation distribution when 0 and becomes more concentrated around the signal zo when is larger. itself is
"Exert
7

---

Published as a conference paper ICLR 2025
DNA Accessibility
Diffusion (Ours)
50
RMSE (4)
Flow Matching
14
Target
Samples
Diffusion (Prev)
Diffusion (Prev)
Random
Flow Matching
Diffusion (Ours)
Position
150
(a) Example samples
(b) Average error
Figure 5: Improved simplicial diffusion performs accurate conditional DNA generation. We generate DNA samples length 500 conditioned on accessibility with a classifier. (a) For an example target, we plot predicted accessibility profiles the centre 150 positions of 5 example samples from each model. We smooth profiles with a bandwidth 2. (b) For 1000 targets and 10 samples from each model, we plot the error between the predicted and target profiles and its standard error.
an integer sampled from distribution A 72) that represents, going back in time 7r, how many ancestors the population descend from it is small when 7, is large, when everyone descended from handful of individuals from far back in time. Benton et al. (2024) proposed the same procedure, but applied a toy setting.
Computing the loss For the loss, Avdeyev al. (2023) derived a likelihood that involved calculating the derivative the predictor qo (20 2z, making too expensive to train on. They instead suggest training heuristically motivated loss matching the vector to a ground truth. In Thm. 5.1 we recognize this loss as an ELBO and derive the appropriate scaling z and metric diag (z+) -,,
Low behaviour Both the simulation of A Te) and the calculation the gradients Vz. log p(a; go) involves an infinite series (Tavaré 1984). Luckily the terms converge extremely fast. This is true however at low i, which is the primary cause of the instability of simplicial diffusion. This instability is also well known in the genetics literature, with Griffiths (1984) emphatically stating that using the infinite series at low "produces nonsense from a computer."
The solution at low L is replace the series approximation, which gets worse with lower with central limit approximation for A T2) (Griffiths) 1984, Jenkins and Spanò 2017) that improves with lower this analogous how reflected diffusion models were made stable despite their own infinite series expansion with the same problem Luo et al. 2022). We picked the < 0.05 threshold as recommended by JJenkins and Spanò (2017). In App. we describe how use this approximation to also stabilize the loss computation.
State of the art DNA generation conditioned on classifier Simplicial diffusion models are state of the art tools for generating DNA conditioned on high-dimensional epigenetic properties (Avdeyev et al., 2023); however they have recently been surpassed by flow-matching models (Stark et al.) 2024), which are more stable but sacrifice closed-form ELBO and access to diffusion sampling algorithms. Given stability improvements above, expect to be able to generate higher quality sequences than previous methods. We fit the state of the art diffusion model (Avdeyev et al. 2023) and flow- fow-mngching model (Stark et al. 2024) to DNA data (B 4) of length D 500 and generate samples conditioned on achieving target "DNA accessibility profiles."
First we see our model leads much better fit of the data. The diffusion model from Avdeyev et al. 2023), was only able to achieve an average ELBO of 8 nats / position (12.7 before training), while trivial model which predict uniform letters in each position achieves 1.39. contrast, our model achieves an ELBO of 1.30. In Fig. we also see our new model generates conditional samples with profiles that much better match the target. Experimental details are App.
6 PRACTICAL UNIFIED DIFFUSION MODELS
Our results show that discrete, Gaussian and simplicial diffusion are three views of the same process. But which view should practitioner choose for their particular downstream task? Unfortunately, there is limited theoretical infrastructure we can use answer such question.
8

---

Published as a conference paper ICLR 2025
Unitied Model
43r
Unified Model
241
3.55
NLL
4:0
Singantel
Gierate
Ampgiiial
Cornae
Samperca
Di
Diseett
Divond
Ohenee
Clocsion
Diunes
Deoction
Divoreta
Gavosiin
(a) Protein likelihood and sample quality
(b) Language likelihood and sample quality
Figure 7: The sufficient statistic parametrization enables a single model perform competitive discrete, Gaussian, and simplicial diffuusion. We compare individual models for each modality with a single unified model using the SSP. (a) We train on proteins and measure sample quality by predicted protein fold-ability (pL.DT).. Each model was trained for the same amount of time. (b) We train on language and measure sample quality using the perplexity of much larger language model. Each model was trained for 33 epochs.
Instead our theory provides a practical solution: leveraging our finding that these methods have comparable likelihoods, show through particular parameter choice (Fig. 6). one train single neural network that can perform diffusion on any domain at test time. In App. D we also show this parameterization will also allow to make any diffusion model time-invariant, explaining and generalizing a celebrated property of masking diffusion.
6.1 THE SUFFICIENT STATISTIC PARAMETERIZATIO, (SSP)
The goal of a diffusion model is predict z To do so, one must integrate over the unseen xp weighted by their likelihood of producing the data x;,"
Domain param.
Unified param.
Discrete
& (T f
x =
p(ag x =
= f [paa zy d)p(2= x ()
Gaussian
A
x, = [0.5, 0.6, -0.4]
B
Simplicial
x.= [0.3, 0.6, 0.11
We can summarize this "evidence" in the normalized vector (( t)oxp(a? t, xz6" b) (Fig. 6p.
Figure The sufficient statistic parameterization represents T from all diffusion models the same space.
Some algebra shows that Ø's are sufficient statistics, that is, they contain all relevant information about the diffusion process and I, leaving a regression task that invariant to both.
Proposition 6.1. (Proof in App. F.7 diffusion process or t, such that
There is function Fd, depending on p(zo) and not the
p(ag 2 (e.00)
()..,, (((., 3(55.1))
F'.., for Therefore we can parametrize our neural network go (x xt .) a neural network Fd that tries to learn the "universal" Fd
6.2 APPLICATION: UNIFIED DIFFUSION MODELS
Practitioners must commit upfront to the domain their diffusion occurs. The SSP instead enables training a single neural network that can perform diffusion on any domain at test time: as long the target distribution p(z)) remains constant the optimum Fd remains the same. Furthermore. shown above that the ELBOs of each modality are comparable, so we can train Fe by alternating minimizing the ELBO of different modality in each batch.
ve
We train discrete, Gaussian, and simplicial diffusion models on proteins and compare to a single model trained using the SSP which alternates between discrete, Gaussian, and simplicial training steps. We trained our models to approach the performance of state-of-the-art protein diffusion model DPLM (Wang al. 2024a) in likelihoods (2.36) and a "foldability" metric for samples (45.2) (Amin) al. 2025). In Fig. we that single SSP model trained proteins for 48 hours is competitive

---

Published as a conference paper ICLR 2025
in perplexity and sample quality with three single-domain models each trained for the same amount of time. We perform similar experiment for discrete and Gaussian language models (simplicial diffusion models are challenging to scale to large vocabulary size of B x 3 x 10*) and see similar results. We trained our language models with the same amount of training data as state-of-the-an diffusion language model (Lou et al.) 2023), matching its likelihood (SEDD uniform has NLL of 3.70). Experimental details are in App. another downstream task is tested in App. G.1 and we also repeat these results MNIST images App. G.2
7 CONCLUSION
Our theoretical and practical unification developed foundations that we used to improve simplicial diffusion and avoid the need choose a specific model at train time. However, the theory suggests number of other directions we have not yet explored.
Notable omissions from our presentation are reflected diffusion (Lou and Ermon 2023). flow matching (Campbell et al., 2024), masking diffusion (Shi et al. (004)) and diffusion with insertions and deletions (Johnson et al. (02T). The later two can likely be easily accommodated with previous theories unifying masking and uniform diffusion on one hand (Amin et al. (025), and substitution and insertion deletion diffusion on the other (Johnson et al. 2021).
As well, our framework suggests of diffusion which we only use as a lens for understanding existing models. Implementing these intermediate models may be of independent practical interest.
types of diffusion models "between" the three existing streams
Finally, the SSP can be used to unify models beyond the three modalities. For instance it can be used to train models across hyperparameter settings, optimize hyperparameters without retraining. In principle, the SSP can even be used to transfer a model to a modality it was not trained on.
ACKNOWLEDGEMENTS
This work was supported in part by NSF CAREER IIS- II--2145492, NSF CDS&E- MSS 2134216, NSF HDR-2118310, BigHat Biosciences, and Capital One.
REFERENCES
S. Alamdari, N. Thakkar, R. van den Berg, A. X. Lu, N. Fusi, A. P. Amini, and K. K. Yang. Protein generation with evolutionary diffusion: sequence is all you need. bioRxiv, Sept. 2023.
A. N. Amin, N. Gruver, and A. G. Wilson. Why masking diffusion works: Condition on the jump schedule for improved discrete diffusion. In Frontiers in Probabilistic Inference: Learning meets Sampling, Apr. 2025.
B. D. O. Anderson. Reverse-time diffusion equation models. Stoch. Process. Their Appl., 12(3): 313-326 May 1982.
J. Austin, D. D. Johnson, J. Ho, D. Tarlow, and R. Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Adv. Neural Inf. Process. Syst., 34:17981-17999 2021.
P. Avdeyev, C. Shi, Y. Tan, K. Dudnyk, and J. Zhou. Dirichlet diffusion score model for biological sequence generation. arXiv [cs. May 2023.
E. Baron, A. N. Amin, R. Weitzman, D. S. Marks, and A. G. Wilson. A diffusion model to shrink proteins while maintaining their function. In The Exploration in Al Today Workshop at ICML 2025, June 2025.
R. F. Bass. Stochastic Processes. Cambridge University Press, Oct. 2011.
J. Benton, Y. Shi, V. De Bortoli, G. Deligiannidis, and A. Doucet. From denoising diffusions to denoising markov models. J. R. Stat. Soc. Series B Stat. Methodol., (6(2):28-3011, Apr. 2024.
D. Calderon, R. Bleeher-Gonen, X. Huang, S. Secchia, J. Kentro, R. M. Daza, B. Martin, A. Dulja, C. Schaub, C. Trapnell. Larschan, K. M. "'Connor-Giles" E. M. Furlong, and J. Shendure. The continuum of <i>drosophila</i> embryonic development at single-cell resolution. Science, 377166666eeabm58800 2022. doi: 10. 10.112664cieeeeeaam5800000 URL https: ://www/ science 11t.
abn5800
10

---

Published as a conference paper ICLR 2025
A. Campbell, J. Benton. V. De Bortoli, T. Rainforth, G. Deligiannidis, and A. Doucet. A continuous time framework for discrete denoising models. In Advances in Neural Information Processing Systems, Oct. 2022.
A. Campbell, J. Yim, R. Barzilay, T. Rainforth, and T. Jaakkola. Generative flows discrete state-spaces: Enabling multimodal flows with applications to protein co-design. In Proceedings of the 41st International Conference Machine Learning. arXiv, 2024.
N. A. Chandra, Y. Hu, J. D. Buenrostro, S. Mostafavi, and A. Sasse. Refining sequenccett--actvvty models by increasing model resolution. bioRxiv, 2025. doi:
O. Davis, S. Kessler, M. Petrache, I. I. Ceylan, M. Bronstein, and A. Bose. Fisher flow matching for generative modeling over discrete data. arXiv [cs. Lcs. May 2024.
S. Dieleman, L. Sartran, A. Roshannai, N. Savinov, Y. Ganin, P. H. Richemond, A. Doucet, R. Strudel, C. Dyer, C. Durkan, C. Hawthorne, R. Leblond, W. Grathwohl, and J. Adler. Continuous diffusion for categorical data. arXiv.org, 2022.
Bijkelboom, G. Bartosh, C. Andersson Naesseth, M. Welling, and J.-W. de Meent. Variational flow matching for graph generation. Advances in Neural Information Processing Systems, 37: 11735-11764, 2024.
N. Ethier and T. G. Kurtz, Markov Processes: Characterisation and Convergence. Probability Mathematical Statistics S. John Wiley & Sons, Nashville, TN, May 1986.
G. Floto, T. Jonsson, M. Nica, S. Sanner, and E. Z. Zhu. Diffusion on the probability simplex. arXiv [cs, Sept. 2023.
Gotze. On the rate of convergence in the multivariate CLT. Ann. Probab., 19(2):244-339 1991.
R. Griffiths. Asymptotic line-of-descent distributions. Math. Biol., 21(1):67-75, Dec. 1984.
N. Gruver, S. D. Stanton, N. C. Frey, G. J. Rudner, Hotzel, J. Lafrance- Lafrance-Vanasse, A. Rajpal, K. Cho, and A. G. Wilson. Protein design with guided discrete diffusion. In T'hirty-eveenth Conference on Neural Information Processing Systems, Nov. 2023.
X. Han, S. Kumar, and Y. Tsvetkov. SSD-LM: Semi-auttregrrsssvv simplex-based diffusion language model for text generation and modular control. arXiv [cs. CL), Oct. 2022.
B. L. Hie, V. R. Shanker, D. Xu, T. J. Bruun, P. A. Weidenbacher, S. Tang, W. Wu, J. E. Pak, and P. S. Kim. Efficient evolution of human antibodies from general protein language models. Nat. Biotechnol., 42(2):275-22) Apr. 2023.
J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 6840-6851. Curran Associates, Inc., 2020.
F. M. Hoppe. Polya-like urns and the ewens' sampling formula. J. Math. Biol., 20(1):91-94, Aug. 1984.
A. Jenkins and D. Spanò. Exact simulation of the Wright-Fisher diffusion. Ann. Appl. Probab., 27 (3):1478-1509, June 2017.
Johansson and Others. mpmath: Python library for arbitrary-precisionn flotting--oiii arithmetic (version 0.14), Feb. 2010.
D. D. Johnson, J. Austin, R. van den Berg, and D. Tarlow. Beyond in-place corruption: Insertion and deletion in denoising probabilistic models. ICML Workshop on Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models, 2021.
M. Kimura. Solution of a process of random genetic drift with a continuous model. Proc. Natl. Acad. Sei. U. S. A., 41(3): 150, Mar. 1955.
B. Li, Z. Gao, and I.. Xu. Unifying continuous and discrete text diffusion with non-simultaneous diffusion processes. arXiv [cs. CLJ, May 2025.

---

Published as a conference paper ICLR 2025
Z. Li, Ni, G. Xia, W. Beardall, A. Das, G.-B. Stan, and Y. Zhao. Absorb & escape: Overcoming single model limitations in generating heterogeneous genomic sequences. Advances in Neural Information Processing Systems,
2024.
Z. Lin, H. Akin, Rao, B. Hie, Z. Zhu, W. Lu, Smetanin, R. Verkuil, O. Kabeli, Y. Shmueli, A. dos Santos Costa, M. Fuzel-Zarandi, T. Sercu, S. Candido, and A. Rives, Evolutionary prediction of atomic-level protein structure with language model. Science, 379(6633):112333111 2023. doi: 10.1 (0.1166/sieecce.ade25577 URL https: www.wwscinnc..... www.iiinnn..rr/////////oo./ ottp:///////////000000a// 1126/science ade2574
A. Lou and S. Ermon. Reflected diffusion models. ICML, 4b522304.004444222755522222 Apr. 2023,
A. Lou, C. Meng, and S. Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. In 41 st International Conference on Machine Learning. Oct. 2023.
S. Luo, Y. Su, X. Peng, S. Wang, J. Peng, and J. Ma. Antigen-specifie antibody design and optimization with diffusion-based generative models for protein structures. In Advances in Neural Information Processing Systems 35. Cold Spring Harbor Laboratory, July 2022.
R. K. Mahabadi, H. Ivison, J. Tae, J. Henderson, I. Beltagy, M. E. Peters, and A. Cohan. TESS: Text-to-text self-conditioned simplex diffusion. arXiv /cs.CL.J, May 2023.
J. W. Miller. Asymptotic normality, concentration, and coverage of generalized posteriors. arXiv [math.ST], July 2019.
Ou, Nie, K. Xue, Zhu, Sun, Z. Li, and C. Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. arXiv [cs.LG], June 2024.
A. Raghu, S. W. Ober, M. Kazman, and H. Elliott. Guided sequence-struee strucccre generative modeling for iterative antibody optimization. In ICLR 2025 Workshop on Generative and Experimental Perspectives for Biomolecular Design, 2025.
P. H. Richemond, S. Dieleman, and A. Doucet. Categorical SDEs with simplex diffusion. arXiv [cs, Oct. 2022.
H. Robbins. A remark on stirling's formula. Am. Math. Mon., 62(1):26,, Jan. 1955.
S. S. Sahoo, M. Arriola, Y. Schiff, A. Gokaslan, E. Marroquin, J. T. Chiu, A. Rush, and V. Kuleshov. Simple and effective masked diffusion language models. arXiv (cs.CL), June 2024.
S. S. Sahoo, J. Deschenaux, A. Gokaslan, G. Wang, J. Chiu, and V. Kuleshov. The diffusion duality, arXiv [cs. June 2025.
A. Sarkar, Z. Tang, C. Zhao, and P. K. Koo. Designing DNA with tunable regulatory activity using discrete diffusion. bioRxiv, page 2024.05.23.595630, May 2024.
A. Shabalin, V. Meshchaninov, and D. Vetrov. Smoothie: Smoothing diffusion on token embeddings for text generation. arXiv [cs. CLJ, May 2025.
J. Shi, K. Han, Z. Wang, A. Doucet, and M. K. Titsias. Simplified and generalized masked diffusion for discrete data. arXiv [cs.LG], June 2024.
H. Stark, B. Jing, C. Wang, G. Corso, B. Berger, R. Barzilay, and T. Jaakkol.. Dirichlet flow matching with applications DNA sequence design. arXiv (q-bio.BM), Feb. 2024.
C. Stone. Limit theorems for random walks, birth and death processes, and diffusion processes. Illinois Math., 7(4):638-66) -6060886666610666 Dec. 1963.
B. E. Suzck, H. Huang, McGarvey, R. Mazumder, and C. H. Wu. UniRef: comprehensive and non-redundant UniProt reference clusters. Bioinformatics, 1282- May 2007.
Tang, Y. Zhang, A. Tong, and Chatterjee. Gumbellsoftmax flow matching with straight-through guidance for controllable biological sequence generation. arXiv [cs.LG], Mar. 2025.
12

---

Published as a conference paper ICLR 2025
S. Tavaré. Line-of-descent and genealogical processes, and their applications in population genetics models. Theor. Popul. Biol., (662):119-166 Oct. 1984.
A. W. van der Vaart. Asymptotic Statistics. 1998.
X. Wang, Z. Zheng, Ye, D. Xue, S. Huang, and Q. Gu. Diffusion language models are versatile protein leamers. ICML,, abs/2402.18567, Feb. 2024a.
X. Wang, Z. Zheng, F. Ye, D. Xue, S. Huang, and Q. Gu. DPLM-2: multimodal diffusion protein language model. arXiv [cs.LG], Oct. 2024b.
L. Winkler, L. Richter, and M. Opper. Bridging discrete and continuous state spaces: Exploring the ehrenfest process in time-continuuss diffusion models. arXiv [statMLJJ, May 2024.
R. Wu, F. Ding, R. Wang, R. Shen, X. Zhang, S. Luo, C. Su, Z. Wu, Q. Xie, B. Berger, J. Ma, and J. Peng. High-resolutionde novostructure prediction from primary sequence. bioRxiv, page July 2022.
2022.07.21
K. K. Yang, N. Fusi, and A. X. Lu. Convolutions are competitive with transformers for protein sequence pretraining. Cell Systems, 15(3):286-295 294, 2024.
Y. Zhang, Hartl, Hazan, and Levin. Diffusion models are evolutionary algorithms. arXiv preprint arXiv:2410.02543, 2024.
K. Zheng, Y. Chen, H. Mao, M.-Y. Liu, J. Zhu, and Q. Zhang, Masked diffusion models secretly time-agnostic masked models and exploit inaccurate categorical sampling. arXiv [cs.L.G], Sept. 2024.
13

---

Published as a conference paper at ICLR 2025
A EXTENDED RELATED WORK
We add more related work beyond those in Sec.
Classical theories unifying discrete and continuous stochastic processes There is a long history of deriving continuous limits of discrete processes, the "forward" processes of diffusion models. Groundbreaking work by Stone 1963) derived Gaussian diffusion as limit to biased one-dimensional random walks. In one of the most celebrated results in mathematical genetics, [Kimura| (1955) also derived a continuous limit of the Fright-Fiishe process with non-zero reproduction. These models were originally developed to describe the stochastie fluctuations of allele frequencies in the absence of selection, also called genetic drift. We (1) apply these results to understand and improve diffusion models, (2) also show convergence of the ELBO of diffusion models, and, to our knowledge, (3) derive new result- the multi-dimensional Gaussian-diffusion limit of Wright-Fisher with zero reproductions demonstrating previously un un-characterized behaviour dependent on the eigenspace of the mutation operator. Results (2) and (3) are what allow us to compare likelihoods and hyperparameters.
Parameterizations of discrete diffusion models In diffusion, one uses a neural network to "denoise" sequences; we call the choice of inputs and outputs of these neural networks the "parameterization" number of works suggest superficially distinct. but ultimately equivalent parameterizations (Camp) bell et al.] 2022 Lou et al. 2023). Austin et al. (202T) however suggested distinct parametrization for discrete diffusion models by scaling the output of the neural network to "automatically" incor- porate the information about the noised token about that particular location; we call this choice the "hollow" parametrization for reasons discussed below. [Zeng et al. (2024), Ou et al. (2024), and Sahoo et al. (2024) suggested that masking diffusion enables special choice of
parametrization; in App. D our theory shows on the contrary that every diffusion model can be made iime-invariant.
Gaussian diffusion which appears as simplicial diffusion Han et al. (2022 Mahabadi et al. (2023); : Shabalin al. (2025), and Floto et al. (2023) suggest stable diffusion model on the simplex by applying softmax to Gaussian diffusion and using Itô's 's theorem. This parameterization is stable because forward and backward diffusion can occur as Gaussian diffusion in the logit-space. [Lou and Ermon (223) has a similar idea, swapping the softmax for an asymmetric transformation and Gaussian diffusion with reflected Gaussian diffusion. With these simplifications however, the process is exactly (reflected) Gaussian diffusion except the input to the neural network is transformed onto a simplex; in particular, it doesn't 'tt interact with the topology of the simplex. In other words, this implements simplicial diffusion in the parametrization of the neural network, but not in the sampling or loss computation.
Another unification theory and connection to evolution Li et al. (2025) looked Gaussian diffusion with a generalized noising strategy; they noted a special case resembled masking diffusion. However the training procedure and ELBO of this special case are distinct from standard masking diffusion (Shi et al.) (2224). Zhang et al. (2024) connect diffusion with the evolution process to suggest an optimization algorithm, but do not formally establish connections with biological evolution. In contrast, our work makes this connection explicit.
B MATHEMATICAL ERROR IN SAHOO ET AL. (2025)
In Theorem 3.1, Sahoo et al. (2055) shows that the BL of a discrete diffusion model always tighter than that of Gaussian diffusion model. In its proof, with w, from Gaussian diffusion, 2 = argmaxxww) and æ = 2g = Wo. they state "Since the transition 2 - Z. is Markov, we get: q(2,, Wa, 2t, x) = q(2, 22, 2))". Putting aside the correctness of this statement, it is clear that the proof as stated requires the Markov property of (z)).
The way the Markov property is shown is as follows. They first define discrete diffusion model, let's call this (z)), such that &o comes from the data distribution and E evolves with respect to uniform forward process with rate parameter ((t) chosen such that the marginals match p(=a|=0) p(ă (20). In Eqn. 29 they compute d 4.p(z, ze 20) and in Eqn. 32 they compute d p & /n) for all starting points and show they are identical. After noting the equivalence of equations 29 and 32, they state "This pmf and the ODE are the unique signatures of a Uniform-stat discrete diffusion process (Lou et al.,
14

---

Published as a conference paper at ICLR 2025
2023; Schiff et al., 2025)." and from this conclude that the path distributions of (2t) and (zt), are equivalent, and in particular, that (2t) is Markov
However, despite similar result for Markov chains (two Markov processes with identical semi- d
d groups are equivalent), p(2x = p 2 (2) and ai {(2((=(2)) = a d( Eo) for all starting points is not enough to conclude the identity of the path distributions p( p((2))=lz))
((()))))))) First note that
4 ((2)(2)) de p &n) is not independent condition: it follows from p(2x p (1))2))) Next
consider this counter example:
20 1 and (2)) evolves by switching sign with rate Therefore p(2 = 0) 1 2 2
20 1 and (2 has 50% chance to stay at 0 forever and a 50% chance to swap sign at time -
2 log U for a U ~ Uniform and never again. Therefore p(2, = 1) 2 (1 +
- log Uniform > f)) = =1 _022. -2
When =o or 8g = 1,, then swap signs.
We have p(=)/20) 202) for all 2g and therefore p(=)2200) 2.p(Ez)20) but clearly p((()))) + p(()))))
Simple computer simulations indeed show that p((argmax( wy ()) and p(()))) are different. We show this in Fig. 8 Indeed a statistical test applied to these simulations shows pl ((aggaax(w)))) o p((()):): Mann- Whitney test shows that the paths of the argmax of Gaussian diffusion have more transitions that those of discrete diffusion with p 10 10-300.
(w
Discrete diffusion
Marginals
10
Example D
Discrete difusioon
argmaxtee)
0.5
Marginal probability of
0.0
0.6
00
02
04
0.6
0.5
Example 1
0.-
0.4
0.0
0.6
06
Example 2
02-
0.0 02
0.0
0.6
066
time
time
time
Figure 8: The argmax of Gaussian diffuuion appears different from discrete diffusion in sim- ulation, despite having the same marginals. We compare example paths of p( (w)))c) (left, red; we show Gaussian diffusion W; in grey), p(()))) for uniform discrete diffusion (centre, blue). and their empirical marginals over 10'000 simulations (right); we simulate using grid size of 0.0001. Note the two processes have the same marginals but their paths appear different; in particular, whenever we is near 0, (argmax(w::) undergoes very large number of transitions in small time [0
C WRIGHT-FISHER SAMPLING AND SCORE CALCULATIONS
Here we discuss the details of the methods in Sec. 5 Note, just like App. E. we can deal with o rather than the actual sequences .4.- In App. C.1 we discuss details about our algorithms, in particular
how to sample form A (1/2 7,) and calculate the functions ( xo)) In App. C.2 we discuss the
computational complexity and stability of these algorithms in theory and in experiment. In App. C.3
Thii interpretation of the text was confirmed in personal communication with the first author Sahoo et al. 2025)
"ndeed, noting the self similarity of Brownian motion, one can show that, conditioned on wi = 0, with probability 1 (zx): makes infinitely many transitions in the interval [t, + c) for any > 0. The probability of infinitely many transitions in bounded interval for discrete diffusion however is 0.
15

---

Published as a conference paper ICLR 2025
we discuss how we sample and calculate the ELBO at low t. Finally, in App. C.4 we discuss our condiitonal sampling procedure.
We also note two more differences between our method and that of Avdeyev al. 2203
Our neural network directly predicts Zo rather than the "score" g.
As described in App. E.5. we use the natural permutation--ymmeeriic "multi-allelic" exten- sion the ID SDE when B 2, while they use a brikkbre procedure.
We use sech-precision operations to calculate large alternating series accurately, described in App.
C.1 ALGORITHM DETAILS
Sample noisy 24 We've discussed the algorithm from Jenkins and Spanò (2017) in the main text. We now present their algorithm for sampling from A(e, Tt)-
Algorithm 5 Exact sampling from ancestral process A( Tc)
Define coefficients: c.m V 11- ((1+0-117// for k m
ml(k-m)l
2: Sample ~ Uniform 0, 11
3: Initialize M
4: Initialize an empty vector h = ()
5: while True
6: Find kas 2 M such that at C(KM +1)M
7: Make kas even: kM /2]
8:
Update lower bound: 5 - S + - (-1) k> CKM
9:
Update upper bound: S S
2k M 1 M
KM
KKM
10:
Update 1- (ko..... 1M- kMM) while 5 < - < 5 do
11:
12:
Update lower bound: S - 5- + SMd
2m-o(c (Am +2m (lim +lmm+3)m +3)ml
13:
Update upper bound: St - S + /m=
W (-c))
(C +1m + (K.. +2mm
14:
Update T= (.....,
15:
end while
16:
if S > U then
17:
return m = M
18:
else if 5* < U then
19:
M - M +1
20:
end if
21:
end while
Compute loss We present a formula for 5(Ø Fo, = V log Zo. f) to enable computation of the loss. Avdeyev et (2023) computed these scores using a previously determined result with B = 2 then generalizing to higher dimensions with their stick-breakin procedure and a change of variables. We are instead able to derive directly from first principles.
There are two infinite series which will be important,
20
GolT, 20, =1)
Σ
(-1) 1)* k ag Toor Fr,0))
k=1
Fo(T, = 1++ >
(- 1) 1) b2 (T, Tzzy Fe,00)
k=1
16

---

Published as a conference paper at ICLR 2025
where
k(b+w-i); 2
2 (-k; + k 1;40aazi aro) (2k + vb (6-1)
Og (T, Toru F2,
k!
600-a g Dy (1)) (2k + 1b () k) 2 (- ki tb + k + 1; 1/T 1; 1e
(T; (T; Tang Fttzo)
k
(1/ + 1)17
where 2 is the hypergeometric function. Although these look complicated, in practice, most terms in the numerators and denominator of and b nearly cancel to 1, and, when is not too small, e -=( decays extremely quickly.
Using the results Tavaré (1984) we compute s( (o) in terms of these series. Since we're only interested in differences for calculating the ELBO, ((() z0, t) ( To. t) we ignore constants not depending on Zo.
Proposition C.I. (Proof in App. E.6
p( No. 1) = Dirichle((zx)((@)))otr.... ū).
For () V log Dirichlet(Ty))) (T() = (भे L)/d, which does not depend on 2o,
( to a(T) + Fow((Fo 17)
(2)
where
e 2 (1b + 1) To, ū) w(2o, ū)
TT (200) GT( t)
With the hollow parameterization, calling Wh = w(b), we geteet
i To. t) = a()) + e - ( + 1)
30,t Fo,,Fo(Te,
xxxo)
C.2 COMPUTATIONAL COMPLEXITY AND STABILITY
Numerical stability Sampling from A T²) and calculating G. and F&. involve alternating series of many terms which vary by many orders of magnitude, and cancel out leaving very small residuals- known as "catastrophic cancellation". To calculate these accurately, precision than provided by float 64; we perform any high precision calculations using the mpmmat th library Johansson and Others 2010). Avdeyev et al. (2023) did not high precision in their calculations, potentially introducing errors and instability to their loss computation.
may need higher
We perform all calculations at float 64 take advantage of parallel GPU computations, estimate the error of each computation using a condition number and recompute just those terms with condition number above threshold in mpmath on CPU. In practice, we only need to perform calculations at high precision for small f, before we switch to the "low time regimen" T₃ 0.05 where we switch tot he Griffiths approximation.
Σnm (am 1/ml//1 Em aml | one can The condition number of a series ai + 02 + +... + das is defined as estimate the error of their summation at finite precision by
error A x precision.
When there is catastrophic cancellation, the denominator in the definition of will be very large, rep- resenting that error might be high. To estimate 7), we keep track of laml ( representing our finite- precision summation) and estimate > Σ., lam.l///./ aml If > desired error/ float 64 precision = 10 x 252 then we recompute at higher precision.
Sampling The complexity for sampling Zf involves (1) (()) for sampling m, which is O(1/T) in expectation (see App. C.3). and (2) O(B) time for sampling H. from Dirichlet. Crucially, the complex calculations involving an infinite series occur in (1) and are independent of the alphabet size B. Comparatively, sampling from the SDE requires O(BT/ At) compute. A higher At will decrease compute but lead to lower-fideliyy samples, especially at low where even small fluctuations in x; can lead to instability. We also parallelize the computations in Alg- to benefit from GPU acceleration. The result that, except when we must switch to high precision, our sampling procedure is much faster than that using SDE (Fig. 9a), and much more stable at low (Fig. 9b)
17

---

Published as a conference paper ICLR 2025
10 2
1e+120
Time (5)
Giffiths (Ours)
SDE (Prev.)
Relative Error
lew 1's 0.05;
10-3
1e-80
high rSt 0.05)
10"4
SDE
1e-10
Curs
Emor threshold
10-5
0.5
Luo liUb
Luo much
Kmax 120
Density
# terms
(a) Fast sampling
(b) Stable sampling
(c) Fast and stable loss calculation
Figure 9: Leveraging mathematical geneties literature, we build fast and stable simplicial diffusion. (a) We plot the time it takes to sample a sequence of = 500 using an SDE, versus our exact sampling for various values of & on A100 80GB GPU. We threshold switching to the Griffiths approximation at 7, = 0.1. (b) For 7 = 0.1 and B we sample 3 x 10? points from the exact sampling method, Griffith's approximation, and using an SDE with 25 steps used in [Avdeyvv] et al. 2023). We then perform density estimates of these data and plot the error to the exact samples. We plot a x6 zoom into the vertex A. We the SDE struggles sample
the corner. We use ; = 3, न = [0.25, 0.4, 0.35] (c) We plot the accuracy of our approximation of the infinite series GG zo, σe) including different numbers of terms for various values of [0, 0.2]. We choose ab = B = 4 and म uniform, and plot the relative error for two values of Zz,zg We only use the series approximation for r 3 0.05 (grey), which allows to only use 80 terms. Meanwhile (Avdeyev et al., 2023 used 1000 terms to accommodate smaller 7 (red). Our error threshold is 10-6.
Loss computation The complexity for computing the loss involves (1) O ((Bimax) computations for the series Fg and Ge. and (2) (BB) computations for computing the loss given the vectors s. Crucially, the complex calculations involving an infinite series occur in (1) and can be parallelized across B allowing massive acceleration on GPU. kimax should become very large as / becomes small, leading (Avdeyev et al., 2023) to choose kmax = 1000. Instead we only use the series computation for 0.05, allowing to use kmax 80 (Fig. 9c}, and compute O(B) ELBO for r < 0.05 in App.
C.3 Low TIME REGIMEN
When is small, sampling from A( cb. TT) or calculating Gø: F& become unstable and can require unbounded compute. Griffiths (1984) suggested Gaussian approximation for A(v,, T,) which will also use for deriving stable approximations of 5 zo, t) which require bounded compute.
Sample noisy It scale with 7.
We copy the following from Jenkins and Spanò 2017). Note compute does not
Algorithm Sampling from ancestral process A( Tr) Low t approximation
1: Set 8
(1b 1)
2: if 3 * 0 then
3: Set
8/(e@ 1)
4: Set μ
2n
Tt
5:
Set 2² -
2n
(1) + 2)2 (1
7+3 24) g-2
6:
else
7:
Set
2.
Tt
8:
Set 2² 3r,
9:
end if
10: Sample Z ~ NN,, 02)
return m max(0,, IZ 0.5l)
Round to nearest nen-negetive integer
Compute loss The loss in this regimen, even with the Griffiths approximation, becomes intractable; instead we use the Griffiths approximation to simply bound the loss.
18

---

Published as a conference paper ICLR 2025
When is small, To is almost always b* = argmax, We therefore set 2g = ōg.. In practice, in our protein setting, we only see bs y argmax,55. with T₁ < 0.05 at a rate of less than 1 in 7 10?. Since 20 # 6* is so rare we only aim find loose bound. Calling ū H we bound the loss by
$ ft (1 ((
No, 1) - ū Diigg B7 + |8 b0, ((7)) Ding(i)
-55)) -iu
Tt (w zo: / Uoo + w(b*, J) 6b+
In the next proposition we give an alternate formula for w( 8) which will allow us to Griffth''s approximation and a saddle point approximation to estimate w 87. It will also allow us bound w(ro, U). To our knowledge, this strategy is original.
Proposition C.2.
w( 8) Uza BB Ba Frg ml
where aa, is over the weighted, normalized distribution p(A(U, = =m)) (0))
(Wo $00
Proof. Inspection of first expression of the proof of Prop. C.I
We can now bound w(2o, ū) v w(b*, 8) & Uep Bi where p Tzo A Toa. Therefore, we get
2 mm).
Now we only need to calculate Ei,p to do so we apply saddle point approximation to Griffith's approximation to get Eqn. Jel... Note compute does not scale with т₁-
Saddle point approximation Let's take the Griffiths approx as becomes small, so w, N(,, where μ, a are form Alg. 6 We can use a Stirling approximation to get
(m)) x011 0(1/m)) (1 (pp)(m)) =(1 ((1m)))))) + m, - ()
(1 p)db
(op4m,-1)+1/2
(sb + m4 1) (1-p)))
vpp me -1
We take a saddle point approximation of Bi, i.e. take its value as the maximizer of the approxi- mate log likelihood
C We therefore get the approximation
(mm
2 (m= 100--wa3 21)2 + p) log( + me 1) (11mm))
E, x ( (w - 1)) + ( (11 4(1- r/ /2.
(4)
Noting mh Tr this approximation has relative error roughly O(7²). And T. 0, and r 112 so
E ~ p R 2/TT-
C.4 TIME REVERSAL SDE
Reversing the SDE Eqn. 5 using the result of Anderson (1982), we get
dz
2 i Σr)
B(1/B Σz) - (diag(z=) ž-) E V Pogg(ET t) dr
diag Zvzz) (I-V V dw,
where ž, Zy. 5(Ø To. t) approximates E s log Fu, V p( t). meaning can substitute it into the place of V log p(ū t). We sample by discretizing this SDE and sampling backwards.
To perform classifier guidance conditioning ( t). In practice, we perform the classic one-step approximation V logp(y rr) R V log Exo-nn I zo). If we have classifier f(2)) p(v Øo) then approximate Er. Seonguu 0 20) using the "one-shot" prediction f(āo).
variable y. we can add V log xx) to
19

---

Published as a conference paper ICLR 2025
D TIME-INVARIANT DISCRETE DIFFUSION MODELS
Zheng et al. 20244, Ou et al. (024 and Sahoo et al. 2024) noted that for masking diffusion, is necessary to pass to the neural network has iame-invariant" parametrization, Zheng et al. (2024) suggests this makes masking models a fundamentally different object than other diffusion models: "we reveal that both training and sampling of [masked models] are theoretically free from the time variable, arguably the key signature of diffusion models, and are instead equivalent to masked models." " Our sufficient-statistie parameterization shows on the contrary that every diffusion model can be made time-invariant by a choice of parameterization, with masking as special case.
Does this suggest that every diffusion may perform well masking diffusion after this choice of parameterization? Amin et al. (2025) suggests that masking performs well not because of its choice of parameterization, but because of "schedule conditioning".
Time-invariance is function of parameterization: Masking is time-invariant due to a choice of parametrization. To see this, imagine applying time-dependent rotation to each u we essentially performing the same diffusion but now must also pass to qo so it can "undo" the transformation. The á can be thought of as automatically transforming so fd is independent of time in any diffusion model.
Masking uses SSP: Indeed the SSP masking diffusion, 3(х () 6z, 2t *f mask and ő(अ t) [ .... I otherwise, is exactly the canonical parametrization. Thus the time-invariang of masking isn't special rather masking's convenient parametrization happens to be the SSP.
THEORETICAL RESULTS
E.1 MUTATION POPULATION DISCRETE DIFFUSION LOSS
In this appendix we derive Alg. 3 by showing it is equivalent to Alg. Namely, we assume D 1 and 2; is sequence of length and show
Predict de-noised 2o: the target of go(xo Fe, t), p(zo 2,, only depends on the vectorized H.
2(x')20.0) pla'(wo.)) pla, 20.0)
Compute loss: L Lntt on, Lor-on; FeD in 3
is equivalent to the form
plus (o.1)
Given prediction and loss computation only depend on o,, we can also replace sampling x, with just sampling He ~ Mult((, e giving Alg. 3
Predict de-noised w Simply note
p(zo 2r, t)xxp(z))))) t)
=p(xo) I[
(5g Tcc
x(0)
=-0
=p(xo)
[
(Te.)
b=1
Compute loss For sequences of length which differ in exactly one position, say x(*) b = (+), then La-za' La- and for every xp
p(x' Zo. t)
p(x
xo, t)
20

---

Published as a conference paper ICLR 2025
If x differ in more than position, then La-ee' 0. Call z
|= sequence which has all the
same letters as ze except has b in position z. Then calling p = z eT and q = c
pla'
vo. t)
p(a'
Co. t)
=
Σ F2
C -u,- ii
p(
Io, t)
p(zz
To. t)
Pb
qui
c (000 F,I
P (a)
#1z
(2) = b} Σ
Pi
9b'
Pb
Uxb
Σ Co-otiCTe,DD
Pb
96'
Q6
E.2 PROOF OF GAUSSIAN CONVERGENCE
Our formal statement the theorem adds some mild positivity assumptions for т, n and P1 which are satisfied by any reasonable choice of and almost every choice of L. It is also more specific about the limiting behaviour of 2 in non-dominant eigenspaces: also limit to Gaussian diffusion, but with meaningless embeddings sampled from random Gaussian vectors independent of zu-
Let us interpret the embedding Q1. In the case that L is doubly stochastic, reversible, и = [ B and & is symmetric; in this case Qı = jı P1 is just the orthogonal projec- tion onto the dominant eigenspace. In the more general case that L satisfies detailed balance, (diag(=)) 1/2 (diag(=) -=)) = V Tt Ciy is symmetric so Q1 is the orthogonal projection onto the dominant eigenspace of the "symmetrized" generator.
In more general cases, we don't get a symmetrized operator or an orthogonal projection QI, so we must "correct" for this with the adjustment (Q) -1)-1/2 which makes Q Q an orthogonal projection.
Theorem E.1. (Formal statement and proof of Thm. 4.1 Call > >
the negative the projections onto the corresponding left eigen-space. Without
eigenvalues of L and loss of generality, assume d1 1. Assume ft is bounded on every compact interval of (0, 1), To > 0 and P,b * 0 for all and p # P, for any b * b'. For each pick time dilation TÊ = log (5e2Ti ( + 1) and rescale क v/- (5 1)e (x) x)/ Define the embedding where Q₂ = diag(T) 1/2 P.diag(T)) and ję is any isometry into grank(R.), Q+ = )((
from (m,) - (P.)
Fix an To-
(Path convergence) Call (Z) the paths with 3o = Q1 VT) evolving under the Constein-Uhlenbeck process dē + V2dW,
for a Brownian motion (W+)T= and call E, Zn Then (Q1T))EE00.) converges in distribution to (Z+))e(0.1) in the sense of Lem. E.8
(Convergence of nom-nnminant directions) The component of zh in KerQi is Eis Qctt. Each component (Q,4)) ) also converges to a Gaussian diffusion independent of To with modified time dime-iiation and scaling: call (Zt)_o the paths with Zo NN, I) independent of xo evolving, forward and buckward on (-00, under the stationary Ornstein- Ornstein-Uhlenbeec Uhlenbeck process
dž, = - dr + V2aw, for a Brownian motion (W,)7- and call غ, 2.(0 where 7(9) ds log(e2t 1). Then
((1 - 22: 1 Q (E((,1)) (ā))
21

---

Published as a conference paper ICLR 2025
Call the ELBO in Alg. 3
L L, To. Bo) = Σ
Lbg-b t CTt,,, (o) ((0(2))> b2 (10((80)b2.b))
bi tbz
where Tr (8) is the inverse of the transform from Lt to oh,0,. Then, for all U, To. To
Tye -2n
e1- e- /emb(ro) emb(
-
the ELBO in Alg. which, particular, is independent of the value of d.
Proof. We prove the convergence of paths using Lem. E.8 which makes use of standard techniques. We break the proof up into four sections: the first three verify the conditions of Lem. E.8 and the last shows the convergence of the ELBO.
Part Convergence of Marginals: Note
Z ~ " Eo + V 1 - 2m
We want to prove convergence to this quantity. Note, writing Mult for multinomial distribution,
V (C - 2)
Ml - {म) /V
=(1+ ((1))/1) - 2- (5- (/uut( c - o o )) ( - /V7.
The second term is
v1 - e- 1/1 ( - it) = =v1 - 2 Σ 5 - P,
C(1 - 27x)
P,5o
(C(e22 1) 1)
- PIã0.
For the first term, need "uniform" central limit theorem the underlying distribution changes with Ç because of Eof c Lem. shows that (-1/2(Mult((, #0 e c C) z et c) approaches N(0 diag(p:) p for p o e c, which itself approaches म as o co. Therefore the first term, divided by / approaches
V1- -A,1-vVV""
Note Q. = =V V Therefore, as desired,
P. = 0 for each i and, for i > 1, Q.(PFF/VVT = V
P.P. = 0.
Qiag 2 (0, "mmb(ao),
and for i > 1,
(1 (1-e 21) ~ 10,
Part Local uniform convergence of conditionals: Note
17 (r- Z, /1 2(n N),,1).
We want to prove convergence to this quantity. Note
Te Σ Df
where H. = v (C - 1)e + π are the "unscaled" versions of the vector and Z, is similar. It will be convenient below to extend this definition to Æ for which (T.,s are not integers, but which still satisfy E, / V O. To do so, just round ÇT.,6 down to [[7,,o].
22

---

Published as a conference paper ICLR 2025
Fix J. We now show 251 = 8 ~ 2 = o; a very similar argument also shows 2 ~s Zg. Call xx
variable distributed o
| U7 so, calling
wf vS- - 15- 17e
5
No. /TTib//f /u5 CTTb
c ~ Mult (1N5s) , (0 independent across b,
then
CS, -c /
N N,, (5 (5600000--9--=))) m1)) /VT
Σ [( - No 6
noting E.pE, = Ç. This is exactly the "noise, signal" breakdown we had in the proof sketch.
For the signal (second term), first note
Σ (1 - -) 5" (f =#= 0,
so, ignoring the ı term in NS,, the second term is
Wl
wt w. Σ v Th (b c(T +5) #) N
oo /VTT
1 c-ee-2 (1-11)/2
= =[1+
Σ
(
e-2 27c
For the first term, again apply Lem. E.9
noting NS 2.b (1 o(1))(T to get
Eui(Ch
NS. No. 50 (78-/
b
v VTN (0, diag((' el elrí-1511) - elré -2-2- ( c(5)-))) W
1 2
2 (0, diag( clf (59--99)) - (1 -5 (+5 //7
AslTE (9) /V =VI-e-21 0. 0 Ce + -
-V1- 2
Σ 0, didigg())
- d.iii 77)
i
Therefore, as desired,
- e 2r.
Q1 5 = ~ e Q10 v (1-
2( [7)) N(
27.
- Q10+ vI 2(t N(0,11
23

---

Published as a conference paper at ICLR 2025
and similarly
1 1-
11/2
(1 10-e-2n)-122000
155 =Ü~
e e-Nilte-ou(11 (11---11)--
-
1 1- d
1
= (7 ( --(²) ((1 - E 2m²) 1/2
+ V 1 - c - -r.") N((01) 1)
Finally, convergence is clearly uniform for nearby using the uniformity of Lem. E.9
Part 3. Tightness: Pick s < e (0, 1).
E E
=5|12 EEERTTE -551 + 11 -LE7119]]
The first term has, for each To.
B| [
-512 El we (I, (T8 +500 -)) - 02512
1
EE Wo [el (F 155 -5 +t)) -(w, - (,) 7)1²
ming Tbb
[well -1-5) z. 1 + lwa, - w/ | - #1 2
IE
ming T5
[Iwell(F - #)²(T -((5ー-5)))) -+و)رر + |w> we || デ
E
ming Tb
JE [[w:((1 -ts -=5)>>))| ना + lu., well न 2
ming Tbb
(1wr((1 E - (rg -1)) + lw,, - wel)
LE F. - F|
ming T56
S
(1- 61-6-
(té (r)-))) +11 - We
ming, Tb
We
EETrCov H6 e)))(() + | c5 c 71/²)
x
2
(11---(r---)) + +11 111
Wa
(1 + Se -+5)
miny Tbb
wi
1
+ 11
))²
2
(1+
1
ming Tb
1/
cZT 1
Now,
1 (78 1--- =1 1-1-e-2autt -2hu(7--7)
é 27, (1-5-) 2m (1 - (11)
AB/2
<1 - E (7
+1 1 - E -- (1 (1- Ag/2
1 27 1 -
(1--11) (1-
When |T, Tel < 1/4XB
- e (1B(7 & <4AB(T - Ta) < 4 - s| sup ig.
we[s.t]
24

---

Published as a conference paper at ICLR 2025
Next note that if > 1, x 1-x" has decreasing derivative, from to -a on the interval 2 E [0. 1].
so, it dominated on this interval by a(1 - x). If > 1,
1 - e 2T. (1 (1-5-1)
An/2
1
25 25, (1
(A
$1
e 2T: (-1-
1
25 (1 5-
1 - 2T. (1 5
(11 (AB/2))
1
c 25 (1 5-1
(c-2t, c 2rx (1
<(1 (AB/2))
1 - E 2
1 (Ag/2)e-
27,
1 E 2(T(-Fo)
--22
4 (2AB)e
2T,
sup fu we| [e(]
1 2T:
Finally
) e- - (-1)
1 - c (1 1-
1/2
Wg
1
W4
which is similar to above.
The second term has
E 2
20
EM 8 clt el - xt))
mins T
2
Σ Er
Tot (+ - 8" -
ming T&
by
2
Σ
6"
c cr
15) " Beelts (5clt5-53 -+5 -5))
min; Tb
4
Σ - 1J c(( -5)16))
ming Tb b
4B
(1 e (rf -T))Am)
min&
which is bounded similar to the first term
Part 4. Convergence of the ELBO: Define p = Fo E. We've shown above that
p = ने + V ((e27e -1) P1šo + ols 1/2)
so
pb2
The
1
1
1/2
b2 by Tb;
T
pa 1/22)
pb
Tbe
Tbi
C(e27e - 1)
and similar for q. Using a second-order Taylor expansion on IB, we get
1 ((i-
2
D (10(z)) bn (100(6)))),))) = Thy 2 To, nh, <(c2T) - 1)
The by Tb,
Pi Pi - 300
+ +o(5
2 Next note #f p o(1. Finally note
Z((0) V - 1)e- 27c + π n o(1).
25

---

Published as a conference paper at ICLR 2025
Putting this together, we get L(ū,t, To, 2o)
Σ
bby t (T)
Pbg
Sb2
pbl
Ub
b. xb2
2
= Σ
-b 2 -
2T
1
(e2te -1) 1) ((x-
1
Tbe bi
Pi ( -00) - +
2TTh Tbi
Tbi
bi bi
i
Σ
bbz ((60-)
The bi
Q1
(To zol/vA)) o(1)
2m -1
Thi
by Hbb
Fie -2e 27
O0x ((Tt zto/V²/
11== ss o(1)
(1 - e- 22.232
where
Σ
1
Σ Lbg
52 V
Tbe Tb bi
82 V Tb.
Tbe D1
T
2
To:
by #bb2
To solve E, we note
Σ
Lbg-bb b2by
E
br
Σ
Lb
Σ
b2b2 Lbz
bi bg
bn
bi bbbb
by
Σ
The Lbg bby T22bh2
Σ
b, of
The Lig -b1
Σ by oi bb
Tbe
Tbe
61 bábz
b2
fr bbb
T2 -by b2
diag(v (VF) (F) (C diagC) diag(1/V
TDi
by b1
Th2 Lbg b1bg = (diag( v (C diagC) diag(1/ l/V7))"
Th:
by Ab2
So,
Σ =
diag( (v Cdiag 1/VR) (diag( (v (ong(VV)) (diag(1//R))
2
2
In particular, since @fdiag v 7)) Ldlag (1 1/V QQT
--' = QI =QQQ.
This gives us
-
L(U,t, 30)
(1 e 27 (emb(20) emb(
E.3 HOLLOW PARAMETERIZATION SOLVES GAUSSIAN ELBO SINGULARITY
Here we show that the hollow parametrization introduced in Sec. 4.2 resolves the singularity of the Gaussian ELBO in Alg. 2 at t 0+. Before going into the proof, let us give some intuition. Assume, zy were distributed unitormly and independently. Then
p(ağ xe, ()xpp(a ag Op(2 x .1,,
where x; includes all positions but d. However
p(æg , t) = plag dpp(2 d 2
= Uniform.
Therefore, we get p(ag x,, 1)p ag t). At initialization, we can say our neural network ge(ağ) x, () Uniform, so,
go (x6 21, p(æ6 zt,
26

---

Published as a conference paper ICLR 2025
Therefore, the hollow parametrization initializes the diffusion model near a uniform, site-wise independent model. The proof below involves a lot of algebra, the basic intuition for why we should not see singularities is that by initializing at valid diffusion model, we get comparable ELBBOs.
Again assume D for simplicity results are straightforward to generalize to higher D.
Proposition E.2. Assume emb is injective and Ti is increasing and differentiable. Define
tge 2r.
e1-E 2r, emb((2)) emb(
and the normalized vectors o(I;, t)ocp(xt zo t). For To build using the hollow predictor To- o(z,,t) - q/o(x,,t) t)² for a vector bounded away from and co,
0 c min Tb max q6 < C 0C,
we have
Eg Et 00.
Proof. Note first
- - emb(2o)ll & |emb||31300 - Tol
and, simplifying H(r, !)
E Exolar 1:0 -5ol-ol---
11 3/ 30 // r
for pb = p(zo).
Call b argmax; so
33000"--000000
bbPb
obth
+
66)
Σ
Dy
To
T p
T p
T7 g
8xb
1 +
bb
Fv.FN dapb
1
Σ b
By 4s!
9%.4
C
+
B(1 - ob)²
2
1
CB(1-6a)
1+
CB
(1- 2b) ( B(1 db)2.
We've therefore bounded Er, above by some constant times Et,,, (1-e-25e)) (1 max; 2
220-27
Note without the hollow parameterization, we wouldn't 't have the (1 - maxx 96)² 2 term; we now show this becomes small very fast as /. - (because un becomes "obvious" from 244), cancelling the singularity.
Next note, calling b llmb(() l.l.
(1 max 2
1 1 Eves expl - Xा- ( (lemb(()) - F |eebb()) - (2))
2(1 - e- 2 (llemb(b') æzll ||emb(b) Fe 2)
cxp
(T
ydb
27

---

Published as a conference paper ICLR 2025
which is only large if ž is roughly equidistant to two potential 20. Call minb l|emb(6) emb(b))1/44, so, miny emb(t// / < e then, by the triangle inequality
lemb(b') Till² (emb(b) - æzll² > (|)mb()) emb(b')l llemb(b) - Fzll)²
lemb(b) 3 1
lemb(b) emb(b')l
- 2 emb - emb(("))) -ž,
> 16c SE² Sc".
Therefore, Bi,z1 (I-e 20132 (1
te - 2
max 96)2 is bounded
t,e 2
4e²
+ + p(mins/lemb(b") Eel 7 c))
BE,
(1 e- )2
27x
exp ( (1- 10-e--2))
To deal with the first term, perform change of variables
(1 2 giving
Eg (1 -1-65 El---22.) 2n.j2 exp (- (1----27) (e-2r) =
25,
4c²
Jo
~
du du -42pp u) < so
Jo
For the second term, note
p/miny/embl/ - Fell c) pl b)p(II (1 25 2t >)
b
=p(x:/e 1 2
where X² is chi-squared distribution with degrees of freedom. Finally, by the same change of variables u as above, we get
tH ES (1-e e 2 ya mmmm/mii/emb((// - Fz
I 2
1 0 duplx? > u) =2 _ < co.
E.4 EVERY EMBEDDING CAN BE INDUCED FROM SOME INFINITESIMAL GENERATOR
Define an injective embedding emb fl,..., B} - R" for some r. For an infinitesimal generator L with unique stationary distribution म,, define Q1 = iá -1/2 jı is some isometry, Q = diag ( 1122 Pj didgg 1/2 where P1 is the projection onto the first left eigenspace. Is there choice of L such that Q (To/ (...) emb(xx) for every 2o?
If we restrict to L E BB then the answer is no. To see this, call W E Bxr the matrix with W = emb(b). Then, defining D = diag(#) we need WTW = =DQIQTD= = DPD for some orthogonal projection P or rank r. The space (WTW W e [ generates all rank-r positie-semiidefinit matrices, an algebraic variety of dimension B x Meanwhile, P has r(B - r) degrees of freedom and D has
1, so DPD generates an algebraic variety of dimension at most B - + B - 1, which is less than B x r when r is large.
If however we allow r + 1 "dummy" tokens, to let L E (B++)))(++r) then the next proposition shows that the answer is yes. This demonstrates an important distinction between the design space of Gaussian and discrete diffusions: dummy variables which never appear in the data have no effect on the training of Gaussian diffusion, but can serve as transient states discrete diffusion.
Proposition E.3. There is some infinitesimal generator L [(B+++1)x(B+++1) such that @To V emb(x0) for every Do (1,..., B). There infinitely many such generators.
Proof. Call W I Bxr the matrix with W emb(b). Call A WTW and without loss of generality, assume its first r rows are linearly independent. We split the proof into two parts: first we show that QT Q1 can equal DPD for any orthogonal projection matrix P with PV7 = 0 and D diag( ()) -1/2 for any distribution then we show that A can be written as the top B B submatrix of DPD for some choice opf D and P. This will show that there Q1 such that (1.// is equivalent to emb up isometry.
28

---

Published as a conference paper ICLR 2025
Part Pick an orthogonal projection P and distribution such that PVT 0 Call P diag ( 1/2 and
- (1-1-17) 1 uPl
Clearly, for every μe, 2, = 0 and म" Lr 0. Also, L1 = 1t so for in a neighbourhood of 1, C, has positive entries off the diagonal therefore it's an infinitesimal generator and Ly has kernel of dimension 1 so is the unique stationary distribution of Cy.
When 1l is slightly greater than O, the first eigenspace of Lu is that of P; in particular, when P is projection, Pi pr Q1 = P. Note Qf orthogonal complement of KerQ1 KerP; therefore it is equal to P.
which is the projection onto the
P and म are the same for any small value of μ. justifying the "infinitely many" proposal in the statement.
First we need to ensure the rare case that 1 is orthogonal to the top eigenspace of A does not occur. To ensure this, simple add another embedding emb(B B + Σs (hemb(b) for some i to get a new matrix A adding this extra token:
Part 2
A
A&
A
(Art)
TT Au
Pick 7 so J" Ač * 0 and w== nÝ. As 00, A/12 T" Ae(ee") where is the indicator vector for position 1. Therefore for some 17, the top eigenvector approaches and is not orthogonal to 1. Below we simply assume that 1 is not orthogonal to the top eigenspace of A.
Decompose A = r/V diag( (x/n) for matrix V E Bxx with orthonormal columns. vector À of cigenvalues, and a scalar 17 > max; to be chosen later. For orthonormal matrix RR be chosen later, define
V Vddag(A/ 1/2 U(( diag( A/1))
so V has orthonormal columns. Define the orthogonal projection P V in particular, the upper B x submatrix of P is A/1-
Finally we'll pick v and U to get positive normalized vector म such that मo 1/1 for all b (I...., B} and V 0, completing the proof. Breaking मे into its first B components and other components, [1/m, #2). we can write the equation VT V म 0 as
#2 -17 -17
3 diag
VT1.
X/n
We can always choose U to rotate to get मे2 = 1n/ where
1/2
n/ =
diag
/VF.
T-
X/n
Finally we need to solve for v in
1/2
X
1 = B/n 3
diag
VT1
1
X/m
This is possible by the intermediate value theorem as the right hand side goes to 0 as oc and goes to oc as 7) - max; A; from above (as we've assumed V of for where ) is the maximum cigenvalue).
29

---

Published as a conference paper ICLR 2025
E.5 PROOF OF WRIGHT-FISHER CONVERGENCE
Define A B c R be the simplex, i.e. the set of non-negative vectors with components summing to 1. Let (of) be a stochastic process on (1 n A with Ty = Tp evolving with respect to f {L#t where
Ç II S
I ( (ult(
and, if F,, F differ by one count b - b',
Came
-5 (1(1 (1(17" 1)),
Div
otherwise it's O. Let ( be continuous Wright-Fisher process, that is, E -5o and
dž = vb 2 (
zt + diag (V (I- VaVVEV aW.
(5)
where (W)), is a Brownian motion.
E.5.1 CONVERGENCE OF THE FORWARD PROCESS We have convergence of the forward processes from previous literature.
Theorem E.4. (Thm I.I Ethier and Kurtz 1986 Chapter 10)) Assume L = ub (1#T 1). In the
topology of convergence of compact sets, (3, )eE(..1) ~ (Zr))t((.1))
Note when B = 2. (Z)), distributed as the Jacobi process described in Avdeyev al (2023).
When B > 2 Avddeev et al. (2023) considers B - 1 parallel Wright-Fisher processes with B = 2: they then use stick-breaking procedure to get an SDE on the simplex. This SDE is distinct to ours in Eqn. 5 and is not symmetric to the order of letters it requires us to specify a first letter, second letter, and so on, which behave differently in paths (zz)² except for at stationary. We instead directly consider the multi-allelic Wright-Fisher from Ettier and Kurtz (1986) Chapter 10) which is invariant to permutations of letters in the alphabet and simplifies our derivations.
E.5.2 CONVERGENCE OF THE ELBO
Call 5(ū mn) V log p(2 1,,-- and 8(ū To, 1)
Es Zo,5ā(ū wo = b, t).
Theorem E.S. (Proof of Thm 5.7 Call the ELBO in Alg. 7
LS k0) = Σ (CL
Cuut J1
pl p(ztf
Io, t)
Σ Too
p(x,
20 b,t)
p(
ro: 1)
p
To
b,t)
For for which Ç are not integers, define L(U, t, No 20, = LSS t, Fo, uo) for a x nearest to
Then, for all T in the interior of A t (0, 1), To E A B and To.
L' 1, Fy, T
2 ত Tu t) ल Tos 1) Dainngg--07 Ā7
2
Proof. Overview of proof: For notational convenience, define
1) Top
zo = b,1) p(
D( ID
Σ
p(
To. 8)
p(
To = b,
Much of the proof consists of checking uniform convergence and regularity conditions. The main idea however is that when ( very large, the transition rates (L#t -2
Luut
are only large for 2
F very close to J. For those terms, we can perform second order Taylor expansion
D(E > | ta,t)
(( Ro. 1)1
[[] - -0)
so
* | Fo. t) T6 5 t)}
L (S (U, T0) R
2
30

---

Published as a conference paper ICLR 2025
where Σ
+2 CL
+
L May >
(2 - ") Finally, we show
8f -2
51 -
EEf 2aS EB -55
(It
(T( - 8)
0, and through a central limit theorem,
EES+EE (L -,
(xt
diagö Ue7
Crucial to our proof is Lem. E.10 which states that for each oे in the interior of the simplex,
p( xo. Pm--a Bg- Mult (()
for distribution A (1), Te) such that (1) (m) A(ú, (m) quickly for each m & Ç as
< 00.
Part Eliminating the boundary For a small é > O, call A the points in B that have an entry less than e; in particular, define < (4B) 2/ ming nb We first show that the contribution form the Hooundnyy vanishes, i.e.
B(S)
Σ (S
2/
+
Cantt E
)ill - 0.
5 EAG
First note for large enough Ç,
- 2. -
= 0 for all E $ A and
LW
(min I.S mins (1) C(c C(22minn min, <
for some C >> Also for any ZT.
>> p( To. t) > p(A(v) TT) = Di (1))))
Taking the leading term of the divergence ((I((),
log Du
E(C) E Σ (4B)
Now Mul > (ming ps) S so the denominator is > 7m 122 Therefore
< (log(2B)
E(s) SS
Σ
(2B)
<2 5 5 B-1
-0
since there are B-1B elements with o $ A
Part 2: Uniform convergence of the likelihood Next we show ( -E
1 + O(5-1)
uniformly in A. While something like this is implied by the convergence of the process from previous work, the fast uniform convergence will be important for our results below.
We will do so by showing the same property for each of the quotients
E AmmA Exp Mult( ( Em Di (Ff
Em-Als Dir(wi māo)
Em-A(y.)) Dir (T)
The first quotient converges by the concentration of a Bayesian posterior Miller 2019). In particular, by the uniform Stirling approximation (Robbins)
1955) uniformly for x.
6/2
-1/2 (2 -(B-1)/2 -CT22/21)
[
M(lt( =(1 + 0(5-
31

---

Published as a conference paper ICLR 2025
We'd like to write this as approximately normal density with mean Ø and variance restricted to vectors summing {1ū IR
n 0). We can do so with a Taylor expansion; for p near 2t5,
K
pp)
2 | - - ding( O(1 -)
We can also write | - pl ( = 1,% to vectors summing to and
pl where = diag
F has kernel orthogonal
A = diag
1 x 1
-
1
1
111
E. t.b
11
B
B
B
Note also that the pseudo-dererminnnt of is II.
oo so we can write
0 -(-
(S -(1+
This allows us to write
Bp ()) -1+ -1+ E (
p(a))
P(
For a small 8 < c/4 call Φ a C function with support in the 5-ball, and which is in the 6/2-ball.
We break the numerator up into
1/2 EB--((AA) - 665 P
P(
0(100))(111)
The second term is less than
Epgn (1 - ol - (1) - p)) <pPT 1( - (0, D))
Le CC82
for some C. For the first term, we can define P(p) = P(p) x ( function. Therefore
which is a compactly supported
No P(T) + < 1/2 O(1 11111//(112) 2/(212))
= -P(z)) + V 2 NB.
1/2 -((1101))(
= =P(T)) + ()
For the second quotient, note the denominator bounded below for A. By Lem. E.10
sup Amm)A)))))7))) Dir(pii qiन + mT0) m77) (m--a(w.7) Dir( + mão) (1) (1))
SC
cm?
sup +ir mā))
Te J B
Since ) Dir(ui + (T) (m (1- 2/2 m- is eventually decreasing in m, the whole quotient is O(5-)). Next note the derivative of Em A(0.7) Dir(v= + mā))(-) bounded on the compact set A B so
sup a
Am Dir -Em-))))))))))))))))))) Dir( mā)) (())
= ū]) - O(.-
-
Part 3: Taylor expansion of the divergence Given the calculation above, for large enough and any E = ( + O((21/2), we can approximate
Fo. t)
exp (logp & = Et{ zo. t) - logp (== (x) No: :)) + O(5-
p(25 (i) xo,t) t)
=1 + कū No. 1)T ( - 0) + O(s-
32

---

Published as a conference paper ICLR 2025
A second order Taylor expansion then gives
D
1
(s( 0 zu, t) $ Cu. t)) (x)5 - 8)
o15-1)
= 2 [ To, t) - ड To. t) (ل (, my + o(5
Given the calculation above, we note that since Cunt with E = (0) + 0(5
of =5) is only non-zero for values of o each
Σ
Cmut
O x-5) = d(1) D(=
2: -
#
This gives
(U, k0)
To, t) 5(ū To. t)| + o(1)
2
where
== Σ SC z -ay
EAB
The proof is therefore finished if we show Σ - diagē Je
Part 4: Convergence of Σ Note, by the uniform Stirling approximation (Robbins) 1955 uniformly for J A the infinitesimal generator approximates a Normal distribution near i
-1/2
(0) =(1 o(1))
[[ 0h
(2m() (2m()-(B-1)22 (B-1)/2 (BLLI)//-K
=(1 + o(1) + O(C] 1' 1 (8.5 (diag( (T) - -58)) (F*S)
Noting that, by Pinsker's inequality, KL ( > 2 - 19l > ū x< 12, for some very small
5 > 0
Σ (diagT - 1
Σ
5 (B-1)/2+ - (KT
EEA -
2S (B-1)/2+ $CC 2/3-26
=0(1)
E.6 WRIGHT-FISHER LOSS CALCULATIONS
See the discussion above Prop- C.1 for definitions. Proposition E.6. (Proof of Prop.
p( To. t) (it
Zt).
For a(T+)) V log Dirichlet(riin mpp b) (Tx) which does not depend on To.
= s(F. zo, 1) = o(Fe) + (ow(
where
w(xo) v- 1b + 1) Fo Lo. Zt) T(za)
GT, Ft)
Proof. For m, A(v, Tt),
p(āe Zo. f) = Emm, Dirichlet(uE + mi
= 11
Em
P(UTED + m)) I Iozzo I(UTT))
I( + m))
21,50
Wi +m
bot220
r(*)
((UT(Z))I(()) + m4)
soti - -imm I(b) (VUTO m,) LeD
1he
(((1)Tb))
bEB
(+) (ms)
Dirichlet (T. (17 (Io) (Xo))
33

---

Published as a conference paper ICLR 2025
From Eqn. 5.2 of Tavaré 1984), we have
p(m+ j) -(++v (-(t)- (2k + vb- 1)( (4-1)
jl(A: j)!
He wrote, in Eqn. A5,
Σ x plm 5)
8
x (j + (k-1)
Σ
k(k+v ((k+0-1)Ta/ + vb 1)
Σ
j! (k - j){
j=1
k
x² (1) (w))++R-)((kk))
Σ
- - 1) c 11( + +
Σ
j!
klb())
j=1
1(k+v-1)),/22 2 + -
-1-1))()) ()(()
Σ xt (vb + k -
k!
j!
2())
The last sum is then written as 2 Fi ( k 1/ + k 1; - for the hyper-geometric function 2 Fi. A very simple extension gives
Σ (1) p(m, =) (1)
k(k++(+ -ー 1)6/2 (- 1)x (2k + 2 (x())
k!
(2
Fi
1s - 2o 2) 1).
Including the j = 0 term, by Eqn 5.3 of Tavaré 1984), cancels out the -1 in the brackets above, so our expectation
(me)
2 27,000 =1 =1 +
k(k+0 (00-t--1rr/222111 (-1) + vb - ()
Ema (1)))))
k!
2 k, 1b k- 1; 1pT )
Gb To, Fz).
Finally, using identities of the hypergeometric function,
x
# 1)7./2 (2k (2k + ()())(=-1) -
k(U
Villao G No, Tt)
=
k!
V/TzO
2 1/ k; oT
1; 20,))
1
k(k+ 2 1)r/22 1) k- (2k + sb - (z/) + -
()(0)
1
(k-1))
x 2
Too + 1; 12,0))
8
(2k + + + 1) + k))())))
(k+1)(++))
h!
x 2 (- ls 1; 17 1; 1r,00
e v-1222 (1b + 1)
(-1)((b)()) (2k: + 16 + 1)( + k)
x(k+01))r
(2 1)
k-0
x 2 Fi ki 1/ + k + 1; 17 + 11
e- 02/2 (1b + 1)
F (t, No. T.).
34

---

Published as a conference paper ICLR 2025
E.7 PROOF OF SUFFICIENT STATISTICS
Proposition E.7. (Proof of Prop. 6.1 diffusion process or such that
There is a function F depending on p(zo) and not on the
p(z 24 2
Proof.
r(ağ Fc xt
p(ug vo "dppp= xt
plag to )((t so ddp(
g)
plag) xo
II plaz"
wi ))p(x
plain)
d'd
Ilw Expla
ax b)
pl p(x?
plat 20 ")
dp(=0
p(2T
Es plaff
d'tdd
=pplo")
II H(u)) )as ddd
/Bele
[I d'
E.8 LEMMAS
Our first lemma establishes conditions for convergence of paths using standard techniques inspired by arguments used throughout [Ethier and Kurtz (1986) or Bass (00T)) for example.
Lemma E.8. Say (5 ) are Markov processes on R' for C = 2.... and (Z)) is another Markov process on R., Say the following conditions satisfied
1. (Convergence of marginals) L.. - Ee for each t.
2. (Local uniform convergence of conditionals) Conditional distributions exist such that for each T IR, I, and bounded compactly supported measurable function there is
e O. such that
sup 17 lie
E 13g
EEJE.. 0.
3. (Tightess)) For every [a, b] c (0,1), there are 8,0, M > such that for all s, [a, b].
supç=M M B 1 < C(s -is-t"" t)
Then, with the topology of convergence on compact set the paths converge in distribution
(o)) ~ (Z))
Proof. Pick a compact set [a, b] c (0, 1). We show (5 ( ~o (Zt) Say (4²" )efja,b) is subsequence which doesn't 't enter neighbourhood of (2) [e(a88) we'll now show a contradiction. By Prokhorov's theorem, since it's tight by Assumption 3 and Thm. 8.8 Ethier and Kurtz (1986) Chapter 3), has a subsequence which converges to process (Te)eeja.) As we'll show below, for every set a & f < 12
< fm & b, (Te (e (t, (Z)ee))) This must mean (7r) (z)). by the Kolmogorov extension theorem, contradiction.
What remains is to show, for & t to sufficient to prove that for any t < ... < im and compactly supported continuous function on 1 h.
< fm b. (z) (e))). (Z))e It is
Eh(Z...... Zm)-
This is standard topology for these results. See for example Thm 1.1 of Ethier and Kurtz 1986
Chapter 10).
35

---

Published as a conference paper ICLR 2025
By the seine-Wresssssss theorem, each such h can be arbitrarily well approximated by product of i univariate functions, so is sufficient to consider .i. Em) Markov property,
mt he (5). Finally, by the
E
Fm
E E
11
(o1
IR 1
l2 (15)) ... E R Mm (T
We can call - ( (mm-1) Am (f Pigg 1as
Nm (m By Assumption
h - (m con-
verges uniformly to hmm 1) ( 1m- Therefore, prove Eqn. 6l it is sufficient show the result replacing h with h h2 hm-1- 1. By induction, we reach h for which we get Eqn. 6 by Assumption
Nm (m). a bounded functtin with compact support.
x hm 2m-22%
Our next Lemma is non-asymptotic bound the convergence of multinomials Normal distri- butions. It states that as long expectation a function by O((-1/2)
of and the probabilities don't get too low, we can bound the
Lemma E.9. Let Yg ~ Muli((, P) for probubility vector ç E BB with min, Ps 2 c Call Ze = 1/2 Cp). For any bounded measurable funcionll f.
= - -Ef2)) (c,
where Z (10, diag(p) - pp and the rate of decay Or, Or,B.,(1) only depends on e, B, and f-
Proof. For every c, pick a compactly supported C function J. such that I|g. fllo c/2, so
(f(Z2) - Ef(2)] = e (Z. - Egg = c Or,B.g. (1)
by Thm 1.3 of Gotze 1991
Our final lemma characterizes the distribution of the finite population Wright-Fisher process as described in Sec. [nd App. C
Lemma E.10. For each of in the interior of the simplex,
p(#5 Fo, (m B MultDit Mu
for a distribution over the natural mumbers (1,))))) TI) supported on { such that (1 (m) A(V, 7m)m = CC- oxp(-C'mm) for constants C, C' only depending on 4, To each m.
Proof. This standard the population genetics literature. Define ( T:) 71) the probability that m alleles survive backwards in the coalescent of population Ç up to time т.. Conditioned on observing m individuals with allele xp. Hoppe (194)) showed that sampling more individuals from the population is equivalent to sampling from Polya urn with allele probabilities पे + mmop giving the Dirichlet multinomials.
Tavaré 1984 shows A(p, (m) limç-co ( (sh, Te) (m) and for m
k(k+o ( (2k + v 1) + v)(k (5- - k: 1)(0) m!(k m)!
(19
(S+ +)
Note
(m + (x-
(m + ()-
(I: - 1)! (k 1)! (k: m)!
mm km
mm!
for some > 0 and
(5 - k 1)(M)
- S.13/..
(C + ()))))))
36

---

Published as a conference paper ICLR 2025
Therefore
+ me
h2
(4(9))) (1,)T)(m)) A(dss:))
mm) 2
1( + vb 1)h2m
k(k+
m! 5 ^1
R-m
m
Σ c
-( 2mm+
m!
k=m
<(-1
m
mmmmmm 1)mz//
Σ
20
(m++ 1)) + m) mm 2m+++
m!!
and
0 11-+//11/////3 + m) m)m 2m+
(j + m) 2m+
=
1=
Σ
é ji 12 + m) 2
+
<m (2 2m+
Σ
e -m (2
=m
<(2m) 20+ + 2
1=0
= -2m) 3m+ + +-
[j 4 vm 1))7 35
< (2m) <(2m) 2m+ + 2-m+
(j 10)
/=1
5(2m) (22m)2+++++ 2
F EXPERIMENTAL DETAILS
F.I DNA
We describe the experiments in Sec. 5.2
Training and data For all DNA models, we use the same base CNN model and optimizer hyper- parameters used to train DDSM (Avdeyev et al., 2023) and Dirichlet flow matching (Stark et al.) 2024) with code from https: /ttp:/ggtttbbbt http:uu.ocoo///hhhaaaa/////eeeee their licence and hntps:: httpp:://ittuu. htt.::/ccccoaaaaaaaa
used with compliance with fl oww cow-macciing used with an MIT licence. We train our Wright-Fisher simplicial model on the FlyBrain enhancer data from https: /wwodoo..ooooo//oo org wwt.reecods//l1123e0.0
We trained our model on an A100 80GB GPU over 11 h for 700 epochs like Avdeyev et al. (2023). We trained DDSM model using the code in http::::///gttuubcooo////o// and used a pre-trained flow-man matching model from dirichlet wellow-matching
com http//Hannneettkkeeaa
Computational comparison All three models we tested need to pass their noisy Fe through a neural network. We chose the same neural network for our diffusion model as used (Avdeyey et al. 2023) and (Stark et al. 2024). For a reasonably sized model, like the ESM model for protein experiments, the neural network computations took 75% of our compute time on average, meaning the overhead from sampling and loss computations cannot be more than 25%.
However the DNA architecture was very small, at only M parameters. For the DNA setting then we precomputed and cached T.. Fg and G so that majority of training time would come from the neural network. Indeed our model took 3 hours on an 1000 train for 200 epochs, comparable to hours on an A6000 for 200 epochs in (ttark et al. (2024).
37

---

Published as a conference paper at ICLR 2025
DNA accessibility (ATAC) predictor To get accurate predictions of cesition-resolutionn epi- genetic marker for DNA-accessibility (a property one often wants to design), we use the CNN bpAITAC model from Chandra et al. (2025) to predict chromatin accessibility traces with code from https: / / fithab...hh...............e com/ achandra/ /bbpp ITAC. The model trained on embryonic drosophila chromatin accessibility from the Calderon al. (2022) developmental fly dataset 16-20 hour subset, with ATAC-seq reads combined across cell types, using held-out chromosome chr2L for validation. bpAITAC was trained single NVIDIA TITAN RTX GPU (24GB) with early stopping based on validation loss.
bpAITAC produces two outputs: 1) total counts (a measure of regional accessibility), and 2) proba- bility distribution of the counts. The base-pair resolution counts prediction is easily computed by multiplying the two outputs. The resulting per-hase counts are modelled by Poisson distribution. That is. bpAITAC takes a one-hot-encoded sequence To of length D = 500 and predicts a positive 250-dimensional vector that represents the predicted "accessibility-Pron prrbbiilttypooolee"e"e in the centre 250 positions of the sequence. For a target profle of 250 numbers, y e N250, we compute the probability by using the bpAI TAC predictions as means of independent Poisson distributions
D I[ Poisson (Va) (Ma)
p(u
where J = bpAITAC(H)) is the output of the predictor. Since bpAITAC is a neural network which accepts one-hot-encoded zot, we may also pass Zg which have each position Zo,d lying the simplex.
Evaluation We use the @edelikelln diffusion model in the code of Avdeyev et al. (0033).
function evaluate the likelihood of the trained
We collected 100 trace predictions from Calderon al. (2022) validation chromosome chr21 to use as targets, picking the 100 peaks with the highest combined signal. For each target and model we sampled 10 conditional samples using 1000 function evaluations. We sampled from our simplicial diffusion model using the procedure described in App. C.4 To sample from the flow matching model in Staak et al. 2024). we modified the predictor that we used in our App Finally, we write custom code based on reversing an SDE to sample from the simplicial diffusion model in (Avdeye al. (2023). To do so, we note they perform diffusion in space with each position Ui_ E [0. 1|-1 We compute their prediction Zn (7) by transforming the output of their neural network and then compute a prediction of Ve, log p(yl 2o ( (v)) with one-step estimator in in A.p and add it to their score for 전 every step. We add this modification into their function Euler Later_Marrua Maruyama aaaaammm__mmrrrrr.e_aa_
score function in their code to return the one-step
To calculate (T) we note they build neural network to predict 5= Va, logg(T; which equals
Σ To, Ve, log logp(ū;/20 - b)
for some implicit prediction Co.s which we must solve for. Now note, by the choice of the re- verse stick- breakinn procedure of Avdeyev et al. (023)) ÜL (Va, log p(U, = b))sr =
Vo log,((t,,b Vo, 50,1) for 6' and (V log p = b))x VV log Bcta(1, B
8 (Tr,)) otherwise. So, 5 = U.o = - + - (1- (1-28, 1)) = -.:-1-U..111 )8..
U.-1- Therefore we can solve for - by solving this linear system.
F.2 PROTEIN
We describe the protein experiments in Sec. 6.2
Training and data For all protein models, we started from pre-trained ESM2 150M weights (Lin
et al. 2023) under an MIT license in MDLM (Wang al. 2024a). We trained with a learning
rate of 10 5 for an A100 80GB GPU over 48 h for 3 million sequences, substantially less than the
training budget of Wang et al. (0224)). We trained on UniRef50 (Suzek al. 2007) data from
https::/ /ttp///eeoooooooo org/ recee/reecoe records wtt.oordd///55475556
6564798
From each model we sampled 1000 sequences of length 200. We used a uniform grid of 100 points and integrated backwards, and we applied 4 corrector steps per predictor step as described in Camphell et al. (2022). Then we predicted DDDD'ss of sequences with Omegafold Wu al. (2022) under the Apache- 2.0 License, with I cycle for each sequence.
Evaluation
38

---

Published as a conference paper at ICLR 2025
F.3 LANGUAGE
We describe the language experiments in Sec. 6.2
Training and data We used the same architecture and training settings as Lou et al. (2023), using their code at https://github. ctmp louaaron/ under an MIT license. We trained our model on 4 A100 80GB GPUs for between 40 and 50 hours total on 33 billion tokens taken from the 1m1b dataset. We used learning rate 3 x 10-4 and an EMA of 0.9999. Our diffusion transformer had an embedding dimension of 768 with 12 layers and 12 attention heads. The Gaussian models used pre-trained BERT embeddings scaled by a factor 8.
Score- Entropy
For our individual discrete and Gaussian language models, each device used physical batch size of 64 and took 2 gradient accumulation steps for an overall batch size of 512. For our unified model, we accumulated over Gaussian and discrete batches get an overall batch size of 1024.
Evaluation We sample using 1000 iterations. Following Lou et al. (023)) we evaluate the sample quality of our models through the generative perplexity of their unconditional samples according to GPT2-large.
G SUPPLEMENTARY EXPERIMENTS
G.1 ANTIBODY OPTIMIZATION DOWNSTREAM TASK
We test our unified models from Sec, 6.2 on a different downstream task. Hie et al. (2023) suggested that generative protein models can be used to suggest mutations that improve the stability of antibody sequences. To test our diffusion model's ability to successfully improve antibody properties, we perturb a parental VHH sequence by noising UniRef50-trained diffusion model by then denoising with 128 steps 12 To emulate a realistic wet lab setting, we investigate sampling 50 unique single- and double-point mutants of the seed VHH by rejection sampling. We repeat this process 100 times, selecting the top resulting sequence from each repeat "experiment" according to a proprietary ther- mostability oracle. The amount of noising for each of the individual Gaussian, simplicial, and discrete diffusion models was determined by hyperparameter sweep over t e [0.01, 0.02, 0.05, 0.1, 0.2, 0.5]. where the chosen hyperparameter gave the most unique sequences with fewer than or equal mutations to the parental sequence. This hyperparameter was then shared with each sub-model of the unified model.
The thermostabilliyy oracle we used is an ensemble of 10 CARP/ByteNet regressors YYang et al. 2024), pretrained on approximately 537,000 sequences from phage display, processed using Next Generation Sequencing (NGS), and 9556 I'm datapoints obtained from NanoDSF. The resulting ensemble achieved a test cross-validated Spearman correlation of 0.72.
In Fig. 10 we that unification does not substantially harm performance this downstream task.
G.2 FITTING IMAGE DATA: MNIST
We perform the analysis of Fig. 7 for image data and find a similar result. We evaluate our unified discrete diffusion framework on the MNIST dataset, consisting of 28x28 grayscale images. We discrete the pixel intensities to N = 8 levels using uniform quantization, preserving the continuous structure of the token identities while reducing the computational cost. We compare the performance of our single unified model (SSP) to the performance of three individually-trained diffusion models: discrete, simplicial, and Gaussian.
All models blocks, and ReLU activations. Models are trained with the Adam optimizer with learning rate 0.001 and batch size 128 for 20 epochs. For Gaussian diffusion, map each class index E (0..... C) to a 2D continuous embedding with a circular parameterization emb(æ) = (cos(0), (0), sf)( where म. n. This embeddings encodes the similarity of different pixel values and ensures that the resulting diffusion process closely resembles continuous diffusion. We found models with 1-D x
U-Net backbone with embedding size of 128, 4 downsampling/upsampiii
parameterization emb(x) = 2 x )-1)
performed much worse.
We evaluate the model performance using validation likelihood, as shown in Figure 11 First, as in Fig. 7 we find that the likelihoods between the unified model are competitive with the individually
Note that this follows established methods for ML-based antibody diversification in Raghu et al. 2025
39

---

Published as a conference paper ICLR 2025
6.0
4.0
('C)
2.0
47
0.0
Simplicial Discrete Gaussian
Simplicial Discrete Gaussian
Figure The sufficient statistic parametrization enables a single model to perform competitive discrete, Gaussian, and simplicial optimization of antibodies. Using protein models from Fig. 7 we "denoise" antibody sequences and plot the predicted improvement melting temperature in libraries of size 100.
Unified Model
0.25
0.220.23
0.220.22
(J)
0.18
Discrete Simplicial Gaussian
Discrete Simplicial Gaussian
The SSP enables a single model to fit image data across 3 modalities, We perform the analysis of Fig. for image data and find similar result on MNIST.
Figure 11:
trained models. In fact, we are even able to achieve slightly better performance for discrete and Gaussian diffusion, perhaps because the parameterization easier to learn from, or because of benefit from learning diverse data.
As well, while we might expect Gaussian diffusion to achieve the best data fit due to the continuous nature of the data, we the opposite: among individual models, Gaussian surprisingly achieves the worst likelihood. This demonstrates the importance of considering multiple types diffusion paradigms depending the downstream tasks, thereby motivating approach of training single unified model.
We also generate 64 unconditional samples per model using 1,000 steps of ancestral sampling. Through our visualizations in Figure 12 we see that the unified model does not lead to noticeable drop in sample quality compared to individual models.
40

---

Published as a conference paper ICLR 2025
6
4
4
7
8
6
9
7
3
9
1
7
6
4
6
0
6
7
7
4
7
9
7
6
9
6
9
7
7
4
0
1
O
4
7
6
)
4
6
4
/
2
6
7
7
6
9
1
7
7
/
1
4
0
3
6
6
4
9
O
4
6
6
3
3
3
2
5
3
7
7
7
6
/
4
2
9
1
8
7
X
7
q
9
8
1
7
7
8
7
7
6
7
/
4
1
9
9
8
8
9
4
0
7
7
1
3
7
9
1
/
9
8
6
0
0
7
3
4
6
4 9
8
4
9
8
1
b
4
7 9
7 7
3
O
4
4
7
9
9
1
/
9
9
2
6
4
(a) Discrete
(b) Simplicial
(c) Gaussian
6
9
6
4 8
/ 5
4
C
3
7
4
5
9
3
8
S
0 4
2
8
9
6
3
8
9
7
8
6
4
5
3
8
9
4
9
6
7
4
3
9
9
4
3
8
2
6
5
7 2 6 7>
7
9
9
8
8
1
A
0 7 1 6
8
8
0
7
0
6
7
9
8
6
5
7
8 9
8
7
0
8
6
7
6
4
5
6
7
6
7
4 3
5
9
8
4
9
4
5
8
8
4 8
4 L
4
8
3
3
8
4 9 9
7
0
2
0
5
9
2
4
7
6
f 8
8
5
8
4
7
7
5
1
0
7
15
8
4
8
0
9
8
3
4 4
a
4
5
4
6
4
5
/
0
2
3
9
2 3
3
/
7
3
9
6
5
4
S
8
/
(d) Unified Discrete
(c) Unified Simplicial
(f) Unified Gaussian
Figure 12: The SSP results in noticeable drop in generation quality for image models. We plot samples from models trained on MNIST.
41


Reviews:
[{"review_id": "cGIDTHs7FX", "reviewer": "Reviewer_YoGL", "summary": "This paper proposes a unified formulation of discrete, simplicial, and Gaussian diffusion that, in different limiting cases, converges to each of these three diffusion formulations. The method is based on introducing repeats of tokens in a discrete sequence and diffusing each token independently with a Markov transition kernel. The theoretical results show that, in the limit of an infinite number of repeats, the proposed scheme converges to Gaussian diffusion. The proposed approach is evaluated on the protein modeling task.", "questions": "Line 101: If we consider softmax instead of argmax, the connection between gaussian and discrete diffusion in continuous time is achieved via Ito’s lemma. What will fail if we consider zero temperature in softmax? \n\nLine 111: Could authors explain why simplex diffusion sacrifices the ability to calculate a likelihood? If we have an ODE as in [1] we could compute likelihoods via integration of Jacobians of the vector field.\n\nLine 182: Why is the closeness of ELBOs described as paradoxical? The ELBO is continuous with respect to weak convergence of probability measures. If we take a distribution that is a finite combination of delta measures and convolve it with a Gaussian kernel of small variance, then for sufficiently small $\\sigma$ the two distributions are close in Wasserstein distance (and hence in the weak topology). Therefore, if a probabilistic quantity is continuous in the weak topology, we should indeed expect these two distributions to yield close values of that quantity by definition of continuity.\n\nWhat is the computational overhead of the proposed approach? \n\nThe reviewer is willing to raise the score if the weaknesses regarding computational complexity and generalizability beyond the protein modeling task are addressed.\n\n[1] Stark et al. Dirichlet Flow Matching with Applications to DNA Sequence Design", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The paper is well written, and the motivation and importance of the work are clearly explained.\n\nThe idea of introducing repeats is interesting and novel, as are the theoretical convergence results.\n\nThe source code is provided and well-structured.", "weaknesses": "Training a discrete diffusion model for an extended sequence is computationally expensive, so it is unclear why a practitioner would use the proposed approach. This is especially true for Gaussian cases, where working with large sequence lengths becomes infeasible. For example, if one uses a transformer with the number of repeats equal to 10, the full-attention complexity becomes 100 times more expensive. The computational efficiency is not discussed in the main part of the paper. Sequence length used in the evaluation experiments is 200 which is too small for a practical image or language modeling tasks.\n\nExperimental verification is limited to the protein modeling task. There are no evaluations in visual or language domains, for example, on MNIST, where one could generate samples with both Gaussian and discrete diffusion and visually assess sample quality. This raises questions about the generalizability of the proposed approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hp5L2Xzgnk", "reviewer": "Reviewer_ia2Y", "summary": "This paper unifies various approaches to diffusion under the view of population genetics (Wright-Fisher). They then use this theoretical result to draw from the literature of Wright-Fisher diffusion, practically improving existing simplicial approaches for DNA generation. They finally demonstrate that diffusion can be represented by a sufficient statistic parameterisation that is independent of the timestep and diffusion paradigm.", "questions": "See above\n\nMy weaknesses are all fairly minor and I believe the main contribution (theoretical) is good.\nHowever, this is a little unconfident as I am not knowledgeable about the theoretical domain explored in the paper. As such, I am unlikely to increase my score as I cannot confidently verify the main contribution.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Paper is well presented with a good mixture of visualisations for better understanding. \n- Good theoretical contribution, being able to tie together disparate concepts into a unified framework will be helpful for future researchers aiming to navigate the field. Note that I am a little unconfident here with regards to the specifics of the theory, as I did not check it carefully, and it is out of the domain of my knowledge. \n- Demonstrated practical impact from the Wright-Fisher perspective, improving performance on an applied task.\n- The implications of SSP are somewhat interesting", "weaknesses": "The paper load for ICLR this year has been large, and so I have not been able to spend as much time as I would like on reviewing. I encourage the authors to correct any errors/misunderstandings I may have with regards to the paper.\n\n\n1. **Experimental weaknesses**\n    1. The DNA experiment is quite limited in scale (it is also not clear why the stabilised simplicial diffusion surpasses dirichlet FM).\n1. **SSP motivation unclear**\n    1. Although the idea of learning multiple diffusion paradigms in a single model is interesting, I think more clarity on the practical utility of such an approach might be of benefit. For example, I don't see a practical benefit in training an image generation diffusion model with an additional discrete diffusion objective on the 256 intensity levels. That is to say, I don't see that much benefit in SSP over comparing different diffusion approaches and picking the individual best.\n1. **Better intuition**\n    1. For the general reader interested in diffusion, the theoretical results of the paper are a little hard to intuit. However, I also appreciate that the authors have already included a number of helpful visualisations and I found the language and notation to be clear.\n1. **Missing relevant references**\n    1. [1] is a recent large scale DNA generation model that achieves SoTA by combining autoregressive and discrete diffusion/flow models.\n    1. [2] propose to use gaussian diffusion for categorical data, but constrain the *clean data* distribution to the simplex hyperplane and train using cross entropy. \n\n\n[1] Li et al, Absorb & Escape: Overcoming Single Model Limitations in Generating Genomic Sequences, NeurIPS 2024\n\n[2] Eijkelboom et al, Variational Flow Matching for Graph Generation, NeurIPS 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ojD0bXeuU0", "reviewer": "Reviewer_JD9a", "summary": "The paper introduces a one-shoe-fit-all framework that perform diffusion on discrete, gaussian, and simplical data based on Wright-Fisher model. The paper shows a maththematical connection between them, allowing to training multiple types of data at once and yield competitive performance on DNA geneartion.", "questions": "Q1. The authors claimed the proposed method enable the flexibility of performing diffusion at all three domains at test time. I wonder if this still holds for large model (e.g. Stable Diffusion, Large Diffusion Language Model) where the modality-specific model is guaranteed with the scaling laws. \n\nQ2. The experiment setting is limitted to a single modality, no experiment demonstrates the practical value of the unified framework across genuinely different data modalities. For example: Can a model trained on discrete text data leverage the continuous parameterization for improved performance? Can the same model joinly learn from discrete language tokens and continuous image embeddings?\n\nQ3. What's the effect of the ζ parameter in practical settings? Ablation is needed.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Establishing a connection among discrete, Gaussian, and simplex which has a root from Wright-Fisher -- a human population genetics model. This insight is crucial for building next AI model where a model is not limited to any particular type of data.", "weaknesses": "1. Limited practical applicability of multi-domain inference: While Figure 7 demonstrates competitive performance when the model handles different data domains at test time, this scenario has limited real-world relevance. In practice, each modality has distinct statistical properties—images typically follow continuous Gaussian distributions, while text is inherently discrete. Using a discrete or simplex representation for images, or Gaussian diffusion for language, is suboptimal for each respective modality. This raises questions about whether a unified framework provides practical advantages over domain-specific approaches.\n2. The paper lacks experiments on combined domains like image and text. Since the primary contribution is demonstrating the advantages of a unified framework, the authors should have a baseline to train and evaluate a model on both language and vision domains jointly. Without such experiments, the claimed benefits of the unification approach remain unsubstantiated.\n3. The paper lacks vision experiments, despite images being continuous data that naturally align with Gaussian distributions.\n4. The paper does not provide a suitable choice of parameterization for a specific data domain. \n\n\nMinor concenrs:\n1. Paper presentation is quite dense. Theorem 4.1 and 5.1 requires multiple readings to understand.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "biaVXn7DSZ", "reviewer": "Reviewer_eBao", "summary": "This paper addresses discrete data generation using diffusion models and proposes a unified framework that integrates direct discrete diffusion, simplex-based diffusion, and embedding-based Gaussian diffusion. Inspired by genetics, this approach enhances performance.", "questions": "- There is a previous work of \"Diffusion Evolution\" that connects backward diffusion and forward evolution process. The authors proposed the opposite picture: the diffusion process corresponds to reverse evolution, while the denoising process represents forward evolution, naturally incorporating selection and reproductive isolation. How do the authors of the present paper view this distinction? Is there any conceptual connection between the two frameworks?\n- Although this paper mainly focuses on discrete diffusion, it also builds a bridge between discrete and continuous spaces. So, I’m wondering if the framework presented in the paper can be applied to continuous diffusion?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The paper reveals an intriguing connection between neutral evolution in genetics and the diffusion process in diffusion models. This bridge between the two fields allows for mutual conceptual enrichment and represents a high level of innovation.\n\nThe proposed SSP method is very interesting and inspiring, as it provides an intermediate layer that unifies all perspectives.\n\nAccording to the experiments presented, the proposed method achieves significantly better performance than traditional approaches.", "weaknesses": "- The relationship between genetic drift and the n-simplex method is not clearly explained, and the paper would benefit from providing a more intuitive illustration of their connection.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)