Where to Diffuse, How to Diffuse, and How to Get Back: Automated Learning for Multivariate Diffusions
Authors: Raghav Singhal
Mark Goldstein
Rajesh Ranganath
Venue: ICLR 2023 poster (2023)
Keywords: Diffusion models
score based generative model
generative models
variational inference

Abstract: Diffusion-based generative models (DBGMs) perturb data to a target noise distribution and reverse this process to generate samples.  The choice of noising process, or inference diffusion process, affects both likelihoods and sample quality. For example, extending the inference process with auxiliary variables leads to improved sample quality. While there are many such multivariate diffusions to explore, each new one requires significant model-specific analysis, hindering rapid prototyping and evaluation. In this work, we study Multivariate Diffusion Models (MDMs). For any number of auxiliary variables, we provide a recipe for maximizing a lower-bound on the MDMs likelihood without requiring any model-specific analysis. We then demonstrate how to parameterize the diffusion for a specified target noise distribution; these two points together enable optimizing the inference diffusion process. Optimizing the diffusion expands easy experimentation from just a few well-known processes to an automatic search over all linear diffusions. To demonstrate these ideas, we introduce two new specific diffusions as well as learn a diffusion process on the MNIST, CIFAR10, and ImageNet32 datasets. We show learned MDMs match or surpass bits-per-dims (BPDs) relative to fixed choices of diffusions for a given dataset and model architecture.

Published as a conference paper at ICLR 2023
WHERE TO DIFFUSE, How TO DIFFUSE, AND How TO GET BACK: AUTOMATED LEARNING FOR MULTIVARI- ATE DIFFUSIONS
Raghav Singhal Mark Goldstein Rajesh Ranganath 1.2 Courant Institute of Mathematical Sciences', New York University Center for Data Science", New York University
ABSTRACT
Diffusion-based generative models (DBGMS) perturb data target noise distri- bution and reverse this process to generate samples. The choice of noising process, inference diffusion process, affects both likelihoods and sample quality. For ex- ample, extending the inference process with auxiliary variables leads to improved sample quality. While there are many such multivariate diffusions to explore, each new one requires significant model-specific analysis, hindering rapid pro- totyping and evaluation. In this work, we study Multivariate Diffusion Models (MDMS). For any number of auxiliary variables, we provide a recipe for maximiz- ing a lower-bound on the MDMS likelihood without requiring any model-specingg analysis. We then demonstrate how to parameterize the diffusion for specified target noise distribution; these two points together enable optimizing the infer- ence diffusion process. Optimizing the diffusion expands easy experimentation from just few well-known processes to an automatic search over all linear dif- fusions. To demonstrate these ideas, we introduce two new specific diffusions as well as learn diffusion process on the MNIST, CIFARIO, and IMAGENET32 datasets. We show learned MDMs match surpass bits-per-dims (BPDs) relative to fixed choices of diffusions for a given dataset and model architecture.
INTRODUCTION
Diffusion- based generative models (DBGMs) perturb data a target noise distribution and reverse this process to generate samples. They have achieved impressive performance in image generation, editing, translation (Dhariwal & Nichol, 2021: Nichol & Dhariwal, 2021: Sasaki et al., 2021; Ho et al., 2022), conditional text-to-image tasks (Nichol et al., 2021; Ramesh et al., 2022: Saharia et al., 2022) and music and audio generation (Chen et al., 2020; Kong et al., 2020; Mittal et al., 2021). They are often trained by maximizing lower bound on the log likelihood, featuring an inference process interpreted as gradually "noising" the data (Sohl-Dickstein et al., 2015; Ho et al., 2020).
The choice of this inference process affects both likelihoods and sample quality. On different datasets and models, different inference processes work better; there is no universal best choice of inference, and the choice matters (Song et al., 2020b).
While some work has improved performance by designing score model architectures (Ho et al., 2020: Kingma al., 2021; Dhariwal & Nichol, 2021), Dockhorn al. (2021) instead introduce the critically-damper langevin diffusion (CLD), showing that significant improvements in sample generation can be gained by carefully designing new processes. CLD pairs each data dimension with an auxiliary "velocity" variable and diffuses them jointly using second-order Langevin dynamics.
A natural question: if introducing new diffusions results in dramatic performance gains, why there only a handful of diffusions vvariance-preservvii stochastic differential equation (VPSDE), variance exploding (VE), CLD, sub-VPSDE) used in DBGMs? For instance, are there other auxiliary variable diffusions that would lead to improvements like CLD? This avenue seems promising as auxiliary variables have improved other generative models and inferences, such as normalizing flows
Equal Contribution. Correspondence {rsi inghal, gel geldsteing at nyu.edu.

---

Published as a conference paper at ICLR 2023
(Huang et al., 2020), neural ordinary differential equations (ODES) (Dupont et al., 2019), hierarchical variational models (Ranganath et al., 2016), ladder variational autoencoder (Sønderby et al., 2016), among others.
Despite its success, CLD also provides evidence that cach new process requires sigmificant model- specific analysis. Deriving the evidence lower bound (ELBO) and training algorithm for diffusions challenging (Huang et al., 2021; Kingma et al., 2021; Song et al., 2021) and is carried out in case-by-case manner for new diffusions (Campbell et al., 2022). Auxiliary variables seemingly complicate this process further; computing conditionals of the inference process necessitates solving matrix Lyupanov equations (section 3.3). Deriving the inference stationary distribution- -wich helps the model and inference match can be intractable. These challenges limit rapid prototyping and evaluation of new inference processes.
Concretely, training diffusion model requires:
(R1): Selecting an inference and model process pair such that the inference process converges to the model prior
(R2): Deriving the ELBO for this pair
(R3): Estimating the ELBO and its gradients by deriving and computing the inference process' transition kernel
In this work, introduce Multivariate Diffusion Models (MDMs) and method for training and evaluating them. MDMS are diffusion-basee generative models trained with auxiliary variables. We provide recipe for training MDMS beyond specific instantiations like VPSDE and CLD- to all linear inference processes that have stationary distribution, with any number of auxiliary variables.
First, we bring results from gradient-based based MCMC (Ma et al., 2015) to diffusion modeling struct MDMs that converge to a chosen model prior (R1); this tightens the ELBO. Secondly, for any number of auxiliary variables, kernel of linear MDMs, necessary for the ELBO, can be computed automatically and generically, for higher-dimensional auxiliary systems (R3).
derive the MDM ELBO (R2). Finally, show that the transition
With these tools, we explore a variety of new inference processes for diffusion-based generative models. We then note that the automatic transitions and fixed stationary distributions facilitate di- rectly learning the inference to maximize the MDM ELBO. Learning turns diffusion model training into a search not only over score models but also inference processes, at extra derivational cost.
Methodological Contributions. In summary, our methodological contributions are:
1. Deriving ELBOS for training and evaluating multivariate diffusion models (MDMs) with auxiliary variables.
2. Showing that the diffusion transition covariance does not need to be manually derived for each new diffusion. We instead demonstrate that a matrix factorization technique, previ- ously unused in diffusion models, can automatically compute the covariance analytically for any linear MDM
3. Using results from gradient- based Markov chain Monte Carlo (MCMC) to construct MDMS with a complete parameterization of inference processes whose stationary distribution matches the model prior.
4. Combining the above into an algorithm called Automatic Multivariate Diffusion Training (AMDT) that enables training without diffusiin-speciffe derivations. AMDT enables training score models for any linear diffusion, including optimizing the diffusion and score jointly.
To demonstrate these ideas, we develop MDMS with two specific diffusions as well as learned mul- tivariate diffusions. The specific diffusions are accelerated Langevin diffusion (ALDA) (introduced in Mou et al. (2019) as a higher-order scheme for gradient- based MCMC) and an alteration, mod- ified accelerated Langevin diffusion (MALDA). Previously, using these diffusions for generative modeling would require significant model-specific analysis. Instead, AMDT for these diffusions is derivation-free.
2

---

Published as a conference paper at ICLR 2023
Empirical contributions. We train MDMS the MNIST, IMAGENET32 and CIFAR-10 datasets. In the experiments, we show that:
Training new and existing fixed diffusions, such as ALDA and MALDA, easy with the proposed algorithm AMDT.
2. Using AMDT to learn the choice of diffusion for the MDM matches surpasses the perfor- mance of fixed choices diffusion process; sometimes the learned diffusion and VPSDE do best; other times the learned diffusion and CLD do best.
3. There for as much performance improvement over VPSDE as a three-fold increase score model size for fixed univariate diffusion.
and existing MDMs, trained and evaluated with the MDM ELBO, that account
These findings affirm that the choice of diffusion affects the optimization problem, and that learning the choice bypasses the process choosing diffusions for each new dataset and score architecture. We additionally show the utility of the MDM ELBO by showing dataset that CLD achieves better bits-per-dims (BPDS) than previously reported with the probability flow ODE (Dockhorn et al., 2021).
2
SETUP
We present diffusions by starting with the generative model and then describing its likelihood lower bound (Sohl-Dickstein et al., 2015; Huang et al., 2021; Kingma et al., 2021). Diffusions sample from model prior Zo ne and then evolve continuous-time stochastic process 2, RR
dz (z t)dt + (t(t),
[0,
(I)
where B, is d-dimensionsal Brownian motion. The model is trained that Zy approximates the data x ~ Idata Maximum likelihood training of diffusion models is intractable (Huang et al., 2021; Song et al., 2021; Kingma et al., 2021). Instead, they trained using variational lower bound on log po (2r = x). The bound requires an inference process 4o (y x == x..
dy = fo s)ds + go(s)dB.,
s E [0,
(2)
where B, is another Brownian motion independent of B,. The inference process is usually taken to be specified rather than learned, and chosen to be i.i.d. for each Vyy conditional each xy. This leads to the interpretation of the Vt as noisy versions of features (Ho et al., 2020). While the diffusion ELBO is challenging to derive in general, Huang al. (2021); Song et al. (2021) show that when the model process takes the form:
dz = [9 (2, -- 1) So(2, -7--)) t)] dt + go - t)dB,,
(3)
the ELBBO is:
/"
T 1 -11000 V (gāS0 fo)ds
log (z)) 2 [ (x) (x)l)) log log ()())))))
(4)
where fe.gs: $9 are evaluated at sy,,s), ||x|| = x Ax and g² = gg Equation (4) features the Implicit Score Matching (ISM) loss (Song et al., 2020a), and can be re-written as an ELBO Ldsm featuring Denoising Score Matching (DSM) (Vincent, 2011; Song al., 2020b), see appendix F.1.
A RECIPE FOR MULTIVARIATE DIFFUSION MODELS
As has been shown in prior work (Song et al., 2021; Dockhorn et al., 2021), the choice of diffusion matters. Drawing on principles from previous generative models (section 6), we can consider wide class of diffusion inference processes by constructing them using auxiliary variables.
Following Huang al. (2021); Dockhorn al. (2021) integrate all processes in forward time to T.
may be helpful to think of an additional variable &; 2 zr- so that Xo approximates x ~ Glau-
We use y the inference variable over the space the model's
3

---

Published as a conference paper at ICLR 2023
At first glance, training such diffusions seem challenging. First, needs an ELBO that includes auxiliary variables. This ELBO will require sampling from the transition kernel, and setting the model prior the specified inference stationary distribution. But doing such diffusion-specifer analysis manually challenging and hinders rapid prototyping.
In this section show how address these challenges and introduce algorithm, AMDT, to simplify and automate modeling with MDMs. AMDT can be used to train new and existing diffu- sions, including those with auxiliary variables, and including those that learn the inference process. In appendix A we discuss how the presented methods can also be used to automate and improve simplified score matching and noise prediction objectives used to train diffusion models.
3.1 MULTIVARIATE MODEL AND INFERENCE
For the jth data coordinate at each time I, MDMS pair 3ij E R with a vector of auxiliary variables Viy E R R--1 into a joint vector u, and diffuse the extended space:
Ze Vz
Ug ne,
du ho (ur
t)dt + (L)dd
(5)
MDMs model the data with 2T. a coordinate in ps. For the jth feature x,, each Uty R consists of a "data" dimension ui and auxiliary variable uzz- Therefore RdK We extend the drift coefficient he from function in I x 12 + Rd to the extended space RaK R4 RdK We likewise extend the diffusion coefficient to matrix Sg acting on Brownian motion B, E R
Because the MDM model is over the extended space, the inference distribution y must be too. We then set q(y) = x) to any chosen initial distribution, c.g. (O, and discuss this choice in section Then y, evolves according the auxiliary variable inference process:
dy (y, s)ds (o(s)d..,
where the inference drift and diffusion coefficients før Go are now over the extended space y = y"). The function fo lets the and coordinates of Yıj interact in the inference process.
ASSUMPTIONS
This work demonstrates how to parameterize varying Itô processes, used for diffusion mod- eling, to have stationary distribution that matches the given model prior. To take advantage of automatic transition kernels also presented, inferences considered for modeling are linear time-varying processes and take the form:
dy As(s)yds + (d)BB
where (s(s): TR+ > dK dK and go(s) R_ - dK dK are matrix- valued functions.
3.2
ELBO FOR MDMS
We now show how to train MDMS to optimize a lower bound on the log likelihood of the data. Like in the univariate case, we use the parameterization in eq. (3) to obtain tractable ELBO.
Theorem 1. The MDM log marginal likelihood of the data is lower-bounded by:
1-1 2
logpo(2) (x) 2 Egs (xlx)
log (ogTo(yT+ - 11s0l + (93.0 fo)ds - log go (y
(mism
FT
('
1
= -ygolylo) lr +
|100113
|1:0
s0l1g + (V So)ds eg (midem))
2
where divergences and gradients are taken with respect to y and 84 Vy. log (y.
4

---

Published as a conference paper at ICLR 2023
Proof. The proof for the MDM ISM ELBO _ is in appendix F. In short, we introduce auxiliary Huang et al. (2021) (equivalently, Theorem of Song et al. (2021) appendix E of Kingma et al. (2021)) to the joint space, and then apply additional variational bound to Vo- The MDM DSM ELBO Cuddam is likewise derived in appendix F, similarly to Huang et al. (2021); Song al. (2021), but extended to multivariate diffusions.
variables, apply Theorem
We train MDM's by estimating the gradients of Cudsm as estimates of _ can be computationally prohibitive. For numerical stability, the integral in eq. (7) computed on [c, T` rather than [o, I] One can regard this as a bound for variable u, To maintain a proper likelihood bound for the data, one can choose likelihood Un u. and compose bounds as we demonstrate appendix I. We report the BL with this likelihood term, which plays the same role as the discretized Gaussian in Nichol & Dhariwal (2021) and Tweedic's formula in Song et al. (2021).
3.3 INGREDIENT 1: COMPUTING THE TRANSITION go(y,(z)
To estimate eq. (7) and its gradients, we need samples from ((y,,=)) and to compute V log g lz). While an intractable problem for MDMS in general, we provide two ingredients for tightening and optimizing these bounds in generic fashion for linear inference MDMS.
We first show how to automate computation ((y,)1y))))))) and then g(y,la). For linear MDMS of the form:
dy (s)ydd g
the transition kernel (yo)) is Gaussian (Särkkä & Solin, 2019). Let f(y,8) A(syy. Then, the mean and covariance solutions to the following ODES:
dmz = As))m.jo
dE dE.o/ 20/ ds
A (s)) - A ( g
(8)
The mean can be solved analytically:
m = exp I/ A(u)de Yo = -xx(ssAyo
(9)
integration A(r)
The covariance equation does not have as simple solution because eq. (9) as the unknown matrix Enjo is being multiplied both from the left and the right.
Instead of solving eq. (8) for specific diffusion manually, as done in previous work (e.g. pages 50- 54 of Dockhorn et al. (2021)), we show that a matrix factorization technique (Särkkä & Solin (2019), sec. 6.3) previously unused in diffusion-basse based generative models can automatically compute Σ.jo generically for any linear MDM. Define C, H. that evolve according to:
(dC//ds) dH./ds
(s))
g²(s)
C2
A (s)
H.
(10)
then Σ.jo C,H, for C. Eg and Ho I (Appendix D). These equations can be solved in closed-form,
C, H.
(IA),
[ A
)
En I
exp (
A
Σo
exp
1.
A
(1I)
integration if (v) A. A.g(v)
where [A], = So (v)dd. condition on Yo (xr, v), we set Σn =
Computing (x)). For the covariance Σgjo, to condition instead of yo- set Eg
E. Σvo
To compute the mean, is the same expression as for o(y.lvo) but with a different initial condition:
maju = exp L/- A(v)dv)
(12)
See appendix D for more details.
5

---

Published as a conference paper at ICLR 2023
Algorithm Automatic Multivariate Diffusion Training
Input: Data { inference process matrices Qø, De, model prior Ty, initial distribution (Y
2), and score model architecture 5e
Returns: Trained score model $6
while se not converged do
Sample x Σn 1 de. 20 (YD x)
Sample S ~ 0 1] and ys Vr ~ go (o x) using algorithm
Estimate the stochastic gradient of the MDM ELBO, VoL(O, Φ), using eq. (7)
0 < 0 + vVVaL(e ø). if learning inference then
Φ 1 aV ((,,0)
end if end while Output sg
fast and simple algorithm. We show in algorithm 2 (appendix H) that computing the transition kernel only requires knowing f.g and requires no diffusion-speciie specifcc analysis. For K - auxiliary variables, A. are K x K. Like for scalar diffusions, these parameters are shared across data coordinates. This means matrix exponentials and inverses are done on K K matrices, where K is only 2 or 3 in our experiments. In table I, we compare the time to sample a batch of size 256 from the transition kernel for CIFAR 10 and MNIST. The table shows the extra computa- tional cost of the automated algorithm is negligible. This automation likewise applies to simplified score matching and noise prediction objectives, since all rely on (>(y)/z) (appendix A).
Table 1: Runtime Comparison: we
compare the time of sampling from the CLD diffusion analytically versus using the automated algorithm.
Method Analytical Automated
CIFAR-10
MNIST
0.027
0,0062
0.029
0.007
3.4 INGREDIENT 2: MDM PARAMETERIZATTIN
The MDM ELBO (eq. (7)) is tighter when the inference yr tends toward the model's prior ma. Here we construct inference processes with the model prior Ty as a specified stationary distribution gso-
Ma et al. (2015) provide complete recipe for constructing gradient-based MCMC samplers; the recipe constructs non-linear time-homogeneous Itô processes with a given stationary distribution, and show that the parameterization spans all such Itō processes with that stationary distribution.
Diffusion models usually have time-varying drift and diffusion coefficients (e.g. use of the 3(t) function). To build diffusion models that match the model prior, first extend Theorem from Ma et al. (2015) to construct non-linear Itô processes with time-varying drift and diffusion coef- ficients with given stationary distribution (Appendix C). Then, keep transitions tractable (per Section 3.3), we specialize this result to linear Itô diffusions.
We directly state the result for linear time-varying diffusions with stationary distributions. The pa- rameterization requires a skew-symmetrie matrix -Q(s) Q(s) a positive semi-definite matrix D(s), and a function Vy such that the desired stationary distribution qoo proportional to exp/-H (y)]. Linear Itô diffusions have Gaussian stationary distributions (Särkkä & Solin, 2019) meaning that H is linear and can be expressed as Sy for some matrix S. For matrix A, let VA refer to the matrix square root defined by a VA
A aa" Then, the Itô diffusion:
dy -(((() D(s) Sy ds + (2
((s))
(13)
f(y.s)
8(s)
has Gaussian stationary distribution (O, where Q(s), D(s) and S are parameters. For discussion convergence to the stationary distribution, as well as skew-symmetric and positive appendix C, where we also show that existing diffusion pro-
semi-definite parameterizations, cesses such as VPSDE and CLD are included Q/D parameterization. We display the ELBO in terms of Q D in appendix G and an algorithm in appendix H.

---

Published as a conference paper at ICLR 2023
For score matching and noise prediction losses and given Jø. achieving minimizing value with respect to $8 does not imply that the generative model score will match the inference score. Model- ing the data also requires the marginal distribution of 9e.7 approximate is important to confirm the stationary distribution is appropriately set, and the tools used here for the ELBO can be used to satisfy this requirement for score matching and noise prediction (appendix A).
When q is constant, it
3.5 LEARNING THE INFERENCE PROCESS
The choice of diffusion matters. and the ELBOS in eq. (7) have no requirement for fixed qá. We therefore learn the inference process jointly with sg. Under linear transitions (ingredient 1), no algorithmic details change as the diffusion changes during training. Under stationary parameteriza- tion (ingredient 2), we can learn without the stationary distribution going awry. In the experiments, learning matches surpasses BPDs of fixed diffusions for given dataset and score architecture.
In _udam or _mism,, cd may be set to equal Te. but it is yr ~ q for the chosen that is featured in the ELBO. Learning y6 can choose Yr to reduce the cross-entropy:
(x llog (y
(14)
Minimizing eq. (14) will tighten the ELBO for any 8g. Next, 4s is featured in the remaining terms that feature 8g; optimizing for 4s will tighten and improve the ELBO alongside 88- Finally, 2o is featured in the expectations and the log 9a term:
log pa (u x) 2= Eoolx--olz)) (C or Cism) - log (x w1z)
(15)
The g x) terms impose an optimality condition that pa(u ui-) = (v yö) (appendix E), When it satisfied, no looseness in the ELBO is due to the initial time zero auxiliary variables.
To learn, Q D need to be specified with parameters 6 that enable gradients. We keep S fixed at inverse covariance of xg. The transition kernel ( lær) depends Q.D D through its mean and covariance. Gaussian distributions permit gradient estimation with reparameterization or score- function gradients (Kingma & Welling, 2013; Ranganath et al., 2014; Rezende & Mohamed, 2015; 2014). Reparameterization is accomplished via:
Titsias &
y. 0 + Lyjoe
(16)
where c ~ (0, Tak) and Lajo satisfies L,jo L 1/0 Σ.jo- derived using coordinate- wise Cholesky decomposition. Gradients flow through eq. (16) from y, to m.jo and Σ, to Q,
D to parameters é
Algorithm displays Automatic Multivariate Diffusion Training (AMDT). AMDT provides a training method for diffusion-basse generative models for either fixed Q, D matrices or for learning the Qo: Do matrices, without requiring any diffusion-speciff analysis.
Learning in other diffusion objectives. Like in the ELBO, learning in score matching or noise prediction objectives can improve the match between the inference process and implied generative model (appendix A).
4 INSIGHTS INTO MULTIVARIATE DIFFUSIONS
Scalar versus Multivariate Processes. Equation (13) clarifies what can change while preserving Joo. Recall that Q and D are x K for K 1 auxiliary variables. Because is the only 1 skew-symmetric matrix, scalar processes must set Q = O. With 00.00 = (O, I), the process is:
dy = Dss)yds + /2D(s)dB..
(17)
What left is the VPSDE process used widely in diffusion models where D(s) 1 (s)) is 1 x
1 (Song et al., 2020b). This reveals that the VPSDE process is the only scalar diffusion with stationary distribution. This also clarifies the role of Q: accounts for mixing between dimensions in multivariate processes, as do non-diagonal entries in D for K > 1.
There are processes such sub-VPSDE (Song et al., 2020h) which are covered in the sense that they tend members of this parameterization T grows: sub- VP converges VPSDE.
7

---

Published as a conference paper at ICLR 2023
CLD optimizes log-likelihood lower bound. Differentiating L (eq. (7)) with respect the score model parameters, we show that the objective for CLD (Dockhorn et al., 2021) maximizes a lower bound on log Po (z). not not just log (u(), without appealing to the probability flow ODE.
Does my model use auxiliary variables? An example initial distribution is q(v)lz) = N I). also common to set Te - (0, 1). Because the optimum for diffusions is Pa = q. the optimal model has main and auxiliary dimensions independent at endpoints O and T. Does this mean that the model does not use auxiliary variables? In appendix B, we show that in this case the model can still use auxiliary variables at intermediate times. A sufficient condition is non-diagonal D.
EXPERIMENTS
We test the MDM framework with handcrafted and learned diffusions. The handcrafted diffusions are (a) ALDA, used in (Mou et al., 2019) for accelerated gradient- MCMC sampling (eq, (32)) and (b) MALDA: modified version of ALDA (eq. (33)). Both have two auxiliary variables. We also learn diffusions with and auxiliary variables. We compare with VPSDE and ELBO-trained CLD.
Table 2: BPD upper-bounde bounds image generation for fixed architecture. CIFAR-10: learning outperforms CLD, and both outperform the standard choice of VPSDE. MNIST: learning matches VPSDE while the fixed auxiliary diffusions are worse. IMAGENET32: all perform similarly. Learning matches or surpasses the best fixed diffusion, while bypassing the need to choose a diffusion.
Model
K
CIFAR-10
IMAGENET32
MNIST
VPSDE
3.20
3.70
1.26
Learned
3.07
3.71
1.28
Learned
3.08
3.72
1.33
CLD
3.11
3.70
1.35
MALDA
3.13
3.72
1.65
ALDA
29.43
33.08
124.60
Table 3: Parameter Efficiency. The first two rows display diffusions from previous work: VPSDE and CLD, both using score models with 108 million parameters CIFAR-10. We train the rest using a score model with 35.7 million parameters. The learned diffusion matches the performance of VPSDE-large; changes in the inference account for much improvement 3x increase in score parameters. BPDS are upper-bounds.
Model
K
Parameters
CIFAR-10
VPSDE-large (Song al., 2021)
108M
3.08
CLD-large (Dockhorn et al., 2021)
108M
3.31
Learned
35.7M
3.07
CL.D
2
35.7M
3.11
VPSDE
35.7M
3.20
Following prior work, we train DBGMs for image generation. We use the U-Net from Ho et al. (2020). We input the auxiliary variables as extra channels, which only increases the score model parameters in the input and output convolutions (CLD and Learned 2 have 7, 000 more parameters than VPSDE on CIFAR-10 10 and IMAGENET32 and only 865 more for MNIST). We use simple uniform dequantization. We report estimates of mdsm (which reduces to the standard ddsm for = 1). We sample times using the importance sampling distribution from Song al. (2021) with truncation set to e 10-3 To ensure the truncated bound is proper, we use a likelihood described in appendix I.
Results. Table 2 shows that the inference process matters and displays. displays DBGMS that we train and evaluate CIFAR-10, IMAGENET32 and MNIST. This includes the existing VPSDE and CLD, the new MALDA and ALDA, and the new learned inference processes. All are trained with the 35.7M parameter architecture. For CIFAR-10, learning outperforms CLD, and both outperform the standard choice of VPSDE. For MNIST, learned diffusions match VPSDE while the three fixed aux- iliary diffusions worse. On IMAGENET32, all perform similarly. The take-away is that learning
8

---

Published as a conference paper at ICLR 2023
matches surpasses the best fixed diffusion performance and bypasses the choice of diffusion for each new dataset or score architecture. In Figure we plot the generated samples from CIFAR
Table 3's first two rows display diffusion models from previous work: VPSDE (Song et al., 2021) and CLD (Dockhorn et al., 2021) both with the 108 million score model from Song et al. (2021) (labeled "large"). The rest DBGMs that we train using the U-Net with 35.7 million parameters for CIFAR- 10 and IMAGENET32 and 1.1 million for MNIST. Despite using significantly fewer parameters, the learned diffusion achieves similar BPD compared to the larger models, showing that changes in inference can account for as much improvement as a three-fold increase in parameters. While the larger architecture requires two GPUs for batch size 128 on CIFAR-10 on A100s, the smaller one only requires one; exploring inference processes can make diffusions more computationally accessible. Table also demonstrates a tighter bound for CLD trained and evaluated with the MDM ELBO (S 3.11) relative to existing probability flow-based evaluations (3.31).
Figure 1: CIFAR10 10 samples generated from the "learned 2" and MALDA generative models.
RELATED WORK
Evidence Lower Bounds. Song et al. (2021); Huang al. (2021) derive the ISM and DSM lower- bounds on the model log likelihood. Our work extends their analysis to the multivariate diffusion setting to derive lower bounds on the log marginal of the data in the presence of auxiliary variables.
Auxiliary variables. Dupont et al. (2019) shows that augmented neural ODES model a richer set of functions and Huang et al. (2020) uses this principle for normalizing flows. Hierarchical vari- ational models and auto-encoders marginalize auxiliary variables to build expressive distributions (Ranganath al., 2016; Sønderby et al., 2016; Maaløe et al., 2019; Vahdat & Kautz, 2020; Child, 2020). We apply this principle to DBGMs, including and extending CLD (Dockhorn et al., 2021).
Learning inference. Learning Ga with pe is motivated in previous work (Kingma & Welling, 2013; Sohl-Dickstein et al., 2015; Kingma et al., 2021). Kingma et al. (2021) learn the noise sched- ule for VPSDE. For MDMs, there are parameters to learn beyond the noise schedule; Q can be non-zero, D can diagonal or full, give Q and D different time-varying functions, and learn VH.
7 DISCUSSION
We present an algorithm for training multivariate diffusions with linear timevvarying inference pro- cesses with a specified stationary distribution and any number of auxiliary variables. This includes automating transition kernel computation and providing a parameterization of diffusions that have specified stationary distribution, which facilitate working with new diffusion processes, includ- ing learning the diffusion. The experiments show that learning matches or surpasses the best fixed diffusion performance, bypassing the need to choose a diffiusion. MDMs achieve BPDs similar to univariate diffusions, with as many as three times more score parameters. The proposed MDM ELBO reports a tighter bound for the existing CLD relative to existing probability flow-based evaluations. This work enables future directions including interactions across data coordinates and using new stationary distributions.

---

Published as a conference paper at ICLR 2023
8 ACKNOWLEDGEMENTS
This work was generously funded by NIH/NHLBI Award R01HL148248, NSP Award 1922658 NRT-HDR: FUTURE Foundations, Translation, and Responsibility for Data Science, and NSF CA- REER Award 2145542. The authors would additionally like to thank Chin- Bein-Wei Huang for helpful discussing regarding Huang et al. (2021).
REFERENCES
Andrew D Barbour. Stein's method and poisson process convergence. Journal of Applied Probabil- Hl, 2554):175-184, 1988.
Andrew Campbell, Joe Benton, Valentin De Bortoli, Tom Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models. arXiv preprint arXiv:2205. 149888 2022.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron Weiss, Mohammad Norouzi, and William Chan. Wave- grad: Estimating gradients for waveform generation. arXiv preprint arXiv:2009.007/3, 2020.
Rewon Child. Very deep generalize autoregressive models and can outperform them on images. arXiv preprint arXiv:2011.10653 2020.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34, 2021.
Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Score-based generative modeling with critically- damped langevin diffusion. arXiv preprint 12.0006,,, 2021.
Emilien Dupont, Amaud Doucet, and Yee Whye Teh. Augmented neural odes. Advances in Neural Information Processing Systems, 32, 2019.
Bradley Efron. Tweedic's formula and selection bias. Journal of the American Statistical Associa- tion, 106(496))1002-16114 2011.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. arXiv preprint arXiv:2006. 11239, 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Sali- mans. Cascaded diffusion models for high fidelity image generation. J. Mach. Learn. Res., 23: 47-1,2022.
Chin-Wei Huang, Laurent Dinh, and Aaron Courville. Augmented normalizing flows: Bridging the gap between generative flows and latent variable models. arXiv preprint arXiv:2002.07101, 2020.
Chin-Wei Huang, Jae Hyun Lim, and Aaron C Courville. A variational perspective diffusion- based generative models and score matching. Advances in Neural Information Processing Sys- fems, 34, 2021.
Diederik Kingma and Max Welling. 1rXiv:1312.6114 2013.
Auto-encoding variational bayes.
arXiv preprint
Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. 2021.
arXiv preprint arXiv:2107.000
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009.0977 2020.
Yi-An Ma, Tianqi Chen, and Emily Fox. A complete recipe for stochastic gradient meme, Advances in neural information processing systems, 28, 2015.
Lars Maaløe, Marco Fraccaro, Valentin Liévin, and Ole Winther. Biva: A very deep hierarchy of latent variables for generative modeling. Advances in neural information processing systems, 32, 2019.
10

---

Published as a conference paper at ICLR 2023
Gautam Mittal, Jesse Engel, Curtis Hawthorne, and Ian Simon. Symbolic music generation with diffusion models. arXiv preprint arXiv:2103.1609) 2021.
Wenlong Mou, Yi-An Ma, Martin J Wainwright, Peter L Bartlett, and Michael Jordan. High-order langevin diffusion yields an accelerated meme algorithm. arXiv preprint
2019. Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. arXiv preprint arXiv: 10t.099200.00 2021.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:21 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
Rajesh Ranganath, Sean Gerrish, and David Blei. Black box variational inference. In Artificial intelligence and statistics, pp. $12. PMLR, 2014.
Rajesh Ranganath, Dustin Tran, and David Blei. Hierarchical variational models. In International conference on machine learning, pp. 324-333. PMLR, 2016.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. Interna- tional Conference on Machine Learning, pp. 1530-1538. PMLR, 2015.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Scyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image to-ee diffusion models with deep language understanding. arXiv preprint 2rXiv:2205.11488 2022.
Simo Särkkä and Arno Solin. Applied stochastic differential equations, volume 10. Cambridge University Press, 2019.
Hiroshi Sasaki, Chris Willcoeks, and Toby Breckon. Unit-ddpm: Unpaired image translation with denoising diffusion probabilistic models. arXiv preprint arXiv:2104.05358, 2021.
Jianghong Shi, Tianqi Chen, Ruoshi Yuan, Bo Yuan, and Ping Ao. Relation of interpretation of stochastic differential equations to ito process. Journal of Statistical physics, 148:579-590, 2012.
Jascha Sohl-Dicksteins Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing, pp. 2256-2265. PMLR, 2015.
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. Advances in neural information processing systems, 29, 2016.
Yang Song. Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. In Uncertainty in Artificial Intelligence, pp. 574 584. PMLR, 2020a.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020b.
Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum likelihood training of score- based diffusion models. Advances in Neural Information Processing Systems, 34:1415-1428, 2021.
Michalis Titsias and Miguel Lázzro-Gredilla, Doubly stochastic variational bayes for non-conjugate inference. In International conference on machine learning, pp. 1971-1979. PMLR, 2014.
Arash Vahdat and Jan Kautz. Nvac: deep hierarchical variational autoencoder. Advances in Neural Information Processing Systems, 33: 33:19667-19677 19679, 2020.

---

Published as a conference paper at ICLR 2023
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural compu- tation, 23(7):1611-167, 2011.
L Yin and P Ao. Existence and construction of dynamical potential in nonequilibrium processes without detailed balance. Journal of Physics A: Mathematical and General, 39(27):8593, 2006.
Zhenzhong Zhang and Dayue Chen. A new criterion on existence and uniqueness of stationary distribution for diffusion processes. Advances in Difference Equations, 2013(1):1-6, 2013.
12

---

Published as a conference paper at ICLR 2023
A AUTOMATED SCORE MATCHING WITH LEARNED INFERENCE
Like for the MDM ELBO, the methods in this work apply to training with the score matching loss:
Lsm (x, 0, ()- = T 1, (u) [A(() (1,0(y,, - Vy. log go xx) +
where X: [0, R+ is weighing function. The score-matching loss is often optimized in its
simplified noise prediction form:
(xr 0, 6) - 11. [Ieo(yy,, () =|
where 88 L, Eo and Y: = Mr + Lie and e is the noise used in sampling Y.. We describe here how the improvements to the ELBO studied in this work carry over to LsM and Lxp- In the following let go be the data distribution, let P(o.a).o be the model's 's distribution of the data, and recall that the model is defined by (sg, før ge) and prior и via a continuous-ting stochastic process with drift coefficient g2 So and and diffusion coefficient Jø-
First, minimizing Lsm or LxP so that yy, (o 5 so (y,, t) does not alone imply that P(o.a).o will equal Qo: it must also be that 9ø.r R и Foregoing this requirement means x will produce samples that the generative model may not be able to push onto the path the model was trained on (formally, the score of the generative model would not equal the time-reversal of the forward score even if se equals the forward score). This condition can be satisfied if go can be chosen with stationary distribution π. Section 3.4 describes how to accomplish this.
Next, for any fixed qo. automatic transitions from section 3.3 streamline the computation of the score matching loss, allowing for simple score computation for wide class of diffusions beyond VP.
Finally, for a fixed gs with qs,T Re π and a score architecture sg, minimizing Lsm or CN w.r. may be suboptimal. Optimization, like for the elbo, carries over to score matching and can close this gap; learning w.r.t. both 0, ø increases the ability to successfully minimize the loss at each f (section 3.5). In other words, since the generative model is defined by (50, fo: Io), learning q means the loss trains all three components of the generative model rather than just one. In summary, score matching is automatic and can learn over the space of linear diffusions that tend to the model prior.
B DOES MY MODEL USE AUXILIARY VARIABLES?
In section 3 we gave the example choice of ((yūl=)) = (0, I) coordinate-wise It is also common choice to set me = (O0 I). Because the optimum in diffusion models pe = q for all we see a peculiar phenomenon under this choice: the model has main and auxiliary dimensions independent at both endpoints 0 and T. Does this mean that the model does not use auxiliary variables? We show that even when (y)) and Ty have main and auxiliary variables independent, the model can the auxiliary variables. A sufficient condition is Q + D is non-diagonal.
To make this precise, recall that model with (u x). To show the model is using auxil- iary variables, we just need to show that uz (main coordinate at 7) depends on u² (aux. coordinate at l) for T L. At optimum, po po uf) qo y) Therefore it is sufficient show that for some time s, (y))))))) * go (y))) Because yū. is determined by
need to show that (zz) * go (y)). ). To do that, we first derive q(y, ( and then marginalize to get ( from it. Since the former is 2D Gaussian, the latter is available in terms of the former's 's mean and covariance.
Suppose E[yū] = 0, Q = [[0, -11, [1, 0j] and D = [[1, 0], [o, 1]] and we have = .1 We have:
E |x]] exp xQ D)]
0 exp [[= =]
00.9003z (0) (-0.0008
(18)
Regardless of the covariance any ID of this 2D gaussian will have mean that is function of x, meaning that q(y"|z) (x) does not equal q(y") (which also a Gaussian but with mean depending on x's s's mean rather than itself. Therefore, even under the setup with independent endpoints, the optimal model makes use of the intermediate auxiliary variables in its final modeling distribution (u = z).
Are there choices of Q and D that lead to learning models that don't 't make use of the extra dimen- sions? As mentioned, in the inference process, Q is responsible for mixing information among the
13

---

for Published as a conference paper at ICLR 2023
coordinates, and is the only source of this when D is diagonal. Then, if Q 0 and D is diago- nal, none of the coordinates for given feature x, (including u.y. u ...... 4t xx-1) interact for any t. Then, since Pg q at optimum, independence of the coodinates at all t in q imply the same in po and the model will not make use of any auxiliary variables when modeling the marginal log po (nz. = x).
C STATIONARY PARAMETERIZATTION
The non-linear time-homogeneous Itô process family is:
dy f(y)dt B,
(19)
This family can be restricted to those with stationary distributions. Ma et al. (2015) show complete recipe to span the subset this family with a desired stationary distribution. Let Q be skew- symmetric (-Q = Q and D positive semi-definite. Suppose the desired stationary distribution qo0 (oo (y). For a matrix A, let v A refer to the matrix square root defined by a VA =
aa Then, Ma et al. (2015) show that, setting H(y) = log 900 (y). = (D and
0 r.(y) Σ dz,
f(y) -Dy) Q T(y),
(Di + Q,y(y)), (y)),
(20)
yields process with stationary distribution goo. homogeneous) processes.
We extend it to time-varying (time in-
Theorem 2. Jo0 (y) expl- Hy)) is stationary distribution of
dy - (Dyy,t) QQy +(y, dt + (DD(,,)B,,
(21)
F,(y,t) Σ dy,
(Diy(y.t)+
(22)
Proof. The Fokker Planck equation is:
0edl,,1== = Σ à Ty [A Da(x + Σ 6,
0² Ty,0y By,
Diy ()(,
(23)
A stationary distribution is one where the Fokker-Planck right hand side is equal to 0. To show that the stationary characterization also holds of iimeiinomoogenuus processes with D(y, t) and t)y,t), we take two steps, closely following Yin & Ao (2006); Shi et al. (2012); Ma et al. (2015), but noting that there is no requirement for Q, D to be free First, we show that the Pokker-Plade equation be re- ee-ritten
Qcx, (p
Vols.o)
0.gly, =
(24)
Second, because the whole expression set to 0 when the inside expression equals 0
cly. ()y))))H())+))))) ()) =
(25)
just need to show that this holds when g(y, (x)p-H(()))/ The second step is concluded because
ogly, () + va(y. =
[exp[[(H(()] = 00
where Z is the normalization constant of exp(- exp(-H(y)).
14

---

Published as a conference paper at ICLR 2023
It only remains to show that Fokker-Plack can be re-written in divergence form with time-dependent 0 and let
Q, D. In the following let Qiye denote Qi (y, t) and likewise for Dajt- Let 0 denote ōy, denote ay, for scalar functions. We will use [Ar] = Ej Aejzn.
dige= V. (D (D(y,t) Q0y Vol)
[a Val
Σo. 0, EiP Q Val,
Ea, 0, ΣiPir Q001100, 0,4
Σ 0, DiDijt + Qujellad H Σ 8. E(Des + Qbye
= Σ 0 > Pije + Qudl1od, H + Σ 0, > 1ip0 + D. Qipl0gal
3
3
We re- write the 2nd and 3rd term. Holding i fixed and noting q is scalar, we get the product rule Ej (d)r( = Σj 0, Pejal 9 9E, 02 Deje for each i, and likewise for q:
Σ 0: > Dija Qual H|
" 0. E
Degs 15
Σ 0: Σ Rijh
Σ 0 > Dijt Qistll90,HI H| Σ 0, Σ
0 Dijia - -9
0, Dige
Σ 0 Σ 0, 01(ii 0
Because Q t) is skew symmetrrme, we have that Σ, 0. 0, 05QQua = 0, leaving
0.gt Σ a
Σ Disa + Qu11l90,H1
+ Σg a
Σ La -9 -o E0,0,
Σ a
ΣDist + Qu0110HH19
+ + a Σ a,[Dijual - ag + Qui))
0, + Quo)) + +ΣΣ 8²
Σa a
Dije + QQdd10HH
yW (Dija)
Recalling that fly, (00))(((110))) (-ID+Q/VH++) and again that [Ax]; E, Aijz,, we have equality with the original Fokker-Planck
Σ Dist + Qistl10,H1 Oo + +Q5))) + Σ 8²
= 0
(Dijeg)
y.y,
a
Σ
Σ ij y,y,
Ty
= 0.gly,0 1)
We have shown expl-H(y)]/2 is a stationary distribution of the time-varying non-linear Itô pro- cess:
dy DDy,t) QQy, F(y, dt + 22D(y,)) t)B..
(26)
15

---

Published as a conference paper at ICLR 2023
However, for some choices of Q. expl-H (y))/ is not necessarily the unique stationary distri- bution. One problematic case can occur as follows. Suppose that row i (Q (Q+D) D) is all-zero; in this case, dy; 0 which implies that (y:), (y:)n for all 0. Then, the initial distribution is also a stationary distribution. To rule out such pathological diffusions, we make the assumption that Q +
D is full rank. Then, for uniqueness, recall that stationary distributions are the zeros of
o.gly, = v. (D ((xe))+Q(x11) Q(y, ((x, vV(
where the expression of the form Av for A D(y, 8) Q(y, ((y,t) t) and gly: (y) + Vg(y,
Under the assumption that Q + D is full rank, the expression can only be zero when zero. To show uniqueness under the full rank assumption, one must then show that
Vgly, - (V
holds only if () = expl-H(y)l) /Z. Even if [xxp]=H()]// J is the unique stationary distribu- tion, convergence to that distribution question. See Zhang & Chen (2013) for more details.
Learning Q₂ Di in the MDM ELBO helps push Yr to the model prior my and avoid issues like those discussed.
C.I LINEAR PROCESSES
Next, we specialize this general family to linear Itō processes to maintain tractable transition distri- butions. A linear process is one where the drift f(y,t) and diffusion g(y,t) are linear functions of We express the drift function of a non-linear time-varying Itô process with stationary distribution proportional to exp]-][(y]] as
-QQ t) + D(y. ()) + t).,
Next, linear Itô processes have Gaussian stationary distributions (Särkkä Solin, 2019) H(y) must be quadratic and V (y) is linear, and neither are constant in y. Because V VH is linear, can be expressed as Sy for some matrix S where S is the inverse of the covariance matrix. Because V is multiplied by Q D. this means that Q, D must be free of y. Recalling that is expressed as a sum of derivatives w.r. y of Q + D. this means that I must satisfy r O. Next, because of the stationary requirement that gt) that the diffusion coefficient function must be independent of the state y. Our final form for linear vire-ng processes with stationary distributions N N(00 1-1 is:
(y, t), we can also conclude by the restriction on D
dy Q(0 + D(1) Sy di 22D(1)
(27)
f(y.t)
g(()
C.2 PARAMETERIZING Q2
Suppose (s) is positive scalar function defined the time domain with known integral. Suppose Q is any matrix. Then Q Q is skew-symmetric with Q Poji We can set Q₂ to
(s( by s) [Q, 2
(28)
This is a general parameterization of time- ime-inddepnndent skew-symmetric matrices, which have num- ber of degrees of freedom equal to the number of entries in one of the triangles of the matrix, excluding the diagonal.
C.3 PARAMETERIZING D
Suppose ba(s) a positive scalar function defined on the time domain with known integral. Suppose D. is any matrix. Then D is positive semi-definite and spans all time-independent positive
16

---

Published as a conference paper at ICLR 2023
semi-definite matrices. We can set D to
(o(s)= = ba(s)
D
(29)
To show DD spans all positive semi meni-definite matrices: suppose M positive semi-definite. Then it is square. Then it can be eigen sigen-decomposed into M VEV The degrees of freedom in VEV! are just R VvE since VEV = R.R. and the square root is taken element-wise because Σ is diagonal and is real because each Σnj 0, which is true because M is positive semi-definite. Take D R.
In experiments parameterize D diagonal-only matrix.
C.4 INTEGRALS
The known integral requirement comes from the integrals required in the transition kernel, and can be relaxed two possible ways:
numerical integration function with unknown integral. This is expected to have low error given that the function is scalar-in scalar-out.
Directly parameterize the integral and use auto-grad when needing the functions not- integrated.
We stick with the known integrals. In conclusion, the underlying parameters are positive scalar functions (s), ba(s) defined on the time domain and with known integral, and general matrices Q₂
C.5 INSTANCES
VPSDE. VPSDE has K = 1. Consequently, Q. are K K. The only x skew-symmetric
matrix 0, so Q Setting D(1) ()((1) recovers VPSDE:
3(1) dy = 2 ydt + B(tt)BB
(30)
( y so H(y) 11. The stationary distribution is N I).
CLD. The CLD process (eq 5 Dockhorn et al. (2021)) is defined as dz dv, = dy. =
(-) 3
0
(
0
1
yt +
v 2
dB..
M
In Q/D parameterization, we have
1 2
H(y)
|
2
+
||
V.H(y)=
2M
0
3
0 0
Q 3
D
0 r3
The stationary distribution of this process is:
Gd.cc x (xp(-H(y))
0, O,
(31)
ALDA. Mou al. (2019) define third-order diffusion process for the purpose of gradient-based based MCMC sampling. The ALDA diffusion process can be specified as
0
1
0
Q
1
71
D
0
0
(32)
II
8
Note that Q is symmerrii and D is positive semi-definite therefore we have that ge(u) -
00.00- In this case,
-0.00 = -, - (1))) I.)
17

---

Published as a conference paper at ICLR 2023
MALDA. Similar to ALDA, specify diffusion process term MALDA which specify
t 0
+
0
Q
7
1
D
0
1
0
(33)
7
0
Note that is skew-symmetric and D is positive semi-definite In this case this is
66,00 = N(2 + (a) 0, =(a
1
D TRANSITIONS FOR LINEAR PROCESSES
For time variable & and Brownian motion B, driving diffusions of the form
dy [(y, s)ds g(s)ddB
(34)
when f.(y,,)) (s))) are linear, the transition kernel (y, is always normal (Särkkä & Solin, 2019). Therefore, we just find the mean maju and covariance Σdjo of 4(y,lyo). Let f(y, A (s)y. The un-conditional time & mean and covariance are solutions to
dm./ds (s) /2//ls A()), + E. ( gg))
(35)
By (6.6) in Särkkä Solin (2019), for computing conditionals g(y=lyu). we can take the marginal distribution ODEs and compute conditionals by simply setting the time 0 mean and covariance initial conditions to the conditioning value and respectively. We take (6.36-6.39) and set my ug
and Eg = 0 to condition. Let [A]s = So (vV)dv. The mean is
mgj0 exp 10 A(U)du yo =0=-e =xx ([*].)
exp(sA)y
(36)
no integration if A(r) A
where exp denotes matrix exponential. (6.36-6.39) state the covariance g(y,lyn as matrix factor- ization, for which a derivation provided below Σ, CH, for C., H, being the solutions of:
C.
A(s)
(s)
C. H
ds H.
(s)
A
(37)
To condition and get Egjo from E., we set Eg
O, and initialize C.,, H, by C. and Hg =
H
C2
exp
(1A'=
[A
L
1.)
(
exp 5
A
A
(38)
no integration if A(v) A. A,g(v)
Finally, Ee10 -H,
D. DERIVATION OF THE COVARIANCE MATRIX SOLUTION
Equation (35) gives an expression for dE./ds. To derive the matrix factorization technique used in
eq. (37), we use eq. (35) and the desired condition Σ, = C.H.1 to derive expressions for dC,/ and dH./d. /ss and suitable intial conditions so that the factorization also starts at the desired Eg. Let Σ. C.H.1 then note that C,, H, satisfies
d Σ,
d
C,
ds
ds
C. d H
(
dd C,
+
H.
ds
ds
18

---

Published as a conference paper at ICLR 2023
And using the fact that
H 8 0
ds
d
+ d
Hg
H
H. (H 1 = 0
ds
ds
d H 1 HH," d ds
H. (H.")
ds
get that
d
d
H. = C, H. (H,)) + (
C
H +
C,
C, H.¹
ds
ds
ds
ds
-C,H, - H. (H
+
C,
d
H. = A(s)C,H, A. (s)
ds
A, + C.H, A HH, +g H.
C,, d
H.
- C,
d
H (A(s)C, + C.H, A (s) + H.) H.
ds
ds
C.H, d H. +
d
C, = A(s)C, C,H, A (s) + (s)
ds
ds
C.H,
La ds
d
H.
[C.H,' L.
(s) A(s) H
C
Now, we note C,,, H, satisfy the following
d
H H,,
ds
d
C. +g
ds
which implies that
d
C,
A(s))
(3(s)
)
C.
ds H
=
0 -A (s)) H.
(39)
with C. Eg and Ho L,, CoH Eg.
D.2 HYBRID SCORE MATCHING
Instead of computing g v,y. .o). we can apply the hybrid score matching principle (Dockhorn al., 2021) to reduce variance by compute objectives using ((y,lz) instead of gl).)yo), which amounts integrating out Vo. To accomplish this, following Särkkä & Solin (2019), we simply replace Yo with [r. [[vo]] in the expression for m.jo- i.e. replace the conditioning value of vo with the mean of its chosen initial distribution:
Efy,la] = exp [ A(v)du) (B(vol)
(40)
For the convariance, instead of using Co = En O, we use block matrix to condition on but not Vo- We decompose Eg into its blocks Eo, Eo,ev Eo As before, to condition on æ we set En.re 0. Because q(vo) is set be independent of Σa,rv is also set to O. Finally, instead of O, to marginalize out Vo. Eg,ve is set to the covariance the chosen initial time zero distribution for Vo- E.g. if vo.j NO. 7) for each dimension, then Σg... (I.
We operationalize this in simple piece of code, which makes the ELBO tractable and casy, i.c. skips both analytic derivations and numerical forward integration during training.
19

---

Published as a conference paper at ICLR 2023
D.3 TRANSITIONS IN STATIONARY PARAMETERIZATION
In of Q, D, the transitions (Yo))) for time are normal with mean majo and 30 equal
maj0 exp (- [Q-D)))
C₂ H.
[-10 + D],
2DD
)
(i)
cxp
0
IQQ D)
(41)
where Σnjo CH, For the time invariant case, this simplifies to
C2
D)
2D (Q
)
0
mg exp D)
H
exp
0
D)
T
(42)
E GENERIC CHANGE OF MEASURE AND JENSEN'S S FOR APPROXIMATE MARGINALIZATION
Suppose u [z, v] and we have an expression for p(u u)) - p = 2, v v). By marginal- ization, we can get p(z 2). and can introduce another distribution q to pick sampling distri- bution of our choice:
p(z = 2) =
plz V = v)dv
1
plz 2/v v)p(v v)dv
(43)
q(v = z 2)
1vv z 2) p(z 2/V =vp((v v)dv
plz 2, = U) E Eg(v=o|a==) =) q(v u|z
We often work with these expressions in log space, and need to pull the expectation outside to use Monte Carlo. Jensen's 's bound allows this:
plz = 2, v =
logp(z z) = log By(v=r(/==)
g(v v z
plz 2, V = v)
Bg(v= log
q(v = z 2)
The following shows that the bound tight when q(v= u/z 2) p(v z/z 2):
By(v=u(e==) log p(z = 2, u) q(v u/z 2)
log plz v = 4)
assume Ep Ep(v=e/e==)
plv v/2 =
plz 2, =
E Bp(v=o/==2)
log
p(z p(z==
p(v U, 2 = =)
(44)
logp(z = z))
Bp(v=e|x=z))
log pl z = z)
20

---

Published as a conference paper at ICLR 2023
F ELBO FOR MDMS
log pe (x() log
pe po vo)doo
(45)
log
po(uo vol)
(46)
{
g(volx)) (vvolz))
log
Pe (uo = [z, vol)
(47)
po = [zz vol)
log (vollr)
(48)
((volz)
Eg(vola)
logpoluo = [ar, vol) - log ((ol)))
(49)
2 E (xl2)) logTo(yr) + / -1ssoll3- - V -0"920 f)ds log ((16)=)) x))
(50)
The first inequality holds due to Jensen's ' inequality and the second due to an application of Theorem 1 from Huang al. (2021) or Theorem from Song et al. (2021) applied to the joint variable uo.
F.1
ISM TO DSM
F.1.1 LEMMA: EXPECTATION PARTS
We will need a form of multivariate integration by parts which gives us for some f and some q(x),
EV f(x)] - (x) V, log g(z)]
Eg(z) V. [((x)] = ʃ ((x))
V 2, [i()))dx
= 1 Σ 2z,
Σ Fi
Σ / [((an) _ - / de
Σ V ((=)) di
Σ 1 / v(x) logg(z) (og dr_,
/ / Vz) log (
Bq(t)) Vn, log (ogg(()fi(x))
= Ex(t) (f(x) V log ((x)]
This equality also follows directly from the Stein operator using the generator method to the Langevin diffusion (Barbour, 1988).
21

---

Published as a conference paper at ICLR 2023
F.1.2 DSM ELBO
Using the "expectation by parts", we have:
[agurla][Vs (t) (ue, t)] Bo [ng (t)sa f)) V
T logg(u((
Also we have, for 59 evaluated at (ue, 1). by completing the square.
1 - マ 2 (001l10000 + + o² log (())))) = - -21100 v log (0g)(()))))) + Vo (ogg(w))>))
The two together give us:
[g(ue(z)) V. .ggss 511s0113200) + V / flat)
log logp(x) Be(arla)
log π
x log /--² [e((s(())
(g2s0) Vu, logg(ur(z) -5|1s0 -511501(3000 + V r|at
[x(
T
1 -51lsp V 10gg((r(>)13340
(x)
log
E Eo(uslz)
[Vogg(u<<>))830 Vue - dt
(51)
F.2 NOISE PREDICTION
We have that for normal (y.; m.jo: Edjo). we can sample y, with normal noise ~ NN(0 7) and Y. m2jo LE where L is the cholesky decomposition of Σnjo Then, the score is
y, log y.=m,jo+Ie
=-Σ
y, mm
maju + Le - m,jo
Σn1
Le
[LL
Lc)
(L") L 'Le Le
- (LT) - (I.-1) - 6- -- TT-1
Parameterize so(y,, 8) as so(y.. - T, co sy) This gives
T,-1
T,- co(y,
-L T.- ell
-I
2
1
T T. -T.-ccoly.oo s)1l33(00
L
2
L
Ti-le
L TT te (s) L LT-lle - L LT-1 (o
2
1
L
Ti-
c so(y,s)) (o(s)
L LT.- co(s.s))
2
22

---

Published as a conference paper at ICLR 2023
We can also this insight analytically compute the quadratic score term (following is computed per ditendimeesson. so must be multiplied by D when computing the ELBO):
"yolBy.lv Vy, log (x
Byo 1y..o
Vy, log go (y (6((s) (Vy. log go (xxxsll)))
(L (8(0) - [TT
-By, By,No
-y, By.Bo
c c- (-L (s} T
=By By lvo
6
-Ey, E.
T (1)))(()) 1--1)
(L- (2) (6))) Ti-1)
E,
Trace
L L- (2 (+)
G ELBOS IN STATIONARY PARAMETERIZATION
We use the stationary parmeterization described in appendix C. We now specialize the ELBO to the linear stationary parameterization.
Recall (y, +) - (s)() (oly) Recall go (s) = v (+) We have gz(3) = (s). We
can write the MDM ISM ELBO as
m Eungg Eg-Uni((O [((tom)) lr + Eg
(52)
where
1 11s0(y,. s. $)1
l, = 2
3)12D (s)
ldiv.fgs Vy. (1p((s) (s()) (s) so 8)
So Cap + ldit.
(53)
glam E (an,
de Q.D
eT E Lo.rr (x, v)
log To (Xx)
deyasts on Q.D
& = log og (1(22)
For the DSM form,
mddsm -Eong
Em-Unif(O.T) [(ddmm)] + Ey lgg
(54)
23

---

Published as a conference paper at ICLR 2023
where
laiv. = Vy, (QQ((s) + +Da(s)ly,
f0
ffwd-score
Vy, log ts (y vo) 2D (s)
depends on Q,D
pogg-scoredift
2 --21s8((ss:)) - Vy, log (F,l8o)
12 (x)
depends on Q.D
p(dem)
E (1,0) dependic Q.D
Eoog-scoreditt liwd-score + ldirs
24

---

Published as a conference paper at ICLR 2023
H ALGORITHMS
H.1 GENERIC TRANSITION KERNEL.
Algorithm 2 Get transition distribution y,
Input: data a. time 8. A,
compute: (s) and g(s)
compute: M, -l A (f)dt (integrated drift)
compute: N₂ = -o (()dl (integrated diffusions squared)
compute: 7s/0 exp M. (mean coefficient)
set: Yo [z, 01 ... K-
Σ0,z2 and Σ0,20; Do,en chosen initial distribution
compute: majo = o (mean)
compute:
)
C2 H
MM
N₂ -M,
-
En
exp
(ingredients for cov.)
(55)
compute: Σgjo C..)) (cov.)
Output: ((m.jo: Σx10)
H.2 TRANSITIONS WITH Q
Current param matrices Q₂. Do and along with fixed time-in scalar-out functions (s), ba(s) and their known integrals By (s), Ba(s). (200/22 = x) taken to be parameterless so that eg N(0, 7). Model params are 59 fixed
Algorithm 3
Get Q, D and their integrated terms M,N N
Input: time and current params
compute: [byle = S by (v)dv using known integral (s - B0(
compute: [bu]s = f ba(v)dz dv using known integral Ba(s) Ba(
compute: [Qp]a [6y].
for current params Q₂.
compute: [De] ps [ba]
D
DeD
for current params D
compute: M, - ([Q5]
+ D Dola) (M just variable name)
compute: N, = [2D
2
[ (N just a variable name)
compute: Q, = bs(s)) Q
(not integrated)
compute: D. = ba(s) D&D (not integrated)
compute: A, -QQ, + D.] (drift coef.)
compute: o² 2D. (diffusion coef. squared)
Output: A 9. M,, M.
H.3 ELBO ALGORITHMS
25

---

Published as a conference paper at ICLR 2023
Algorithm 4 Get transition distributions
Input: Sample Yo (z, e) and time s. Current params &
set: A.: g. M. N, algorithm
compute: m,jo = exp [, Yo (transition mean)
compute: ingredients for transition cov. matrix:
C2 H
exp [ M
N₂
)
(1)
MM
(56)
compute: 20j0 = CH, (transition cov).
instantiate: (0,,,(a, = q (y.))) (m Enjo).
Output: (z,,, As, g
Algorithm Compute ELBO with ism or dsm
input: Data point z and current params 0, o
draw: sample v q- (v/2z)
draw: sample ~ Uni
set: Yo = (I,
set: 4e. A.: + algorithm 4 called on Yu. 5
draw: y, 96.s.v
compute: 4, with dsm(s) (algorithm or ism(s) (algorithm 7) on y,, As. ge 90.5.0
set: qe.T.y-' + algorithm 4 called on Yo. T
draw: YT ~ Qa.T
output: f, log (y) log (w)
Algorithm Compute dsm(s)
input: y., 0, As, -. -o.
compute: fwd-score Vy, log loggolys<yo)
compute: model-score soly,,
compute: fwd-score-term
(core)corre 9² (fwd-score)
compute: score-diff = model-score - fwd-score
compute: diff-term = -score-diff" gf score-diiff
compute: div-f Vy, Aay,
output: dsm(s) s) = wwd-score-term + diff-term + div-f
Algorithm Compute ism(s)
input: y. 8, Ag, &. Jo.-No'
compute: model-score = so
compute: score-term
model--sore modeess-scrre
compute: div-gs Vy, gg
compute: div-f Vy y. Asy,
compute: div-term div-gs div-f
output: ism(s) = score-term div-term
I VALID ELBO WITH TRUNCATION
The integrand in the ELBO and gradients is not bounded time Therefore, following Sohl- Dickstein et al. (2015) and Song et al. (2021) the integrand in eq. (7) is integrated from [e, T"), rather than [o, T]. However, that integral is not valid lower bound on (x().. Instead, can be viewed proper lower bound the prior for a latent variable y.- Therefore, to provide bound for the data, one can introduce a likelihood and substitute the prior lower bound into standard variational bound that integrates out the latent.
26

---

Published as a conference paper at ICLR 2023
To provide valid lower bound for multivariate diffusions, extend theorem 6 in Song et al. (2021)
from univariate to multivariate diffusions.
Theorem 3. For transition kernel go(yss yo), compute upper bound the model likelihood at time O as follows. for any c
log Po(Yo ye) go yo) + Cmmm (Ye, c) - log go (yö 22)
log log))(x)== 2 E (a) E Bo
(57)
where Cy E) is defined as fmmam (e; = (y log fogTolyT)
T [(]
1 2
||
11:0
80
V
So
Proof. For transition kernel (y. yo), we can compute upper bound the model likelihood at time O following an application the variational bound
log po (2)) log
po(yo
[xx, dol)
log
po po N. )doodye
vo.y.
x)) (yo. dundy.
log
golye yy)
x g yo)
vo.y
yo) 9(2
x) Po
y. (y).) dundy.
log
go
x) (
yo)
vo.y.
log (y y.)
Egrro (x.
log 2 (y æ) + (y
go(ye yo)
A lower bound for log (y:) can be derived in similar manner to eq. (7), such that
log (ye)) 2 L.mam (Ye; c) E 1l log (y - [." |1s0l 2
1 ||s0
The choice of pe (yo ye) arbitrary, however following Sohl-Dickstein et al. (2015); Song et al.
(2021) we let pa (Yo yy) be Gaussian with mean Ppo.e and covariance Epo.e. Suppose 4s (y.
yo) -N(y. Ayo, E), then we select the following mean /'pa.c and covariance Epp. for (Y yc)
Ppp. A -1 c) A A
Epp.c ---AAAA
where pps.e: Epp.c are derived using Tweedie's s formula (Efron, 2011) by setting μe Elyo yel and Σ. = Var (Yol y.).
We next derive this choice as an approximation the optimal Gaussian likelihood.
I.1 LIKELIHOOD DERIVATION
Suppose yo (y)) and y. ~ ~Nyy Ayo. E). Here, A.E are the mean coefficient and covari- ance derived from the transition kernel at time e. We use Tweedie's formula to get the mean and covariance of Yo given y, under q. This mean and covariance feature the true score Vy, log g(y.) We replace the score with the score model 8y and then set (o to have the resulting approx- imate mean and covariance. We make this choice because the optimal pe (Voly.) equals the true g(yoly.) ) as discussed throughout the work.
Here Yo [Xo, vo] where Xo Qdata-
Let be the natural parameter for the multivariate Gaussian likelihood N(y. Tweedie's formula (Efron, 2011) 1) states that:
Ayo, E). Then,
Fly u
(yy)()))= Vy. loy.)
27

---

Published as a conference paper at ICLR 2023
(yye) = (y))
sy <) is taken to be the true score Vy, log so that Vy.)(y.) = (ve. c)
lo is the log of the base distribution defined in the exponential family parameterization.
The base distribution a multivariate Gaussian with mean and covariance Σ, therefore Vy loy.)) -
Bin y.] so(ye: +
However, Tweedie's formula is not directly applicable since our y, is not directly normal with mean Yo- Instead, to derive the conditional mean of Yo given Yo. we use the relation 7) Σ -'A and the linearity of conditional expectation get
Elyo ye] = EiA Enly.
A =A-'EE(n yel
A A (yo()er)) +
= A A -
For the variance, we use the following relation Ayo + VEe, which implies that
Yo A A - VEe
Var (yo ye) = A A-EEA-
Therefore, for the model posterior distribution Po ye) we choose Normal with mean and covariance
Ppo,e = A A A5 c) + A A
Eppoc A EEAAA
28


Reviews:
[{"review_id": "X7EKpYqXUh_", "reviewer": "Reviewer_bfsX", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces Multivariate Diffusion Models which extend standard diffusion models to include auxiliary variables. Towards this end, the authors provide a new learnable inference and generative process that generalizes many familiar diffusion SDEs (e.g. VP) and provides a modified ELBO required for training. The authors also introduce two linear diffusion models in ALDA and MALDA and show that these lead to state-of-the-art BPD and NLL on MNIST and CIFAR10.", "summary_of_the_review": "Overall this paper is of high quality and has a nice theoretical component. The experiments are limited to small datasets but there does not appear to be a significant bottleneck in scaling them. Based on this I recommend accepting this paper with a score 8.", "strength_and_weaknesses": "First, the paper is exceptionally well written, and the main ideas and motivation are clear and convincing. I really found the idea to include auxiliary variables as a compelling step forward given the numerous examples previously in machine learning (e.g. Augmented normalizing flows). I was initially concerned that having $K$ variables per dimension might incur a high computational cost but table 1 suggests otherwise. \n\nFrom a theory standpoint, I must admit I did not have time to go through the details in the appendix but the items presented in the main paper appeared sound. The main insights of using linear diffusion and computing the transition kernels (eq 15-16) felt a bit opaque to me in the main text but I understand this is fleshed out more in Appendix C. It would be nice if the authors could move some of this material to the main text as well to help readability. The fixed inference parameterization and its extension to a learnable parameterization sounds reasonable, but as a question it appears that we get this for free? So are all MDM models have learnable inference processes by construction? It would be nice to see an ablation on this as table 2 is not very clear on this point.\n\nI did not find many clear weaknesses in this paper. But I will outline some areas of improvement. First tables 2 and 3 can really be combined into one table, the results are identical for MDMs. Secondly, the experiments are a bit limited as they only include MNIST and CIFAR10. Most diffusion papers also include a larger dataset in Imagenet 32 X 32 or Imagenet 64 X 64. It would be good to see if MDMs are equally performant in this setting. Also, it would be nice to get a convergence vs. timestep analysis on ELBO for MDMs vs other diffusion models. Do MDMs converge faster due to higher modeling flexibility? Finally, it appears ALDA is significantly worse than MALDA? The intuition for this is not very clear to me, can the authors provide more details here?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "HlUaP2LhipW", "reviewer": "Reviewer_rSRA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper provides a general derivation of Evidence Lower Bounds in Multivariate Diffusion Models and some parametrization schemes. The paper also introduces MALDA diffusion model under its framework and evaluates it on MNIST and CIFAR10.", "summary_of_the_review": "The paper is not novel and ill-structured.", "strength_and_weaknesses": "## Strength:\n\nThe derivation seems to be unfied and generic. The experimental results seem good.\n\n## Weakness:\n\n### Novelty:\n\nThe novelty of the paper is marginal: \n\n(1) For the theoretical derivations, all the results are readily known from previous papers (e.g., Huang et al. (2021); Durkan & Song (2021), Dockhorn et al., (2021)). The paper is just summarization of the previous derivations using a generic framework. \n\n\n(2) The empirical results are not novel and lack motivation: ALDA is introduced in Mou et al. (2019). The modified version (MALDA) is not well-motivated (at least not shown in the main text) and only marginally different from the ALDA method.\n\n\n### Clarity:\n\nThe paper is a ill-structured and unclear. \n\n(1) Most of the details are deferred to the Appendix, making the paper hard to follow. \n\n(2) The theoretical part of the paper is not structured. It is written as derivations of formula without clear goals. Assumptions are added to the derivations in the middle, e.g. Section 3.3 assumptions on $f(A,s)$; Section 3.4. linear case. Instead, the best way to present theoretical results is through a list of assumptions and theorems. This makes follow-up papers easy to verify the assumptions and use the results without knowing the details of the derivations. (Same philosophy as abstracting codes through APIs, if the authors are more familiar with coding and engineering.)\n\n(3) The experimental part lacks motivations and exposition of the proposed method. E.g., what is the difference between ALDA and MALDA? what is the motivation for MALDA? Do you claim ALDA as your own contribution, as in the abstract and introduction section; or do you give full credit to  Mou et al. (2019)?\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1sYR6nF-IA", "reviewer": "Reviewer_yDwB", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to improve diffusion probabilistic models by considering multivariate diffusions. The motivation inherits previous probabilistic modeling methods that use auxiliary variables. By augmenting the diffusion space, it enables mixing between data dimensions and auxiliary dimensions, leading to better-aligned generative and inference processes. Based on this the paper further demonstrates how to parameterize the inference process, e.g., MALDA. Experimental results are promising in terms of test likelihood estimation.", "summary_of_the_review": "Overall, I think this paper is of good quality, in that it presents a novel and interesting perspective of understanding and a concrete approach for diffusion models, achieving good results on standard benchmarks.", "strength_and_weaknesses": "Strength:\n\n- The paper is well-written and introduces an interesting novel approach for establishing diffusion probabilistic models in auxiliary space.\n- The motivation of the work is explained very well in Section 1 and Section 4, which is helpful for understanding the paper.\n- Technical contributions are solid (to the best of my knowledge).\n- Experimental results achieve state-of-the-art likelihood (in terms of bits-per-dim).\n\nQuestions:\n- How is the auxiliary variable initialized in time zero? Is it sampled from $q_\\phi(\\mathbf{y}^v_0|x)$ during training? It seems that $q_\\phi(\\mathbf{y}^v_0|x)$ can be any distribution in theory. In the paper, an example of $q_\\phi(\\mathbf{y}^v_0|x)=\\mathcal{N}(\\mathbf{0},\\mathbf{I})$ is given, but this could be quite different from the data distribution, i.e., $q_\\phi(\\mathbf{y}^z_0|x)$. Since in the practical implementation the score for each dimension is output from different channels from the same model, I wonder whether a better choice of $q_\\phi(\\mathbf{y}^v_0|x)$ could alleviate the training burden and perhaps results in a better likelihood or faster training?\n\nMinors:\n- In Eq.(18) no text explanation under the underbrace.\n- On page 6 there is something wrong with the typography.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster