Likelihood Training of Cascaded Diffusion Models via Hierarchical Volume-preserving Maps
Authors: Henry Li
Ronen Basri
Yuval Kluger
Venue: ICLR 2024 spotlight (2024)
Keywords: likelihood-based modeling
diffusion modeling
density estimation
Primary area: generative models
TL;DR: We develop a framework for cascaded diffusion modeling that supports likelihood training and evaluation.

Abstract: Cascaded models are multi-scale generative models with a marked capacity for producing perceptually impressive samples at high resolutions. In this work, we show that they can also be excellent likelihood models, so long as we overcome a fundamental difficulty with probabilistic multi-scale models: the intractability of the likelihood function. Chiefly, in cascaded models each intermediary scale introduces extraneous variables that cannot be tractably marginalized out for likelihood evaluation. This issue vanishes by modeling the diffusion process on latent spaces induced by a class of transformations we call hierarchical volume-preserving maps, which decompose spatially structured data in a hierarchical fashion without introducing local distortions in the latent space. We demonstrate that two such maps are well-known in the literature for multiscale modeling: Laplacian pyramids and wavelet transforms. Not only do such reparameterizations allow the likelihood function to be directly expressed as a joint likelihood over the scales, we show that the Laplacian pyramid and wavelet transform also produces significant improvements to the state-of-the-art on a selection of benchmarks in likelihood modeling, including density estimation, lossless compression, and out-of-distribution detection. Investigating the theoretical basis of our empirical gains we uncover deep connections to score matching under the Earth Mover's Distance (EMD), which is a well-known surrogate for perceptual similarity.

Published as a conference paper at ICLR 2024
LIKELIHOOD TRAINING OF CASCADED DIFFUSION MODELS VIA HIERARCHICAL VOLUME-PRESERVING MAPS
Henry Li', Ronen Basri Yuval Kluger Yale University, Meta AI, Weizmann Institute of Science
(henry. 1i, yuval. kl klgger @8ya eyyale.co edu ronen. Jonen.basrien www.:baaariiweeeeeem ac.
ABSTRACT
Cascaded models are multi-scale generative models with a marked capacity for producing perceptually impressive samples high resolutions. In this work, we show that they can also be excellent likelihood models, so long as we overcome a fundamental difficulty with probabilistic multi-scale models: the intractability of the likelihood function. Chiefly, in cascaded models each intermediary scale intro- duces extraneous variables that cannot be tractably marginalized out for likelihood evaluation. This issue vanishes by modeling the diffusion process on latent spaces induced by class of transformations we call hierarchical volume- seenee-reerrving maps, which decompose spatially structured data in hierarchical fashion without intro- ducing local distortions in the latent space. We demonstrate that two such maps well-known in the literature for multiscale modeling: Laplacian pyramids and wavelet transforms. Not only do such reparameterizations allow the likelihood function be directly expressed the Laplacian pyramid and wavelet transform also produces significant improve- ments to the state-of-the tf-t-e--rttttt on selection benchmarks in likelihood modeling, including density estimation, lossless compression, and out-of-din di-stiibbutiinn detec- tion. Investigating the theoretical basis of our empirical gains we uncover deep connections to score matching under the Earth Mover's 's Distance (EMD), which is a well-known surrogate for perceptual similarity. Code can be found at this https) url
joint likelihood over the scales, we show that
INTRODUCTION
A widely used strategy for generating high resolution images x is to first draw from low resolution model (z then refine this representation with series of super-resolution models (z(z (=-))())1))
for k = 2, ...... S where z = x. This idea, known as cascaded or multi-scale modeling. has been employed to great effect in generative models large (Oord et al. (2016) Karras et al. 2017 De Fauw et al. 2019), as well as diffusion models in particular (Ho et al. 2022). However, multi-scale models pose a fundamental issue for likelihood computation, as the modeled likelihood Lunction-obtainees as product of all conditional super-resolution models and the base model - is now the joint likelihood Pe (1) ()...,,, z (k)) rather than the desired likelihood (x) (z)) This prevents cascaded models from being used for likelihood- training and inference on x. For (x(x) to regain exact evaluation capabilities, the intermediate resolutions must must either be marginalized out - a very expensive operation high dimensions - or undergo complex redesign to avoid such marginalization (Menick & Kalchbrenner 2018 Reed et al. 2017).
In this work, we propose a different approach to overcoming this problem using class of transforma- tions we call hierarchical volume-preserving maps. These are special subset of homeomorphisms to which the likelihood function is invariant - i.e., a class of functions H/ such that for all h E H. pe(x) po (h(x)). Under these transformations, the joint likelihood computed by cascading model coincides directly with the desired likelihood function po (x). When these transformations are simple and produce useful hierarchical representations of x, are able to retain maximum likelihood

---

Published as a conference paper at ICLR 2024
Figure 1: Images generated from our W-PCDM model trained on unconditional ImageNet 128x 128.
training and evaluation capabilities in cascaded multi-scale setting while introducing little to no computational overhead. We evaluate our multi-scale likelihood model on a selection of datasets and tasks including density estimation, lossless compression, and out-of-distribution detection and observe significant improvements to the existing state-of-the-art, demonstrating the power behind multi-scale prior for likelihood modeling.
We subsequently turn to theoretical analysis of our framework, and show that maximizing the log-like ligllikel of proposed model is equivalent minimizing an upper bound the weighted sum (or integral) of Earth Mover's 's Distances (EMD) on the marginal scores the diffusion process. Distance
The Earth Mover's computer vision known for its properties as high quality surrogate for human perceptual similarity between images (Rubner et al. 2000). However, training diffusion model using this metric between images is expensive and grows as ( log N) with the image size. Surprisingly, by utilizing multi-scale basis, our upper bound can be evaluated in O(N) time, which costs the same standard L norm images. This constitutes a dramatic speed-up that greatly increases the feasibility of training with EMD. We theorize that this special property underpins the empirical success of our model in density estimation and downstream tasks.
alternatively Wasserstein-1
commonly used metric in
Our contributions can be summarized as follows:
We demonstrate the existence of a class of transformations we call hierarchical volume- preserving maps which the likelihood function is invariant. Leveraging this class of mappings, we construct multi-scale cascaded diffusion model that recovers the likeli- hood training characteristies of standard diffusion models without requiring an expensive marginalization step.
We consider that two well-known multiscale decompositions, the Laplacian pyramid and wavelet transforms, and show that they are hierarchical volume preserving maps, where the linearity of the maps reduce the polume-preserviin
property into more familiar notions of orthogonality and tight frames (Unser et al. 2011). We establish the capabilities of our model with significant improvements likelihood modeling, lossless compression, and out-of-distributien detection performance selection of standard image-based density estimation benchmark datasets.
Investigating the theoretical basis for this performance gain, we uncover deep theoretical connection between EMD score matching and cascaded diffusion models trained via two special classes of hierarchical volume- ooemee-reserrvng maps, the wavelet and Laplacian pyramid maps.
2

---

Published as a conference paper at ICLR 2024
2 RELATED WORK
Likelihood Training The maximum likelihood training of generative models involves designing model with tractable likelihood po(x) i.e., the probability that model with parameters generates an image x - and maximizing this function with respect to 0. This practice can be traced back to the wake-sleep algorithm (Hinton al., 1995), which is used select parameters for deep belief networks and Helmholtz machines (Dayan et al., 1995). More modern treatments include variational autoencoders (VAEs) (Kingma & Welling 2013), normalizing flows (NFs) (Dinh) it al. 2014), and autoregressive models (ARMs) (Van Den Oord et al. 2016 Salimans et al. 2017). Diffusion models also fit into this category, as they were originally formulated in variational Bayesian framework (Sohl-Dickstein et al.) (015). However, they are often trained with a non- probabilisticc loss function (Ho et al. 2020; Song et al., 2020b) due to the instability of the originally derived loss. Recently, (Song et al. 2021 explore likelihood training in frequentist setting, whereas (Chen et al. (202]) do so via the lens of the Schrodinger Bridge problem and forward-Backward SDEs, while (Kingma et al.) (2021) introduce several modifications to the originally derived probabilistic loss function that greatly improve its stability during training and inference. But there is to our knowledge no known likelihood-basse treatment of diffusion models in multi-scale setting, to which we turn our attention.
Multiscale Generative Models Hierarchical modeling of spatially structured data with progres- sively growing resolution scales has seen widespread use across image (Karras et al. 2017 Child 2020 |Han et al.) 2022) and audio (Oord al., 2016) domains. hierarchical structure encourages the data synthesis task to be split into smaller sequential steps, allowing models to learn spatial correlations each resolution scale separately, and counteracts the tendency to overly focus on local structure (De Fauw et al.) 2019). In normalizing flows, (Yu et al. (0200) construct multiscale flow by leveraging the orthogonality of the wavelet transform, which can be seen as a volume- preservation condition on linear equidimensional transforms our Eq. 8 is generalization of this observation. In diffusion models, (Ho et al.] 2022); (Guth et al., [202]] Wang et al. 2022) formulate non non-probabillittic hierarchical framework for diffusion models and demonstrate its effectiveness in generating high resolution images at impressive levels of perceptual fidelity. In spite of these advances, likelihood training of multi-scale models is generally unwieldy, requiring expensive marginalization, loose vari- ational bounding, or complex factorization schemes where pixels in low-resolution images must be directly utilized as pixels in the higher resolution images (Reed al. 2017: Menick & Kalchbrenner 2018).
Diffusion Modeling with Earth Mover's Distances We also show that likelihood training with our hierarchical model approximates score matching with an optimal transport cost. Existing works have also noted relationships between diffusion modeling and optimal transport (De Bortoli et al., 202T Chen et al.) 2021 Lipman et al.) [022] Shi et al. [022 Kwon et al. 2022). showing that diffusion models learn to generate data by transporting samples x E R along paths that minimize a cost between the prior p(xx)= ~ .N(0, I) and data distribution p(xo). Our result is markedly different: we show that our model learns the solution to an optimal transport problem in the wwo-dimensional coordinate space of pixel histograms. That is, we consider the pixel values of each image as an unnormalized and discretized histogram on a square in I2, and optimize a transport cost over this grid. This metric on images (as well as other spatial data) is well-studied, and known to correlate well with perceptual similarity (Rubner et al. 2000; Zhang et al., 2020; Zhao et al. 2019). Our perpespective is thus orthogonal and complementary to prior work.
BACKGROUND
Diffusion models (Ho et al. 2020 Song et al.) 20206) are inspired by non-equilibrium thermodynam-
ics (Sohl-Dickstein et al. 2015) designed to model (x( = J pa (x())) dx where data X :=
are related a set of latent varialess x1.7 by diffusion process. Namely, X1:7 == (x(t1).... ..... x(tT) ))
are distributed as marginals of a process governed by an Itô stochastic differential equation (SDE)
dx f(x, 1) di g())
evaluated at times HT and are typically called drift and diffusion functions, and w is the standard Wiener process. Samples can then be generated by modeling the reverse diffusion, which
3

---

Published as a conference paper at ICLR 2024
has simple form given by (Anderson 1982
dx = f(x, t) g(t)² Vx log g(x, f|xo)] dt g(t) dw.
(2)
(x.
where is the corresponding reverse-time Wiener process. Training the diffusion model involves
approximating the true score function Vx log q / xb) with a neural network Sy (x. t) in Eq. (2)
This can be achieved directly via score matching (Hyvärinen & Dayan 2005; Song & Ermon 2019
Song et al. 2020b), , or by modeling the sampling process (Sohl-Dickstein et al. 2015 | Ho et al. 2020
Kingma al. (02T), which is obtained by approximating the marginal distributions of the discretized
reverse-time Markov chain with parametric model pg. i.c.,
T-1 II k=1
((xir//x0) = ((x))
xxx X
(3) and (xo:T) = p(x))
II
po (x(xx(x441) (4)
k=0
Maximum likelihood estimation then be performed by optimizing tractable likelihood bound log peppo(x) (Xo) 2 [(x)
Exprngg
log (x)lxx)) K XI(g(xr/<<)lpo(xx)))
log K (L(((xx (x411,
(x++1)))))
(5)
which forms the basis for variational inference with diffusion models. Several further improvements were proposed by (Kingma et al. (202I). such as antithetic time sampling, continuous time relaxation of the Markov chain, and learnable noise schedule parameterized by monotonic neural network.
Finally, unbiased estimates of log (x can be obtained by modeling the probability flow ODE (PF-ODE) corresponding to Eq. (2) from (Song et al., 20206)
1
dx f(x,t) t) - g() Vx log t) dt,
(6)
using the continuous change of variables formula (Chnn et al. 2018) and substituting the score with the score network sp(x, t).
To build a cascaded diffusion model, x is further decomposed into a series of latent representations {z(*)] of decreasing scales, and ps (2(x)) is approximated. Now, the data may sampled autoregressively - drawing first from low-resolution prior (2) ~ (z()), then from a series of conditional super-resolution models (*)) ~ (2()) for = 2...., S and where 2 = x. However, it is in this hierarchical setting that
into the aforementioned intractability of the
likelihood function, where (5) x) can only be recovered by marginalizing the modeled likelihood over each latent scale, i.e.,
(x(x)= = -f
(1 ((2) .....,(()))8.).),) 12 (8-1)
4 HIERARCHICAL VOLUME-PRESERVII MAPS
Due to the intractability of the likelihood function in many hierarchical models, we are interested in multi-scale decompositions of x that can sidestep the need for marginalization during likelihood computation. In this section, we shall demonstrate the existence of a subset homeomorphisms H we call hierarchical volume-preserving maps to which the likelihood function Pe displays a form of probabilistic invariance - i.e., (>(x) = po (h(x)) for all H. The utility such transformations clear: if h(x) is the scales of the hierarchical model, then we can directly use the joint likelihood over these scales as the desired likelihood function. Finally, we consider two simple homeomorphisms that satisfy proposed condition: the Laplacian pyramid and wavelet transforms.
4

---

Published as a conference paper at ICLR 2024
4.1 A PROBABILISTIC INVARIANCE
To build towards this result, we review the notion of volume--reservvng maps, which can be under- stood transformations that are constrained with respect to a local measure of space expansion or contraction, the matrix volume.
Definition
Maps (Berger & Gostiaux 2012)). A homeomorphic (i.e.. smooth,
invertible) transformation h X 2 is known be volume-preservvng it everwhere satisfies
V (ATA)=
(8)
where det(-) is the determinant and A E Rddim 2) 2)xdim( is the Jacobian of h(x).
Crucially, dim(2) may be larger than dim(X) - h may be a manifold-valued function, so long as A is full rank (i.e. rank(A) 2 ()). Our notion of hierarchical volume-preserving maps further requires that the transformation
multi-scale (Mallat 1999; Burt & Adelson 1987 Horstemeyer
2010), i.e., h X > g x x... x ( where the original data x E X' is decomposed into sequence of representations (*)) E g(s) that describe the data at varying scales - an essential property for cascaded modeling. The following result reveals the utility of working with such transformations.
Lemma 4.1 (Probabilistic Invariance of Hierarchical Volume-preserving Maps). Let h be hierarchi- cal volume--reseervng map such that h(x) = (z(",, ,z!²)..... ...,z(())) (S)),, and pa be likelihood function on z( (2)) .!...!!)), Then the likelihood function finnctionn with respect to the original data pe (x) can be recovered by the simple relation
(og))( log po (xx))))
In other words, the joint likelihood given by the cascaded model under hierarchical volume- preserving map is precisely the desired likelihood function with respect to the data (x),, and we no longer require any additional computation or design considerations to remove the intermediary scales. In this sense, Eq. displays a probabilistic invariance to
Standard Cascaded Hierarchy We now demonstrate how the standard hierarchy of a cascaded
model such as in (Karras et al. 2017) or (Ho et al. 2022) is not hierarchical volume-prees preserving
map. Here, the scales 2 (2) (S) are subsampled versions of the full-resolution sample x, with each scale twice the resolution of the preceding scale. Consider simple illustrative example in
Rd When using the nearest neighbor resampling method, / may be defined as the concatenation of
S different identity-like functions
[Lxxd, 2
di xdy
2
(3) d2xd2
2 (S-11
(9) I
1xds ,
(10)
where dę == d/2 and Ixxd denotes a d x d identity matrix where each column is repeated & times. Inspecting this operator we find that its determinant grows exponentially with d. Therefore, while hierarchical, the standard transformation is certainly not volume-preservee Accordingly, as has been established, there is no tractable form for the likelihood function pe in terms of the joint likelihood pe (z(1), ((())
4.2 SPECIAL INSTANCES OF HIERARCHICAL VOLUMME- MAPS
While our formulation is fully general for arbitrary homeomorphisms that satisfy Eq. highlight two special and well-known classes of maps that are particularly noteworthy for their ease of implementation, computational tractability, and theoretical properties (Section 5.2
Laplacian Pyramids Though the standard cascaded hierarchy is not hierarchical volume- preserving map, it is closely related to the Laplacian pyramid. This representation involves a set of band-pass images, each spaced an octave apart, obtained by taking differences between se- quence of low-pass filtered images with their unfiltered counterparts. Formally, consider base images
Forrallyy, letting (h) po)h(x)), the set of hierarchical volume-ppesennve maps 7t form an equivalence class under the relation 9x (h) = 9x (h') for h, R e H and x E X.
5

---

Published as a conference paper at ICLR 2024
of size j and let d: Rixi R(/2)x((/2)) and Rixi +(--2)x((22) denote downsampling and upsampling operators respectively. this work, we choose them to be norm corm-preserving bilinear resamplers (i.e., images are scaled by factor of 2 every downsample and factor of 1/2 every upsample). Then (8) can be written
= u(d(xx))
(11)
and the intermediary multi-scale representations (2)) recursive relation
(2))) ...., ,2.,, (S-1) may be constructed via the
y() dlyy and for 8 2..... and y(5) = x. Finally, we define the base scale containing the low-frequency residual as (2)) = y d(y(2)).
2 y()) u(d(yl=)))
(12)
We now verify that the Laplacian pyramid is hierar- chical belume-preserving map. Clearly, the Laplacian pyramid is multi-scale given proper choice of and u. our case, we simply choose the bilinear resam- pler. Moreover, the Laplacian pyramid is invertible to reconstruct from fz {2()) 1==1- we simply invert the
recursive relation, letting (1) ()), and computing
Figure 2: A Laplacian pyramid hierarchy with
y(++1) u(y z (*+1)
(13) 5= 4. Left to right: 2(1)
(4))
for s l........ where x = ( Finally, to see that the Laplacian pyramid volume- preneriine.
we observe that it forms a tight frame (Unser et al.) 2011) and therefore satisfies a generalization of
Parseval's Equality, i.e. ||/(x) || = ||x Since this holds for all x, we can conclude that the largest and smallest singular values of h are both 1.
Wavelet Decomposition A more efficient hierar- chical encoding of x can be obtained by enforcing the orthogonality h. While the Laplacian pyramid rep- resentation specifies mapping into latent space of higher dimension Rd - 2d the wavelet transform orthogonal (and thus Rd + R.. There many types of wavelet transforms (Rioul & Vetterli 1991 Graps] 1995), each constructed from scaled and trans-
Figure 3: A wavelet hierarchy with S 4. Left to right: 211) ((),
lated versions of respective (mother) wavelet. We
focus on the Haar wavelet (Stanković & Falkowski
2003) as it is widely used and permits a simple implementation. In this case, the wavelet transform W
can be further described in terms of outer products of the matrices L 2/2 [1, 1 1] and H /2 1-1,,, 1].
There four kernels total, HT. HLT LHT, and LLT each of size 2 2. Letting be the convolution operator with stride 2, the scales of the wavelet hierarchy can be written as
(() HH (*) HLT (()))
LH (()))
LLT xx(*)
and y(() ZLL
(a+1)
(14)
RH
LL
PLC
for 8 2,..... S. where y(S) (e) (8) HHH HHI: Li be seen the high frequency subbands of representing horizontal, vertical, and diagonal edge information, while zit are the low frequency residuals. Like with Laplacian pyramids, (1) := -(()) is taken to be the coarsest scale. Finally, due to its orthonormality, it is
(S+1)
= X. The 5 S.....St scale representations (())
ZLE
clear that this mapping satisfies the preservin property of Eq. and thus hierarchical volume-preserving map.
5 LIKELIHOOD TRAINING OF CASCADED DIFFUSION MODELS
In this section, we construct the probabilistic cascaded diffusion model (PCDM), which is a multi- scale hierarchical model capable of likelihood training and inference. We then investigate the theoretical properties of our model, and uncover rigorous connection between likelihood training and score matching with respect to optimal transport cost.

---

Published as a conference paper at ICLR 2024
5.1 COUPLED DIFFUSION MODELING ON h-INDUCED LATENT SPACE
Letting h : X + g(4) x ...... x (9) be a hierarchical volume-preserving map (where x E X' and introduced in Section 3 which is defined by way of standard unconditional diffusion model po (z and a sequence of conditional super-resolution diffusion models po(z () (+())) This produces S coupled diffusion processes, where each scale (*)) is governed by its own Itō SDE with dynamics that depend on the processes that precede it.
2(0) E ) we return to the cascaded diffusion model on h(x) = (z(") (x(1) , 2
By Lemma 4.1 we see that the desired likelihood function px(x) can be recovered by the simple relation
S
log po (x) = log po (z
+
Σ Po (2(2)))22(5
(15)
Likelihood training then proceeds by lower bounding each likelihood with its corresponding loss function ((z( as in Eq. for the unconditional base model, and
log po (3) 12)") -0 ((00)) - LL(g(z)-) (3) 1z C(z [(z(=l/2(5)) = Exp
2 (l200
KL(g(z) 20 2 ((0)) 1 l2 k 2. (())))
(16)
for the conditional super-resolution models. Combining all terms, the full training loss of our cascaded diffusion model may be written as
8 C(x) ((z(1)) Σ C(z
(17)
x=2
over all x in the dataset.
5.2
CONNECTION TO OPTIMAL TRANSPORT
A notable property of distances in the latent spaces induced by maps from Section is their connection to an optimal transport cost. We begin by introducing the Wasserstein distance on the image domain.
4.2
Definition 2 (Wasserstein p-Metric). Let X E HxW be images, which we interpret unnormal- ized densities on the product space of pixel indices {h] x {whm-n The Wasserstein p-metric can be written as
1/p
W (x, = inf Σ vli, k, - ykell? k.j.ke
(18)
where Xyy (or (or is the i, j.tth pixel of x (or y) and v(i. kc, E) is a scalar-valued function such that Eng p(i,j, k, E) Xxt and kx, vii,j, k,, Xij.
Our next statement builds on prior theoretical work in linear-time approximations to the Earth Mover's Distance (Shirdhonkar & Jacobs 2008).
Theorem 5.1 (Cascaded Diffusion Modeling and EMD Score Matching). Let h be one of the hierarchical colume-preservvee maps defined Section 4.2 and po be a cascaded diffusion model on
h(x) = (z()) z z 18)). Then there exists constant a depending only on the map h such that
aC(x) wy bxo) [W,(Vx log q(āx, xo), So (xx,, tx))]
(19)
where Wy is the likelihood weighting of the variational lower bound, ((Xx (x6) is the marginal of the forward diffusion process, and O p S
7

---

Published as a conference paper at ICLR 2024
Model
CIFARIO 32x32
ImageNet 32x32
ImageNet 64x64
ImageNet 128x128
VAE with IAF (Kingma ct al. 2016
3.11
PixeICNN Van Den Oord et al. 2016
3,03
3.83
3.57
PixelCNN+ Salimans
2017
2.92
Glow (Kingmä & Dhariwll 2018
3.35
4.09
3.81
Image Transformer [Parmar et al.
2018
2.90
3.77
SPN (Menick Kalchbrenner 201
3.52
CR-NVAE (Smha Dieng 2021
2.51
+low++ (Ho ct al. 2019a)
3,06
3.86
3.69
Sparse Transtormer (Child al.,
2019
2.80
3.44
Sparse Transformer w/ dist, aug.
(Jun al. 2020
2.53
3.44
Very Deep VAE (Child 2020
2.87
3.80
3.52
DDPM (Ho et al. 2020
3.69
Score SDE (Song et al. 2020b
2.99
Improved DDPM (Dhanwal & Nichol
2021
2.94
3.54
EBM-DRL (Gao et al.
2020
3.18
Routing Transtormer Roy
2021
3.43
S4 (Gu ct 2001
2.85
Score Flow (Song al. 2021
2.80
3.76
LSGM /Vahdat al. 2021
2.87
Flow Matching (Lppman et al.
2022
2.99
3.53
3.31
2.90
Soft Truncation Kim ct al. 2022b
2.99
3.53
INDM Kim et al. 2022a
2.97
VDM (Kingma et al. 2021
2.49
3.72
3.40
i-DODE (hengg al. 2023
2.56
3.69
Our work
LP-PCDM
2.36
3.52
3.12
2.91
W-PCDM
2.35
3.32
2.95
2.64
Table 1: Comparison between our proposed model and other competitive models in the literature in terms of expected negative log likelihood on the test set computed as bits per dimension (BPD). Results from existing models are taken from the literature.
Theorem 5.1 reveals that optimizing Eq. 17 can also be interpreted as solving an optimal transport problem between the true and approximated scores Vx log ((xx/xx)) and sy (X,,, fx) with respect to the Wasserstein p-metric, for E [o, 1]. Of note the = 1 case, which is also regarded as the Earth Mover's Distance (EMD), well-known measure in computer vision understood to correlate well with human perceptual similarity (Rubner et al. 2000). By giving up exact computation of the optimal transport cost,
able optimize O nn log n. measure in linear time, bringing the EMD measure into the realm of feasible criteria to train diffusion models with in high dimensions.
6 EXPERIMENTS
We evaluate both the the Laplacian pyramid-based and wavelet-based variants of our proposed probabilistic cascading diffusion model (LP-PCDM and W-PCDM, respectively) in several settings. First, we begin on general density estimation task on the CIFARIO (Krizhevsky et al., 2009) and ImageNet 32, 64, and 128 /Van Den Oord al. (0166) datasets. Then we investigate the anomaly detection performance of our models with an out-of--istributt differentiate between in indistribution (CIFAR1O) and distt---istii (SVHN (Netzer al.) 2011), uniform, and constant uniform) data. Finally. we evaluate our model on lossless compression benchmark against several competitive baselines in neural compression literature.
baseline where the model is tasked
6.1 LIKELIHOOD AND SAMPLE QUALITY
We adapt the architecture and training procedure in (Kingma et al. (222)) to our framework (Appendix B). and compare our model to existing likelihood based generative models terms of bits per dimension (BPD) standard image-based density estimation datasets: CIFAR16 10, ImageNet 32x32,,
8

---

Published as a conference paper at ICLR 2024
GLOW
PixeICNN+
EBM
AR-CSM
Typ. Test
VDM
LP-PCDM
W-PCDM
SVHN 0.24
0.32
0.63
0.68
0.46
0.53
0.71
0.74
Const. Uniform
0.0
0.0
0.30
0.57
1.0
0,96
1.0
1.0
Uniform
1.0
1.0
1.0
0.95
1.0
1.0
1.0
1.0
Average 0.41
0.44
0.64
0.73
0.81
0.83
0.90
0.92
Table 2: out-of-distribution (OOD) detection performance of a CIFARIO trained model on a selection of anomalous distributions in terms of AUROC. Results from existing models are taken from the literature.
and ImageNet64464. We additionally evaluate our model on ImageNet 128x1 and compare against a higher resolution baseline (Lipman et al., (022). Table shows that we obtain state-of-the- results across all datasets, demonstrating the advantages of density estimation with a hierarchical prior. We then evaluate the sample quality of generated images the CIFAR1O dataset. Using the likelihood weighting scheme in (Kingma et al. (02T), we obtain an FID 6.31 with LP-PCDM and 6.23 with W-PCDM, which improves on the FID obtained by (Kingma al. 2021) (7.41). Switching to the architecture and weighting scheme specified in (Karras et al. 2022) optimized for sample quality, we obtain competitive FII) of 2.45 and 2.42 for LP-PCDM and W-PCDM, at the cost of increased BPD (9.55 and 10.31, respectively).
6.2 LUT-OR-DISTRIIUTION DETECTION
Finally, we evaluate the out-of-distribution (OOD) detection capabilities of our proposed model. In this problem, probabilistic model is given set of in-distribution (i.e., training) and out-of- distribution (i.c., outlier) points, and tasked with discriminating between the two distributions in
an unsupervised manner. Likelihood-baate
models surprisingly brittle under this benchmark
(Hendrycks et al.) 2018 Nalismick et al. 2018:) Choi et al.] 2018 Shataci et al.) (018)), even though
likelihoods Po permit intuitive interpretation for inlier/outieer classification. We
OOD)
statistic the typicality test ha(x) = 1 Σ log po (x.. (x))) proposed in (Nalisnick
et al. 2019), where log (Xm) is approximated by our likelihood bound C(zz) from Eq. (1)). We
compute our statistic under the most difficult scenario of M = 1. Following (Du & Mordatch 2019
Meng et al. 2020),
evaluate
models against the Street View House Numbers (SVHN) (Netzer
et al. 2011 1, constant uniform, and constant distributions as OOD distributions. Table that obtain significant gains in OOD performance. More experimental details are in Appendix B.1
6.3 LOSSLESS PROGRESSIVE CODING
As discussed in (Ho al. 2020 Kingma et al., 2021; Hoogeboom et 2022)) probabilistic diffusion model can also be seen as a latent variable model in neural network- based lossless compression scheme. We implement neural compressor us- ing our proposed model as the latent variable model in the Bits-Back ANS algorithm (Townsend et al.) 2019), and compare against several promi- nent models in the field, demonstrat- ing significant reductions in compres- sion size on the CIFAR10 1 dataset. We report performance in terms of average bits per dimension of the codelength over the entire dataset in Table 3 One downside of a bits back based implementation is that there is significant overhead of bits required to initialize the algorithm, and encoding and decoding must be performed in fixed sequence decided at encoding- mncdding-time, meaning that single image compression is expensive. This limitation can be removed by implementing a direct compressor (Hoogeboom et al. 202I). which can also be applied to model. We leave this avenue of research for further work.
Model
Compression BPD
FLIF
Sneyers & Wuille
2016
4.14
IDF
Hoogeboom et al. 2019
3.26
LBB (Ho et 2019b)
3.12
VDM Kingma et al. 2021
2.72
ARDM (Hoggeboom et al.
2021)
2.71
I.P-PCDM
2.40
W-PCDM
2.37
Table 3: Lossless compression performance on CIFARIO in
terms of bits per dimension (BPD). Results from existing models are taken from the literature.

---

Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Published as a conference paper at ICLR 2024
7 CONCLUSION AND FUTURE DIRECTIONS
We demonstrated the existence of a class of transformations we call hierarchical colume-preserving maps that sidesteps common difficulty in multi-scale likelihoood modeling, the intractability of the likelihood function po(x). Hierarchical volume-preserving maps are homeomorphisms to which the likelihood function are invariant, allowing the likelihood function pe px(x) to be directly computed as joint likelihood on the scales. We then presented two particular instances of these transformations that warrant attention due to their simplicity and performance on empirical benchmarks. Finally, we demonstrate connection between our training and optimal transport. An open question is whether score matching under an EMD norm enjoys the same statistical guarantees as the standard score matching, c.g.. consistency, efficiency. and asymptotic normality (Hyvärinen) (006) Song ctt al. 2020a). We hope that this work paves the way for future inroads between hierarchical and likelinood-based based modeling, and that theoretical framework opens the door for the further diversity and improvements in the design of diffusion models.
8 ACKNOWLEDGEMENTS
YK acknowledges support for the research of this work from NIH [R01GM131642, UMIPA051410, U54AG076043, U54AG079759, P50CA 121974, U01DA053628 and R33DA047037J. RB's research at the Weizmann Institute was supported partially by the Israel Science Foundation, grant No. 1639/19, by the Israeli Council for Higher Education (CHE) via the Weizmann Data Science Research Center, by the MBZUAI-WIS Joint Program for Artificial Intelligence Research, and by research grants from the Estates of Tully and Michele Plesser and the Anita James Rosen and Harry Schutzman Foundations.
REFERENCES
Applications, 12(3):313-32,, 1982.
Adi Ben-Israel. The change-of-variables formula using matrix volume. SIAM Journal on Matrix Analysis and Applications, 21(1):300-312, 1999.
Marcel Berger and Bernard Gostiaux. Differential Geometry: Manifolds, Curves, and Surfaces, volume 115. Springer Science & Business Media, 2012.
Peter Burt and Edward H Adelson. The laplacian pyramid as compact image code. In Readings in computer vision, pp. 671-679 Elsevier, 1987.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Tianrong Chen, Guan-Horng Liu, and Evangelos A Theodorou. Likelihood training of schr)," odinger borward-backward backward sdes theory. arXiv preprint arXiv:2110.11291, 2021.
bridge using forward-
Rewon Child. Very deep vaes generalize autoregressive models and can outperform them on images. arXiv preprint arXiv:2011..006 2020.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv: 10509, 2019.
Hyunsun Choi, Eric Jang, and Alexander A Alemi. Waic, but why? generative ensembles for robust anomaly detection. arXiv preprint 1xxiv 2018.
Peter Dayan, Geoffrey E Hinton, Radford M Neal, and Richard S Zemel. The helmholtz machine. Neural computation, 7(5):889-904, 1995.
Valentin De Bortoli, James Thornton, Jeremy Heng, and Amaud Doucet. Diffusion schrödinger bridge with applications to score-based generative modeling. Advances Neural Information Processing Systems, 34: 1495599997,779 2021.
10

---

Published as a conference paper at ICLR 2024
Jeffrey De Fauw, Sander Dieleman, and Karen Simonyan. Hierarchical autoregressive image models with auxiliary decoders. arXiv preprint arXiv:1903.0490 04933, 2019.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components
estimation. arXiv preprint arXiv:1410.8514 2014.
Yilun Du and Igor Mordatch. Implicit generation and generalization in energy-based models. arXiv preprint arXiv:/903. 08689, 2019.
Ruiqi Gao. Yang Song, Ben Poole, Ying Nian Wu, and Diederik P Kingma. Learning energy-based
models by diffusion recovery likelihood. arXiv preprint urXiv:2012.08/25, 2020.
Amara Graps. An introduction to wavelets. IEEE computational science and engineering, 2(2): 50-61, 1995.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00330 2021.
Florentin Guth, Simon Coste, Valentin De Bortoli, and Stephane Mallat. Wavelet score-based generative modeling. arXiv preprint arXiv:2208.0500 2022.
Sangjun Han, Taeil Hur, and Youngmi Hur. Laplacian pyramid-like autoencoder. In Intelligent Computing: Proceedings of the 2022 Computing Conference, Volume 2, pp. 59-78. Springer, 2022.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. arXiv preprint 1eXiv:181.004600 2018.
Geoffrey E Hinton, Peter Dayan, Brendan Frey, and Radford M Neal. The" wake-sleep" algorithm for unsupervised neural networks. Science, (68((5214): 1):14)):1111111 158-1161, 1995.
Jonathan Ho. Xi Chen, Aravind Srinivas, Yan Duan, and Picter Abbeel. Flow++: Improving flow- based generative models with variational dequantization and architecture design. In International Conference on Machine Learning, pp. 2722-2730. PMLR, 2019a.
Jonathan Ho, Evan Lohn, and Pieter Abbeel. Compression with flows via local bits-back coding. Advances in Neural Information Processing Systems, 32, 2019b.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6885 6851, 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans.
Cascaded diffusion models for high fidelity image generation. J. Mach. Learn. Res., 23(47):1-33 2022.
Emiel Hoogeboom, Jomn Peters, Rianne Van Den Berg, and Max Welling. Integer discrete flows and lossless compression. Advances in Neural Information Processing Systems, 32, 2019.
Emiel Hoogeboom, Alexey A Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. Autoregressive diffusion models. arXiv preprint 11X0v:21 2021.
Mark F Horstemeyer. Multiscale modeling: a review. Practical aspects of computational chemistry: methods, concepts and applications, pp. 87-135, 2010.
Aapo Hyvärinen. Consistency of pseudolikelihood estimation of fully visible boltzmann machines.
Neural Computation, 18(10):2283-222) 2292, 2006.
Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
Heewoo Jun, Rewon Child, Mark Chen, John Schulman, Aditya Ramesh, Alec Radford, and Ilya Sutskever. Distribution augmentation for generative modeling. In International Conference on Machine Learning, pp. 5006-5019. PMLR, 2020.

---

Published as a conference paper at ICLR 2024
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196, 2017.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. arXiv preprint arXiv:2206.00364. 2022.
Dongjun Kim, Byeonghu Na, Se Jung Kwon, Dongsoo Lee, Wanmo Kang, and II-chul Moon. Maxi- mum likelihood training of implicit nonlinear diffusion model. Advances in Neural Information Processing Systems, 35:32270-32288 2022a.
Dongjun Kim, Seungjae Shin, Kyungwoo Song, Wanmo Kang, and Il-Chul Moon. Soft truncation: A universal training technique of score-based diffusion model for high precision score estimation. In International Conference Machine Learning, pp. 1 11201-11 11228. PMLR, 2022b.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural information processing systems, 34:21696-2111 2021.
Diederik P Kingma and Max Welling. arXiv:1312.6114, 2013.
Auto-encoding variational bayes.
arXiv preprint
Durk P Kingma and Prafulla Dhariwal. Glow:
Generative flow with invertible 1xl convolutions. Advances in neural information processing systems, 31, 2018.
Durk Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. Advances in neural information processing systems, 29, 2016.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. Technical Report, 2009.
Dohyun Kwon, Ying Fan, and Kangwook Lee. Score- generative modeling secretly minimizes the wasserstein distance. arXiv preprint arXiv:22/2.06335 2022.
Yaron Lipman, Ricky TQ Chen, Heli Gem-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
Stéphane Mallat. A wavelet tour of signal processing. Elsevier, 1999.
Chenlin Meng. Lantao Yu, Yang Song, Jiaming Song, and Stefano Ermon. Autoregressive score matching. Advances in Neural Information Processing Systems, 33:6673-6683, 2020.
Jacob Menick and Nal Kalchbrenner. Generating high fidelity images with subscale pixel networks and multidimensional upscaling. arXiv preprint 1rxi:/8//2.01008, 2018.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't 't know? arXiv preprint arXiv:1810.09136. 2018.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, and Balaji Lakshminarayanan. Detecting out-of- distribution inputs to deep generative models using typicality. arXiv preprint 2019.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03449 2016.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran, Image transformer. In International conference on machine learning, pp. 4055-4064. PMLR, 2018.
12

---

Published as a conference paper at ICLR 2024
Scott Reed, Aäron Oord, Nal Kalchbrenner, Sergio Gómez Colmenarejo, Ziyu Wang. Yutian Chen, Dan Belov, and Nando Freitas. Parallel multiscale autoregressive density estimation. In Interna- tional Conference on Machine Learning, pp.
PMLR, 2017.
Olivier Rioul and Martin Vetterli. Wavelets and signal processing. IEEE signal processing magazine, 144): 1991.
Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with routing transformers. Transactions of the Association for Computational Linguistics, 9:53-68, 2021.
Yossi Rubner, Carlo Tomasi, and Leonidas J Guibas. The earth mover's distance metric for image retrieval. International journal of computer vision, 40(2):99, 2000.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelen+++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv:1701.05517, 2017.
Alireza Shafaei, Mark Schmidt, and James J Little. Does your model know the digit 6 is not cat? less biased evaluation of" outlier" detectors. arXiv preprint arXiv:/809.04 1809.0472, 2018.
Yuyang Shi, Valentin De Bortoli, George Deligiannidis, and Arnaud Doucet. Conditional simulation using diffusion schrödinger bridges. In Uncertainty in Artificial Intelligence, pp. 1792-1802. PMLR, 2022.
Sameer Shirdhonkar and David W Jacobs. Approximate earth mover's distance in linear time. In 2008 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-8. IEEE, 2008
Samarth Sinha and Adji Bousso Dieng. Consistency regularization for variational auto-encoders. Advances in Neural Information Processing Systems, 34: 1294-, 2021.
Jon Sneyers and Pieter Wuille. Flif: Free lossless image format based on maniae compression. In 2016 IEEE international conference on image processing (ICIP), pp. 66-70. IEEE, 2016.
Jascha Sohl-Dickstein. Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. International Conference on Machine Learning, pp. 1256-2265. PMLR, 2015.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
Yang Song. Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. Uncertainty Artificial Intelligence, pp. 574-584. PMLR, 2020a.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011. 13456,, 2020b.
Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances in Neural Information Processing Systems, 14:1415-14288 2021.
Radomir Stanković and Bogdan J Falkowski. The haar wavelet transform: its status and achieve- ments. Computers & Electrical Engineering, 29(1):25 44, 2003.
James Townsend, Tom Bird, and David Barber. Practical lossless compression with latent variables using bits back coding, arXiv preprint arXiv:1901.044 04866, 2019.
Michael Unser, Nicolas Chenouard, and Dimitri Van De Ville. Steerable pyramids and tight wavelet frames in 12(r). IEEE Transactions on Image Processing, 20(10)::705-2721, 2011.
Arash Vahdat, Karsten Kreis, and Jan Kautz. basee generative modeling in latent space. Advances in Neural Information Processing Systems, 34:1 1247-1287-1111 1302, 2021.
13

---

Published as a conference paper at ICLR 2024
Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks.
In Interaational conference on machine learning, pp. 1747-1756. PMLR, 2016.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computa- tion, 1367):1611-1674, 2011.
Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. arXiv preprint (.XXv:2222.0049900 2022.
Jason J Yu, Konstantinos G Derpanis, and Mareus A Brubaker. Wavelet flow: Fast training of high resolution normalizing flows. Advances in Neural Information Processing Systems, 33:6184-6196, 2020.
Chi Zhang, Yujun Cai, Guosheng Lin, and Chunhua Shen. Deepemd: Few-shot image classifica- tion with differentiable earth mover's s distance and structured classifiers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12203-12213, 2020.
Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M Meyer, and Steffen Eger. Moverscore: Text generation evaluating with contextualized embeddings and earth mover distance. arXiv preprint arXiv: 02622, 2019.
Kaiwen Zheng. Cheng Lu, Jianfei Chen, and Jun Zhu. Improved techniques for maximum likelihood estimation for diffusion odes. arXiv preprint arXiv:2305.03933 2023.
14

---

Published as a conference paper at ICLR 2024
A PROOFS
Lemma 4.1 (Probabilistic Invariance of Hicrarchical
Maps). Let be a hierarchi- cal velume-preserving map such that h(x) = (z(2), (02))22 " ....,,,))),, and po be likelihood function on 211), 22(2, :....: $8), Then the likelihood function with respect to the original data po po(x) can be recovered by the simple relation
Pe(p)( log
Proof. We begin with the generalized change-of-van -arriables formula on Riemannian manifolds (Ben-
Israel 1 which relates the density function on x with another on y = f(x) under the (potentially manifold-valued) transformation f via the relation
p(x) = ply) det ((xx(1x)
T
(2x(1x))
o xx
(20)
Considering now the likelihood function pa, letting f = h, and taking logs of both sides, we see that
([ ([o
log pa (x = log po (1) (2) z (5) + log V det
h(xx)
b(xx)
Dix
Ox
= Pespa(' (1) (2)
(5)) + log [1|
(o(()))())).
where the second equality follows due to the fact that h is volume-presse preserving We have thus shown that the likelihood function po (x) coincides with the joint likelihood po (z(2()) (2) ....... 2 (5)). This concludes the proof.
Theorem 5.1 (Cascaded Diffusion Modeling and EMD Score Matching). Let h be one of the
hierarchical colume-preserving maps defined in Section 4.2 and pa be cascaded diffusion model on
h(x) = (z(2), (22) z (9)). Then there exists a constant a depending only on the map h such that
aC(x)=
Σ wy Wy (x.) [Wp(Vx log ((& xo), (0 4x))
(19)
where we is the likelihood weighting of the variational lower bound, g(xx (xo) is the marginal of the forward diffusion process, and O < S.
Proof. According to (Ho et al.
2020). the likelihood bound Eq.
((z(1)) 2 Exprog
log po (z) x1 (1))
KL(g(z)) 120 )l
Lo(a(1))
(x((1()))
log KL (2) (1)
2k+11 2 Iloo
(1) (1)
1x411)
(21)
(x(2(1))
can be decomposed into the three elements: decoder reconstruction term La a prior loss term (x(x), and a weighted sum over the diffusion score matching terms ET-1 Lx (x). The conditional likelihood bound Eq. 16 can be decomposed in similar fashion:
C(z(*) (<a) 2 Exirrng
log pe (z)
(x)
Z (a) (ca))
Zo
KL(g(z) 12 Mlpo (z) I (<x))
L (((x())/2(())
(1(2(0))x()))
T Σ
log K KL(g(2)
(0) (2 14411 2 2 (<0)
a0 (2 (0)
12
(<x)
km
La (x(x))) (a()a))
(22)
15

---

Published as a conference paper at ICLR 2024
for ......
Under the standard Gaussian assumption on the marginals of the diffusion process (i.c., diffusion process is Gaussian process), the unconditional diffusion score matching terms also be written as
/B, (23) is the standard deviation of the marginal of the diffusion process at time t, and wy is the likelihood weighting of the score matching loss. Since the epsilon network can be related to the score network by the relation (x (Bso(x,)), (),
Lx = wo. (=()
+x)]l3), where z VI-
rewrite the above equation
(x(z())) wy wx
(2
(1) tx)ll
VB
W(o.e V a0) log q( 2 so (+x)l
where the second equality due to the connection between denoising and score matching discussed
in (Vincent 2011 Ho et al. 2020 Again, the conditional diffusion score matching loss can be
written similarly:
La (z(z woDDe log (igg(()"" (0 (20))) sn (<.),1))1111
(1)
(24)
for = 2, ...., S.
Simplifying notation, denote the score networks Sy
(1)
5o(25 (1)
tx) and s
so (0 (5) {<<)), 1x). Since the same noise schedule is used across the scales of the cascading model, we may combine all diffusion score matching losses to obtain
C+ : )) +
(2(2))
(25)
Wkk
E e Be-N(
220 200 log qz (0) (000))
8s 11).
(26)
Finally, when h is the wavelet transform, Lemma A.2 provides the existence of a constant a such that
S E E log g(Z (*) 200)) - s/ 1/11
wk E x(xx ( (Vx log q( (8x No); so (xR,, 4x))
(27)
And since Lo and LT are both nonnegative, have our desired result
xo). aC(x) 2 wnglExxsgxxx xx) (V log |xxx/xx), so (8 (x))].
(28)
To obtain this result when the Laplacian pyramid mapping, we note that under our choice of downscaling and upscaling functions d(-) and u(-) (the norm preserving bilinear resampling method),
the auxiliary variables y (Eq. 12) at each scale 8 are also those of the wavelet hierarchy (Bq. [4].
This is because the I.I." convolution kernel with stride is the bilinear downsampling operator. Thus
d(-) = LLT * (-). Now, since the wavelet operator W, d(-) and uu-) are norm preserving,
| (e) ll2 = ||x(6" ())
[[y(") mayyk-")ll
= [|[(x( - u(y(
-Il((() (5) (6)
2 Li (1)) [0,
= |1
Therefore, the norms of the scales coincide at each level for all x, and we can conclude that the Laplacian pyramid diffusion score matching loss
C₁ wk Σ
B.-Noon Veto (z)) (2(2)) s 16.1
16

---

Published as a conference paper at ICLR 2024
coincides with that of the wavelet loss
Finally, we invoke Lemma A.2 obtain the desired result.
To show Lemma A.2 we first restate a key result from (Shirdhonkar & Jacobs 2008) using the notation in this text.
Lemma A.1 (From Theorem 2 in (Shirdhonkar & Jacobs 2008 ). Consider the optimal transport
cost W,, (x.y) (Eq. 2 for E [0, 1] between two unnormalized histograms x and y. Let h(x - y)
(z (2) ..... z (5)
and
A = Coll2 + C1 Σ 2- ((ptn/2)
( z 11
(29)
8=2
where n. is maximum height / width of x and y. ( and {z s==2 are the detail and approximation coefficients of the wavelet transform respectively. Then there exist positive constants Ct, Cy such that
CLA (.ASW,(Xy)) Cuñ.
(30)
Now we are able to make the following statement.
Lemma A.2. Consider the optimal transport cost (,...) (Eq. for p E [0, 1] between the true and modeled score functions Vx log q(xx) xo) and so (Rx, k). Let h be hierarchical volume-preserving map.
<( log (xxx = (V log (2) (2! v Ve log ) Ve(s) log (22 20
(9)
(31)
and
h( k)) (5o(z))) (z)) [(x), S0 (z) (2()) (t)),.)........ so(z(( (<55)
(32)
Then there exists constant 3 > 0 such that
s W, Vx logg((x(xo) so (x(x)) S
Σ
| log 12(")120))) (2)))) - so(x)]12
(33)
Proof. Note that for any scale level < s < S. image size n 2 1, and p E [0, 1], we have that s(p + n/2) > and thus 2 2~s(2=ppp))))) s(tnn/2) < This allows to upper bound the right side inequality in
'While the losses are the same for both models for each the gradients and therefore training trajectories very different due to the different parameterizations of the representation space. This is why we get differing hehaviors between the two hierarchical maps.
17

---

Published as a conference paper at ICLR 2024
Eq. 30 as
Wp Vx log (Xxx so
So( 1x)) S Cu CollVV 2011 log (1,"" [ts)]11
(34)
+ + Σ 2 s(p+n/2)) 2(13 log g(2b) 20 211
(35)
-so(z(0) (<0) te)lla
(36)
s CC CollVar 201 loga(z =0 s0(z (2) |t)]ll
(37)
+ +
2 log (2))))
(38)
(z(20))1((<0)
(39)
CC max (Co, C3)
Σ | nis log (z 1 )
(40)
(0) (c) 4 11))
(41)
Applying the Schwarz bound to the norms and collecting all constants into B, we obtain the desired inequality.
B MODEL AND IMPLEMENTATION
All training is performed on 8x NVIDIA RTX A6000 GPUs. We construct our cascaded diffusion models with antithetic time sampling and learnable noise schedule as in (Kingma et al.) 202T). Our noise prediction networks Es also follow closely to the VDM U-Net implementation in (Kingma et al. 2021), ,, differing only in the need to deal with the representations induced by the hierarchical volume- volume-preserving map h. A hierarchical model with S scales is composed of S separate noise prediction networks. For each input x, our model decomposes x into its latent scales {z(*)} /==1 via
and directs each (=)) to its corresponding noise prediction network (z (2(<)), ,)). The noise prediction network can be related to the approximated score function via
(2(())
t) (t)) (<5))
(42)
where (t) is the variance of the diffusion process at time t. The base noise prediction network is constructed entirely identically to the VDM architecture, as it does not rely on any additional input. The conditional noise prediction networks at each subsequent scale additionally incorporate the conditioning information from previous scales (i.e., ((<e)) via channel-wise concatenation of the conditional information to the input, resulting in the full input vector
[z (+) (<x)
(43)
inputt
(<+)
ododo...oody. oodo...oo
(z( (cs)
0 ))
(44)
(S- times
EZ EZ
2
where J. is the inverse of the hierarchical volume-preserving map, d() is a downsampling operator
(Section& 4.2 and ((Ea) can be seen the reconstructed image containing only the conditioning information available scale s. Thus the noise prediction model each scale co (z(()) (<(0), t) takes
as input imput and outputs z!! ((*)
(+)
In the case where h is the wavelet transform. the scales (<*) can directly be used to construct
an image with the same extent as 2(0), resembling downsampled version of the original input
18

---

Published as a conference paper at ICLR 2024
comprising of all frequency information up to, but not including, the present scale. In this case, the reconstructed image may directly concatenated channel-wise to the s-level wavelet coefficients, producing image with 12 channels.
When h is the Laplacian pyramid map, the scales z <<) instead form an image exactly half the size of the current representation z(0), and therefore must be upscaled. Since the representation at each scale always has 3 dimensions (as oppposed 9 with wavelet coefficients), the final image has channels.
For CIFARIO, we use two scales for both LP-PCDM and W-PCDM, resulting in representations at each scale of
[16 x 16
3, 32
for Laplacian pyramids and
[16 x 16 x 3, 16 16
for the wavelet transform. We U-Net of depth 32, consisting of 32 residual blocks in the forward and reverse directions, respectively. We additionally incorporate a single attention layer and two additional residual blocks the middle. We use convolution dimension 128 in the base model and 64 in the conditional super-resolution model, resulting in 35M and 42M parameter models, respectively. We train with Adam W for 2 million updates.
For ImageNet32, we again use two scales for both LP-PCDM and W-PCDM, resulting in representa- tions at each scale of
16 x 16
3, 32
32
for Laplacian pyramids and
[16 x 16 x 3, 16
16 9)
for the wavelet transform. We again use U-Net of depth 32, consisting of 32 residual blocks in the forward and reverse directions, and incorporate a single attention layer with two additional residual blocks in the middle. We use 256 channels for the base model and 128 channels in the conditional super-resolution, resulting in 65M and 70M parameter models. We train with AdamW for 10 million updates.
For ImageNet64, we use three scales for both LP-PCDM and W-PCDM, resulting in representations each scale of
[16 16
32 32 3, 64 x 64 x
for Laplacian pyramids and
16 x 9, x 32 x 9] 16 16 x 3, 16
for the wavelet transform. We now use U-Net of depth 64, and incorporate a single attention layer with two additional residual blocks in the middle. We use 256 channels for the base model and 128 channels in the conditional super-resolution models, resulting in models with 299M and 329M parameters, respectively. We train with AdamW for 10 million updates.
Finally, for ImageNet128. we use four scales for both LP-PCDM and W-PCDM, resulting in repre- sentations at each scale of
16 x 16 x 3, 32 x 32 3. 64 x 64 x 3, 128
128 3]
for Laplacian pyramids and
[16 x 16 x 3, 16 x 16 x 9. 32 x 32
64 x 64 x
for the wavelet transform. We again use a U-Net of depth 64, and incorporate single attention layer with two additional residual blocks the middle. We use 256 channels for the base model and 128 channels in the conditional super-resolution models, resulting in models with 400M and 440M parameters, respectively. We train with A for million updates.
For out-of-Distribttion (OOD) detection in Section 6.2 we approximate the summation E. La in the likelihood bound with an N = 20 sample Monte Carlo estimate, rather computing the full sum. We find empirically that this is sufficient for OOD detection. Moreover, OOD performance only improves with larger N. Therefore = 20 was chosen as a suitable trade-off between performance and runtime complexity.
19

---

does not need Published as a conference paper at ICLR 2024
B.1 OUT-OF-DISTRIBUTION DETECTION EXPERIMENTAL DETAILS
Nalisnick et al. 2018 Choi et al. 2018 Shafaei et al. 2020), benchmark for evaluating the anomaly detection capabilities of a likelinood model (x) trained on true data x ~ p(x) can be framed as binary classification task between in-distribution Fin ~ p(x) and out-of-distribution Tout ~ v(x) data. Namely, we define data-label pairs (Fim,;; 0);²1 U {zout.j; Nete where N₁n- Nout. (Du & Mordatch| Meng et al. (020)) consider three different "outtof-distribution" densities, the Tour drawn from the Street View House Numbers (SVHN) dataset, the distribution of constant zeros (Const. Uniform), i.e., V = 60 is the Dirac delta at the all-zeros vector, and the uniform distribution, ie., V = Unif([-1,11] The rows of Table show classification performance under the AUROC operator.
As proposed in (Hendrycks et al. 2018
2018: Du & Mordatch 2019 Meng al.
C A VARIATIONAL LOWER BOUND WITHOUT THE VOLUME- PROPERTY
Our proposed hierarchical volume-preee preserving maps retain exact likelihood evaluation capabilities in all cascaded diffusion models with latent scales satisfying h(x) (z(1), 2(2), ...... 2z()), where is a hierarchical allume-preserving map satisfying Eq. 8 On the other hand, when
exact likelihoods - e.g., simply lower bound on the likelihood is sufficient- - an alternative method
for obtaining tractable loss can be obtained by folding the latent variables into the variational bound:
log po
2 Ex [log Po x, 2) log (2(x))
= Ex
[log po (x, 2 2o1
(1)
Z
(9)
log g 1:7"
(1)
21.1
(S)
(1)
20
(8)
x)]
Exi.r
log Pa [ 40
,...,, 20
log q 4
...., 20 )xx
(3)
=0
(+) (20.712)) (2)50)) - log (6) (5o)
log pe
(1 k0 ,x)
Exirng
log po (z/")12²) 2 20 ((())) log pe
(0) (2(2)1a(56)) log
2 12
(e) (50)
xx)
8
Σ
Ex ExIrr~g
loggo (2)) 2/ (())) + KT. (po (2(6)1(600) (20))) 110
(0) 12 ((55)) x))
1=1
+ KL. (po (2/1)_11 126 (20)) 117
2 (+) 12 12650)
(50) x))
8 ExiTng
log (z() (0) (2 (<0))
K
KT(g(244) (2) (0)
,
2 (<)) (1p la! (())))
1=1
KL(g(z))) (2) (2 14441 2
(0) (3a) (1o))))
22
2x41 (())))
This produces similar form to our loss function, with the exception being that h is not volume- preserving. However, this bound is much looser than our proposed bound. We denote as C-VDM a cascading diffusion model (Kingma et al. 202I) designed with this standard con-volume-presevviee hierarchical representation (Ho et al. 2022) (i.e., Section 4.T). All model architecture and hyper- parameters are retained from W-PCDM and LP-PCDM, except for the choice of 211)) ...,, (5)) In preseevtreertttooon property of h significantly degrades
Tables 4 5 66 we that the removing likelihood modeling performance. We theorize that this is due to the looser bound obtained by the variational bound derivation, versus the exact equality afforded by Lemma 4.
20

---

Published as a conference paper at ICLR 2024
Model
CIFAR10 32x32
ImageNet 32x32
ImageNet 64x64
ImageNet 128x128
CVVDM
3.20
4.21
3.87
3.75
Our work LP-PCDM
2,36
3.52
3.12
2.91
W-PCDM
2.35
3.32
2.95
Table Ablative study illustrating the importance of the volume-preserving property on density estimation benchmark. Results are reported in terms of expected negative log likelihood on the test computed bits per dimension (BPD). Details Appendix C
C-VDM
W-PCDM
SVHN
0.43
0.71
0.74
Uniform
0.55
Uniform
1.0
1.0
1.0
Average
0.56
0.90
0.92
Table 5: Ablative study illustrating the importance of the volume- preserving property on performance an out-of-distribution (OOD) benchmark CIFAR1O trained model on selection of anomalous distributions in terms of AUROC. Results from existing models are taken from the literature. Details Appendix
Model
Compression BPD
C-VDM
3.22
LP-PCDM
2.40
W-PCDM 2.37
Table 6: Ablative study illustrating the importance of the volume-presen preeeevppreeviiggg property on lossless compression performance on CIFARIO. Results reported in terms of bits per dimension (BPD). Details in Appendix C
21


Reviews:
[{"review_id": "4ntUtAwsGB", "reviewer": "Reviewer_PAyK", "summary": "This paper proposes the cascaded diffusion models for high-fidelity image synthesis that also works as a likelihood model, by overcoming the intractability issue of the likelihood function in multi-scale diffusion models. To do this, this paper provides some hierarchical model of the latent variables, to reduce the marginal likelihood problem into the cascaded conditional likelihood problem. To do this, a class of transformations called hierarchical volume-preserving maps. Some special yet natural instances of the hierarchical volume-preserving maps include Laplacian pyramids, where the image data is divided into its low-pass filter part and its counterparts sequentially, and the wavelet decomposition, which is more general and more rank-preserving transformation, first orthogonalize the image into four filters (low-low, low-high, high-low, high-high) which is also called subbands. Then, the subbands that include high-pass filter in at least one coordinate (vertical or horizontal) is kept and low-low subband is hierarchically downsampled and again taken the wavelet transform. \n\nThen, the likelihood training of this cascaded diffusion model is done by the factorization of the variational objective of the diffusion model into the sum of the conditional variational objective with respect to each hierarchical latent variables. Finally, the paper suggests the connection of this paper to the optimal transport concept, by providing that the Wasserstein-p distance between the approximate and exact score is upper-bounded by the cascaded diffusion objective. The experiments show that the cascaded volume-preserving diffusion model yields better likelihood measure compared to other existing flow-based models and diffusion models. Furthermore,", "questions": "* According to the specific features of the hierarchical models, the architecture description in appendix B should be more precisely described. Now it is not easy to understand which architecture is used for training multiscale models (single model for all scales, or multiple models such that the (conditional) score for each scale is learned in each model?), including the number of parameters. (This might not be a problem since this paper used the same architecture to the VDM paper.)\n\n===============\n\n * Many parentheses are not closed; I recommend using ( \\left[, \\right] ) and ( \\left(, \\right) ) command.\n * In the first paragraph of Section 5.1, $z^{(1)},\\cdots,z^{(S)} = h(x)$ is considered to be the abuse of denoting the hierarchical volume-preserving map. Please consider using clearer notation to represent this.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The writing of the concepts is clear and easy to understand, starting from the necessity of the cascaded latent variable models, and introduce the diffusion models that are involved with this LVM with the volume-preserving map. And in case of the volume-preserving maps, the paper showed that like in the existing original diffusion models, this hierarchical volume-preserving diffusion models also works  as the minimizer of the Wasserstein distance.\n * This paper showed superior performance in case of the log-likelihood measures of CIFAR-10 datasets, compared to the existing semi-autoregressive methods.", "weaknesses": "* The use of semi-autoregressive hierarchical diffusion models is a common approach, including the use of wavelets, latent variables, and null-space vectors for upsampling the images. The use of cascaded volume-preserving maps do not seem to give differences to these existing works, and the related works on multi-scale diffusion models should be more carefully added.\n * The optimal transport theorem (5.1) does not enough evidence that the hierarchical volume-preserving map is more feasible to use for matching score function than the original diffusion models. Kwon et al. (2022), already showed the similar theorem such that the diffusion model upper bounds the Wasserstein distance between the approximate and the exact score.\n\nD. Kwon et al, \"Score-based Generative Modeling Secretly Minimizes the Wasserstein Distance\" (2022)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ue92whAyB2", "reviewer": "Reviewer_ocEJ", "summary": "This paper proposes considers multi-scale diffusion models, and shows theoretically that one needs volume-preserving scale transformations to make the likelihood behave “nicely”. The empirical results are excellent.\n\n--- \n\nPost-discussion update: My concerns were sufficiently addressed, and all the confusions I had were clarified. This is a high-quality, substantial contribution to diffusion models from solid theoretical understanding.", "questions": "Minor comments\n\n- The eq 3 is looks like the simple forward process, but is instead the forward “posterior”. For instance, DDPM paper (Ho et al 2020) eq 2 LHS is the same equation as eq 3 LHS here, but has a conflicting RHS. Can you clarify which q interpretation you are using, and which q the eq 5 is taken over.\n- I’m not sure I if I understand the premise of the paper. I agree that with invariant transformation we have p(x) = p(h(x)) property. However, this is useful only if the joint distribution p(z^1, …, z^S) decomposes independently into p(z^1) * … * p(z^S); and I don’t see why this would happen. Surely the different scales are highly dependent, and you would have to do a chain rule p(z^S | z^S-1) … p(z^2 | z^1) p(z^1) [or some other decomposition], and in this setting I don’t see the invariant property being useful. The paper should clarify the invariance wrt scale independence, preferably in the introduction and in beginning of sec 4. Right now the main claim of the paper does not convince. For instance, *“if h(x) is the scales of the hierarchical model, then we can directly use the joint likelihood over these scales as the desired model likelihood”* is generally false as far as I can see.\n- Eq 8 has square root, but why? It does not seem to do anything when det=1.\n- What does z(1), z(2), . . . , z(S) = h(x) mean? I don’t understand the notation. Does h(x) return S different things, or does h return one thing as in z1=h; z2=h; etc? h seems to be a downscaling operator, is this true? The nature of “h” is a bit confusing at this stage of the paper. What does h output? What resolution?\n- What does “p_theta trained on z1..zS” mean? Trained how? Against what objective? What kind of p_theta? These statements are overly casual, and need to be made precise, explicit, rigorous and transparent.\n- “Then the model likelihood”. What does model likelihood mean? Can you make this rigorous? This is too informal. Or is this supposed to be informal and imprecise?\n- What does p_theta(x) mean in eq 9? Does it refer to eq 7 or something else? What does it mean to apply h(x) to eq 7: do we apply h() to all z’s together, or separately, or what..?\n- I don’t think the statements after eq 9 are true; I see no convincing arguments for this. It’s not clear what the terms in eq 9 even mean or how they are defined, and the dependency issue of the joint still remains.\n- In eq 10 the h is now some kind of linear operator, while earlier it was a function. Is the idea that h(x)=h*x? It would be good to clarify the distinction between function h and operator tensor h.\n- What does * mean in (j*2)? Is this a product or convolution? If this is a convolution, what does it mean to convolve j with 2?\n- What does fig 2 show? I think it shows y^4 and z^3...z^1. Can you clarify?\n- What does tight frame mean? Can you give a conceptual explanation? What does “h” mean in parseval equation? There is no “h” in the Laplace section of the paper at all: does some of this stuff (y,z,d,u) relate to h?\n- Again, what is the “mapping” h in Wavelet case? Is it the z’s, or y’s, or them together, or something else?\n- The wavelet and Laplacian pyramids both retain the original resolution of the image. Wasn’t the goal of this paper to reduce resolution? I’m a bit confused. Can you clarify the notion of resolution vs scale, and clarify which one you want to reduce? The very first sentence of the paper in introduction talks about super-resolution, which leads me to believe that one should change resolution somewhere in the paper, but neither of the transform types seems to change resolution. Can you clarify these aspects?\n- I don’t see what lemma 4.1. has to do with eq 15. It seems that you just apply the x=z1…zS substitution and use chain rule to split the p(x) to p(zs|z_<s). Where is lemma 4.1 here? Also, the paper has already defined p(x) in eq 7, which seems to conflict with the p(x) in eq 15. Can you clarify? Is this a redefinition, or are both true?\n- What is the q in eq 16?\n- What is z_0^s in eq 16? How do you get them? How do you get the z_1’s? How do you get the z_k’s? What are the terms here: I can’t really follow any of them. Can you explain all three terms (the logp and two KLs). For instance, the regular logp term is the probability of observed image given z^1 while marginalising all intermediate states away. Here isntead we have p(z^s_0 | z_1^s, z_0^<s). So we are looking at one scale likelihood, but somehow conditioning with earlier scale “observations”? I can’t really follow at all what is happening, since very little of eq 16 or the underlying processes have been defined or characterised. Please include an algorithm boxes for training and sampling. It’s also confusing what is h here.\n- I’m having hard time understanding what the eq 18 is conceptually representing. Is the idea that criss-cross the pixels from one image to another by moving the pixel locations around?\n- Eq 19 is a lot of stuff without much motivation, introduction or exposition. I have hard time understanding what this means or how it connects with the paper. Somehow optimising the scale losses is equivalent to reordering pixel locations in the scores…? Err.. what? This feels very strange, and I can’t follow. I also don’t see why this is significant: what are you trying to argue here? How do we benefit from this connection? If you want to present this theoretical connection, it needs to be presented in a way that is digestable, and requires more exposition and explanation, and also helpful illustrations.\n- At experiments I still don’t know what does the function “h” mean in this paper, or what it is. I guess it’s the Laplace pyramid, but not sure which part. I’m also not sure how did we now solve the problem of joint needing to be marginalised to only evaluate the final image. I can’t really connect the problem statement in introduction to the methods presented in the paper. I think the paper needs a method summary section that explains how all the pieces come together to solve the original problem, while including an algorithm box.\n- The paper should cite to simple diffusion, that also tackles multi-resolution.\n- In experiment the key comparison is how does this method fare against diffusion models that utilise laplace or wavelet pyramids; or have an explicit super-resolution component. Looking at table 1 almost all comparison targets are irrelevant old methods, and I fail to see many if any super-resolution/laplace/wavelet methods. There is only a single comparison to a 2022 method, and none to 2023. Given the astronomical pace of the research in diffusion, this is not acceptable. The paper needs to compare both to (i) other super/wavelet/laplace diffusion models, and more comprehensively to 2022 and 2023 diffusion models. The claims of state-of-the-art performance is unsubstantiated.\n- The OOD experiments only seem to compare to quite old generative models. Why not compare to other diffusion models? Wasn’t the point of the paper to improve the multi-scale handling of diffusion models, so surely one should then compare to other diffusion models with more adhoc multi-scalings, or to diffusion models with a single scale only. I’m also not sure what the table 2 rows mean.\n- In 6.3. there are comparisons to uni-scale diffusion models (which is great), but one should also compare to competing multi-scale diffusion models.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- Excellent empirical results, which are great throughout.\n- The method sheds light on the Laplacian and Wavelet decompositions, and proposes a relatively principled training scheme for them. The theoretical analyses of the scale transformations are delightful to see, and this discussion is a significant viewpoint to diffusion models", "weaknesses": "- The theoretical presentation is informal and imprecise. I couldn’t follow the theory or substantiate the claims made by the paper. The final model is also ultimately undefined and feels very adhoc despite the principled approach. It’s also confusing what is the contribution of this paper, since it seems that ultimately the paper just uses the Laplace/Wavelet transforms as-is, which are already well known.\n- The experiments do not compare to recent diffusion works [there is no comparisons to 2023 methods, and only one 2022 method (!)], and lack direct comparisons to diffusion models with super-resolution, wavelet or laplace transforms. The experiments are quite roundabout and describe just the overall performance, and don’t directly substantiate specific contributions made in this paper. It is then difficult to see why we improve, or if the theoretical contributions had anything to do with it. There are no ablations. There are very little result exposition or illustrations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8lIIJ5kCa7", "reviewer": "Reviewer_y69M", "summary": "This paper shows that volume-preserving maps can be used to define multiscale (cascaded) generative models with tractable likelihood. Extensive numerical experiments demonstrate the improvements in likelihood modeling, out-of-distribution detection, and compression achieved by the ability to use multiscale models. The paper also makes a connection between multiscale diffusion models and optimal transport with the earth's mover distance.", "questions": "- If $z$ is higher-dimensional than $x$, then $z = h(x)$ does not admit a probability density (it is not absolutely continuous with respect to the Lebesgue measure). How do the authors deal with this difficulty, and how is Lemma 4.1 not meaningless in this case?\n- The authors mention switching to a different architecture and weighting scheme for optimizing FID as opposed to likelihood. Does the improvement in FID come with an increase in BPD? If so, the tradeoff between the two should be acknowledged explicitly.\n- Why not using $M=1$ in the OOD detection task? In high-dimensions, we expect from concentration of measure that $-\\log p(x) \\approx \\mathbb H(p(x))$ for almost all $x$, so we should be able to detect changes in distribution from just one sample?\n\nMinor suggestions:\n- Multiscale image modeling has a much older history than 2017, e.g., [4-10] for a very incomplete list, which seems more relevant than the reference [Horstemeyer, 2010]. Though these earlier works generally focus on other tasks such as denoising or compression, the motivations for using a multiscale representation remain the same.\n- Line before equation (5) should read $\\log p_\\theta(x_0)$ instead of $p_\\theta(x_0)$.\n- Any linear map could be used for cascaded likelihood modeling, even if its determinant is not one (as long as it is not zero). Indeed, it just introduces a constant offset in the likelihood, which has no effect for training, and this offset can be estimated once offline for test-time likelihood evaluation. In particular, the standard cascaded hierarchy is in this case.\n- The discussion below Definition 1 should state that it requires $\\mathrm{dim} \\mathcal{Z} \\geq \\mathrm{dim} \\mathcal{X}$ with a full-rank Jacobian $\\mathrm{rank}(A) = \\mathrm{dim} \\mathcal{X}$. Also, the convention used for the Jacobian matrix should be stated to avoid confusions with its transpose: i.e., $A \\in \\mathbb R^{\\mathrm{dim} \\mathcal{Z} \\times \\mathrm{dim} \\mathcal{X}}$.\n\n\n[4] P J Burt and E H Adelson. The Laplacian pyramid as a compact image code. IEEE Trans Comm, Apr 1983.\n\n[5] S Mallat. A wavelet tour of signal processing: The sparse way. Academic Press, 2008.\n\n[6] A Chambolle, R A DeVore, N Lee, and B J Lucier. Nonlinear wavelet image processing: Variational problems, compression, and noise removal through wavelet shrinkage. IEEE Trans Image Processing, Mar 1998.\n\n[7] R W Buccigrossi and E P Simoncelli. Image compression via joint statistical characterization in the wavelet domain. IEEE Trans Image Processing, Dec 1999.\n\n[8] M J Wainwright, E P Simoncelli, and A S Willsky. Random cascades on wavelet trees and their use in modeling and analyzing natural imagery. Applied and Computational Harmonic Analysis, Jul 2001.\n\n[9] L Şendur and I W Selesnick. Bivariate shrinkage functions for wavelet-based denoising exploiting interscale dependency. IEEE Trans Signal Processing, Nov 2002.\n\n[10] J Portilla, V Strela, M J Wainwright, and E P Simoncelli. Image denoising using scale mixtures of Gaussians in the wavelet domain. IEEE Trans Image Processing, Nov 2003.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The authors tackle an important problem, which is the use of cascaded approaches for likelihood modeling. The proposed solution is simple (which I mean in the best possible way!) and the numerical experiments are solid and convincing. The paper is also clearly written (except that the core concept is not stated until page 6, see Weaknesses).", "weaknesses": "- The theoretical setting of the paper is general, with possibly non-linear volume-preserving maps $h$, but in practice only linear $h$ are used, which thus boil down to using an orthogonal transform (for wavelets) or a tight frame (for Laplacian pyramid). The linear setting already leads to important improvements, and the possibility of the extension to non-linear maps is interesting for future research, but it obfuscates slightly the results of the paper behind unnecessary complexity. In particular, using \"Hierarchical volume-preserving maps\" in the title instead of a more explicit reference to orthogonal wavelet transforms (which yield the best results) does not accurately describe the contents of the paper and makes it harder for the reader to connect the contributions of the paper to known concepts. Orthogonal transforms are not mentioned anywhere before page 6, yet are the core component of the method.\n- Related to the previous point, orthogonal wavelets have been used for multiscale generative modeling in several missed previous works: see [1] for nomalizing flows, [2] for direct likelihood modeling, and [3] for diffusion models (closest to the setting of the paper, though the focus is on image quality rather than likelihood modeling). In particular, [1] and [2] critically rely on the orthogonality of the transform to model the likelihood. Also, the recombination of $z^{(<s)}$ into a same-resolution image mentioned in Appendix B is crucially used by [1-3] to parameterize efficiently the conditional likelihood.\n- The connection to the earth's mover distance seems problematic to me. First, it is unclear what benefits it brings to the paper: I do not understand how to interpret Theorem 5.1, since here the transport is between the true and approximated scores (seen as distributions on $\\mathbb R^2$), as opposed to transport between distributions of images as is usually the case. Second, the soundness of the results look questionable to me. For instance, the Wasserstein-p distance is defined between probability distributions, which thus requires that the scores have non-negative entries that sum to one. The authors do not mention this restriction anywhere in the paper. Second, the proof seems to assume that the wavelet coefficients of the image score $\\nabla \\log q(x)$ are the wavelet conditional scores $\\nabla \\log q(z^{(s)} | z^{(<s)})$, but this is not the case. Indeed, one has $\\log q(x) = \\log q(z^{(1)}) + \\log q(z^{(2)} | z^{(1)})$ (with $S = 2$ for simplicity), and thus the $z^{(1)}$ component of the image score includes an additional term coming from the high-frequency score component (called \"free energy\" in [2]). For this reason, the equivalence between wavelet and Laplacian pyramid for the score matching losses also breaks down (which is a more plausible explanation for the difference in performance than the authors' footnote 1).\n\nDespite these weaknesses, I recommend acceptance (see Strengths above). I am willing to increase my score if the authors address these points.\n\n[1] Jason J Yu, Konstantinos G Derpanis, and Marcus A Brubaker. “Wavelet Flow: Fast Training of High Resolution Normalizing Flows.” In Advances in Neural Information Processing Systems, 2020.\n\n[2] Tanguy Marchand, Misaki Ozawa, Giulio Biroli, and Stephane Mallat. “Wavelet Conditional Renormalization Group.”\n\n[3] Florentin Guth, Simon Coste, Valentin De Bortoli, and Stephane Mallat. \"Wavelet score-based generative modeling.\" Advances in Neural Information Processing Systems, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JSOz9EpyWK", "reviewer": "Reviewer_hZLh", "summary": "The paper proposes the use of cascaded diffusion models for optimizing the likelihood of samples in diffusion models, with the motivation that cascaded models have shown excellent performance on image quality but have not been utilized as likelihood models. The paper goes on to present a method to facilitate likelihood modelling with cascaded diffusion by introducing *hierarchical volume-preserving maps*, allowing to express the model likelihood as a joint likelihood over different hierarchical scales. In practice, this includes wavelet and Laplacian pyramid transforms. The paper also proposes a surprising connection with the denoising score matching loss over the different scales of the hierarchical maps and optimal transport. The new models show impressive performance in model likelihoods on CIFAR-10 & different-resolution ImageNet datasets, out-of-distribution detection and lossless compression.", "questions": "### On the connection to the EMD metric:\n- Given that it is the central part of the proof, I would like more elaboration on how exactly the connection to Theorem 2 of Shirdhonkar & Jacobs is made. One way to do this would be to repeat the theorem and its assumptions in the paper and showing step-by-step how does it apply here. In particular, I wasn’t able to see how do the $2^{-j(s+n/2)}$ terms in their statement of the Theorem connect to the application of it in the paper. \n- What exactly does the paper claim by making the connection with score matching on wavelet representations and EMD? The last sentence of section 5.2. seems to claim that the method allows training diffusion models with the EMD measure, which does not quite seem to be the case here. While the connection between wavelets and EMD is definitely interesting, the connection to diffusion models doesn't seem particularly important as of now.\n### On the ELBO and the necessity of volume-preserving maps:\n- I might be confused, but it seems to me that volume-preserving maps are not necessary to form an ELBO with cascaded diffusion models. My thinking is as follows: We have the data $x$, and S sequences of latent variables $z_{1:T}^s$ with different resolutions and noise levels. The generative process, as defined in the paper, is $p_\\theta(x,z)=p(x|z_1^S)\\prod_{s=1}^S\\prod_{t=2}^T p_\\theta(z_{t-1}^s|z_{t}^s,z_1^{(<s)})p(z_T^s)$. We can then form the following inference process that is factorized for the different resolutions: $q(z|x) = \\prod_{s=1}^S \\prod_{t=2}^T q(z_t^s|z_{t-1}^s)q(z_1^s|x)$. Here $q(z_1^s|x)$ downsamples $x$ and adds the smallest level of noise in the diffusion process. Now if we form the ELBO, we get: $E_q[-\\log p_\\theta(x)]\\leq E_q[-\\log\\frac{p_\\theta(x,z)}{q(z|x)}]=E_q[-\\log \\frac{p_\\theta(x|z_1^S)\\prod_{s=1}^S\\prod_{t=2}^Tp_\\theta(z_{t-1}^s|z_t^s,z_1^{(<s)]})p(z_T^s)}{\\prod_{s=1}^S \\prod_{t=2}^T q(z_t^s|z_{t-1}^s)q(z_1^s|x)}]$, and further $=E_q[-\\log p_\\theta(x|z_1^S) - \\sum_{s=1}^S[\\sum_{t=2}^T\\log\\frac{p_\\theta(z_{t-1}^s|z_t^s,z_t^{(<s)})}{q(z_t^s|z_{t-1}^s)} + \\log p(z_T^s) - \\log q(z_1^s|x) ]]$. From here, we can follow the standard derivation to get to the KL divergences (e.g., Appendix A in Ho et al., Eq.19., with the difference that their $x_{1:T}$ is redefined to $z_{1:T}^s$ and $x_0$ to $x$). I think this results in the same ELBO as in the submission, with the difference that instead of having a $p(x|z)$ term for each scale, this would only have it for the last scale. Do the authors agree with this point of view, or have I potentially misunderstood something? In case I have not misunderstood, this seems to be a major issue with the paper, since there is no comparison to cascading diffusion models without volume-preserving maps, making the significance of volume-preserving maps unclear. \n\n## Overall\n\nWhile the paper is well-written, the method showcases improvements and is well-evaluated, I hesitate to give an accepting score before I can see more clearly what is the benefit of volume-preserving maps. It is possible that I have missed something, and if so, I am willing to raise my score. Otherwise, I think that the paper requires more elaboration on what exactly is the role of volume-preserving maps in likelihood training of cascaded diffusion models. I would also like to see a clearer derivation for the EMD metric connection, as well as more elaboration on what is the significance of the connection. \n\nReferences:\nHo et al., Denoising Diffusion Probabilistic Models, NeurIPS 2020", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "+ The paper clearly demonstrates that the proposed method provides improvements in log-likelihoods over non-cascaded models. As far as I am aware, the paper is the first to showcase that cascaded diffusion models can be used to improve log-likelihoods (in contrast to just the image quality).\n+ The paper is well-written and easy to follow.\n+ The new method is evaluated thoroughly, showcasing the performance of the model on applications relevant to log-likelihoods: OOD-detection and data compression.\n+ I found the proposed connections to optimal transport particularly surprising, and this connection may be useful in further work.", "weaknesses": "## Minor issues\n- The notation in some parts could be improved: E.g., eq. 27 takes the expectation over $\\epsilon$ but refers to the noisy data variables $\\tilde z^{(s)}$. On the next line, $\\tilde z^{(s)}$ transforms to $\\tilde x_k$. Small issues like this make the paper slightly difficult to read at times. \n\nFor other issues, see the questions part.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)