Generative Modelling with Inverse Heat Dissipation
Authors: Severi Rissanen
Markus Heinonen
Arno Solin
Venue: ICLR 2023 poster (2023)
Keywords: diffusion model
partial differential equation
inductive bias

Abstract: While diffusion models have shown great success in image generation, their noise-inverting generative process does not explicitly consider the structure of images, such as their inherent multi-scale nature. Inspired by diffusion models and the empirical success of coarse-to-fine modelling, we propose a new diffusion-like model that generates images through stochastically reversing the heat equation, a PDE that locally erases fine-scale information when run over the 2D plane of the image. We interpret the solution of the forward heat equation with constant additive noise as a variational approximation in the diffusion latent variable model. Our new model shows emergent qualitative properties not seen in standard diffusion models, such as disentanglement of overall colour and shape in images. Spectral analysis on natural images highlights connections to diffusion models and reveals an implicit coarse-to-fine inductive bias in them.

Published as a conference paper at ICLR 2023
GENERATIVE MODELLING WITH INVERSE HEAT DISSIPATION
Severi Rissanen, Markus Heinonen & Arno Solin Department of Computer Science Aalto University severi. rissanenfaalto.
ABSTRACT
While diffusion models have shown great success in image generation, their noise-inverting generative process does not explicitly consider the structure images, such their inherent multi-scale nature. Inspired by diffusion models and the empirical success of coarse-to- fine modelling, we propose a new diffusion-like model that generates images through stochastically reversing the heat equation, a PDE that locally erases fine-scale information when run over the 2D plane of the image. We interpret the solution of the forward heat equation with constant additive noise a variational approximation in the diffusion latent variable model. Our new model shows emergent qualitative properties not seen in standard diffusion models, such as disentanglement of overall colour and shape in images. Spectral analysis on natural images highlights connections to diffusion models and reveals an implicit coarse-to-fine inductive bias in them.
INTRODUCTION
Diffusion models have recently become highly successful in generative modelling tasks (Ho et al., 2020; Song al., 2021d: Dhariwal & Nichol, 2021). They are defined by forward process that erases the original image information content and reverse process that generates images iteratively. The forward and reverse processes of standard diffusion models not explicitly consider the inductive biases of natural images, such as their multi-scale nature. In other successful generative modelling settings, such as in GANs (Goodfellow et al., 2014), taking multiple resolutions explicitly into account has resulted in dramatic improvements (Karras et al., 2018; 2021). This paper investigates how to incorporate the inductive biases of natural images, particularly their multi-resolution nature, into the generative sequence of diffusion-like iterative generative models.
The concept of resolution itself in deep learning methods has received less attention, and usually, scaling is based on simple pixel sub-sampling pyramids, halving the resolution per step. In classical computer vision, another approach is the so-called Gaussian scale-space (lijima, 1962; Witkin, 1987; Babaud et al., 1986; Koenderink, 1984), where lower-resolution versions of an image are obtained by running the heat equation, a partial differential equation (PDE, see Fig. 1) that describes the dissipation of heat, over the image. Similarly to subsampling, the heat equation averages out the images and removes fine detail, but an arbitrary amount of effective resolutions allowed without explicitly decreasing the number of pixels. The scale-space adheres to set scale-space axioms, such as rotational symmetry, invariance shifts in the input image, and scale invariance (Koenderink, 1984; Babaud al.,
Information melting forward process Au
<- -training data->
generativve sequence
Generative inverse problem
Figure 1: Example of the forward process (dur- ing training) and the generative inverse process (for sample generation).

---

Published as a conference paper at ICLR 2023
Standard diffusion model
Inverse heat dissipation model
Non-invertible forward process
Non-invertible forward process
Generative reverse process
Generative reverse process
Figure 2: Comparison of generation by generative denoising and inverse heat diffusion, where the focus of the forward process is in the pixel space in the left and the 2D image plane on the right.
1986), also linking to how early biological vision represents signals (Lindeberg. 201 3a:b). While scale-space has been utilized in the context of CNN architectures (Worrall & Welling, 2019; Pintea et al., 2021), it has not been considered in generative models.
We investigate inductive biases in diffusion-lype generative models by proposing a generative model based on directly reversing the heat equation and thus increasing the effective image resolution, illustrated in Fig. 2. We call it the inverse heat dissipation model (IHDM). The intuition is that as the original image information content is erased in the forward process, a corresponding stochastic reverse process produces multiple plausible reconstructions, defining generative model. Samples from the prior distribution are easy obtain due to the low dimensionality of averaged images, and we adopt a training data based kernel density estimate.
Our main contributions are: (i) We show how to realise the idea of generative modelling with inverse heat dissipation by interpreting a solution of the heat equation with small additive noise as an inference process in diffusion-like latent variable model. (ūi) We investigate emergent properties of the heat equation-bart based model: (a) disentanglement of overall colour and image shape (b) smooth interpolation, (c) the forward process inducing simplicity to the learned neural net function, and (d) potential for data efficiency. (iii) By analysing the power spectral density of natural images, we show that standard diffusion models implicitly perform different type of coarse-to-fin generation, shedding light on their inductive biases, and highlighting connections and differences between our model and standard diffusion models. Code for the methods in this paper is available at: https: /ttp:/ggttttttet com/ Awlttaalaaaa..........aaaaawaaal wwwtttoooegeeeeeeeeee generat ive inverse -eaetehhettt--- - dissipptioon
2
METHODS
The main characteristic of the forward process is that it averages the images in the data set, contracting them into a lower-dimensional subspace (see Fig. 2 right). We define with the heat cquation, linear partial differential equation (PDE) that describes the dissipation of heat:
i Ot
Forward PDE model:
u(x y, Au(x, v)
where u : I22 x R. + R is the idealized, continuous 2D plane of one channel of the image, and A 2 is the Laplace operator. The process run for each colour channel separately. We use Neumann boundary conditions (0u/az- 0v/2y = 0) with zero-derivatives at boundaries of the image bounding box. This means that as + oo, each colour channel is averaged out to the mean of the original colour intensities in the image. Thus, the image is projected to R In principle, the heat equation could be exactly reversible with infinite numerical precision, but this is not the case in practice with finite numerical accuracy due to the fundamental ill-posed nature of the inverse heat equation. Another way to view is that with any amount of observation noise added on top of the averaged image, the original image cannot be recovered exactly (see Kaipio & Somersalo, 2006, for discussion).
The PDE model in Eq. (1) can be formally written in evolution equation form as u(z, y. f) = F(t) u(x, y, t)le=tg. where F(t) expl(t tu) A] is evolution operator given in terms of the operator exponential function (see, e.g., Da Prato & Zabczyk, 1992). We can use this general formulation to efficiently solve the equation using the eigenbasis of the Laplace operator. Since
2

---

Published as a conference paper at ICLR 2023
Standard diffusion model
Initial states 1o
Sample trajectories
Pla (12-3 Ma
UK
q uu uo
q(u Up-13
Forward process
Inverse heat dissipation model
Reverse process
Gauissian
t=t8
Pa (1g-3 uu
t=
Determänistic blurring
2q(k (u)
g( un)
g(0
(R)
Hest equation
(b) Illustration of the parameter and joining of for- ward process paths, enabling branching in the reverse
(a) Graphical models
Figure 3: (a) Graphical model of IHDM vs. standard diffusion model, highlighting the factorized inference process. (b) A sketch of how the stochasticity of q q(uu Ug), controlled by o, allows the forward process paths to effectively merge in the probabilistic model. As opposed to trying to invert the fully deterministic heat equation, this makes the reverse conditional distributions well-defined.
use Neumann boundary conditions, the eigenbasis is a cosine basis (see full details in App. A.1). The observed finite-resolution image lies grid, meaning that the spectrum has a natural cut- off frequency (Nyquist limit). Thus, we can formally write the operator in terms of a (finite) cigendecomposition A & VAVT, where V is the cosine basis projection matrix, and A is a diagonal matrix with negative squared frequencies the diagonal. The initial state is then projected on to the basis with the discrete cosine transform (ū = V u = DCT(u)) in O(N log N) time. The solution is given by the finite-dimensional evolution model, describing the decay of frequencies
u(t) = F(L) u(0) = exp(VAV) 4) u(0) = V exxp(Ar)V (())
a(t) exp(Ar)ū(O), (2)
where F(L) E RRNN (not expanded in practice) is the transition model and u(0) the initial state. The (n"2/n0
diagonal terms of are the negative squared frequencies -Anm n 2n² (nn///a + m² //2/), where W
and H are the width and height of the image in pixels, n 0,.... W 1 and m = 0...., H-1. As A is diagonal, the solution is fast to evaluate and implementable with a few lines of code, see App. A.I.
The heat equation has a correspondence to the Gaussian blur operator in image processing: In an infinite plane, simulating the heat equation up to time & equivalent to a convolution with a Gaussian kernel with variance / 2t (Bredies & Lorenz, 2018). The heat equation has the advantage that it exposes the theoretical properties of the process, e.g.. the frequency behaviour and boundary conditions, and is potentially better generalizable to other forward processes and data domains.
?
2.1 GENERATIVE MODEL FORMULATION
We seek to define probabilistic model that stochastically reverses the heat equation. Even if one could formally invert Eq. (2), we are not interested in the deterministic inverse problem per se, but in formalizing generative model with characteristics given by the forward problem. The generative process should also branch into multiple plausible reverse paths. We formally break the reversibility by introducing small amount of noise with standard deviation in the forward process and incorporate it into the general mathematical framework for diffusion models -Sohl-Dickstein al., 2015). Effectively, this before turning into noise. The idea makes the reverse conditional distributions probabilistically well defined, as illustrated in Fig. 3b. We define the time steps t,, t2........ tx that correspond to latent variables Uj, each of which has the same dimensionality the data ug. Our forward process, or formally the variational approximation in the latent variable model, is defined as
lower limit to how low the frequency components can decay
K
Furward process/ / Inference distribation
q(uiK Uo) = II u( uo)
II Nuu [((x) uo, d²1), k=1
(3)
1=1
3

---

Published as a conference paper at ICLR 2023
where F(tx) is the linear transformation corresponding to simulating the heat equation until time Ex and the standard deviation a is small constant (e.g., 0.01 if data is scaled to [o, 1). Note that instead of having the forward be a Markov chain as in regular diffusion models, we factorize the noise in a way that intuitively treats it as observation noise on top of the deterministic heat equation, as also visualized in Fig. 3a. The generative, or reverse process, is Markov chain that starts with the prior state uk and ends at the observed variable Ug. We define it with Gaussian conditional distributions:
K
K
Reverse process/ / Generative model
po (u):k) = p(uk) II (u u&) = p(ux) II N pe k), II), (4)
A=1
where 0 are model parameters and 5 is the standard deviation of the noise added during sampling. We show the whole structure in Fig. 3a, where we highlight the structural difference to standard diffusion models. Fig. 3b provides intuition on the noise parameters o and ō; the noise o acts as an error tolerance or relaxation parameter that measures how close two blurred images have to be to become essentially indistinguishabl.. With a non-zero an initial state ug has formal probability of going along different paths. The parameter å is a free hyperparameter that controls the sampling stochasticity, which in turn defines the trajectory.
Our goal is to maximize marginal likelihood of the data p(u,) = I (u |u k) (u duj.k- Taking the VAE-type evidence lower bound for the marginal likelihood with the generative and inference distributions defined, we get
log po(uo:k) q q(u1:K un)
log po (uo) S Egg
(5)
K
q(ux uo) -Eg log po(ux)
Pol Po [ug-1 (u)) log (uo uu)))
Σ
log
(6)
(ux |u))
2=
K
B,y
DKL DKula(uk uo) [l((u)]]
+
DKL Dkllg(us-1 un) (po(uu
uu)) log pe (uo (1))
(7)
Lx
L
where the different parts of the process factorize a similar, although somewhat simpler, way as in diffusion probabilistic models booh--Dickstein et al., 2015; Ho et al., 2020). The terms Lk-1 KL divergences between Gaussian distributions
Ey [L 1] E DKL[q (q(uu un)
|| po (u=-1l [ux)]]
(8)
o² N Eeluu uo)
1
+ 2.V log 2
1/2
N
uao u k) F(1x-1) un 113
(9)
2 82
82
fo(ue (k) (F (F(tu-13 ) un u
1
E2, [o] = Ey,1 log po (uo [ui]]= =
Bgtus mo)
uu 1) uo
11
+ N log log(6V/TT), (10)
25
where N is the number of pixels in the image. We evaluate the loss function with one Monte Carlo sample from the inference distribution q(u1 Ug). The losses on all levels are direct MSE losses where we predict slightly less blurred image from blurred image that has added noise with variance o². The sampling proceeds by alternating the mean update steps from the neural network and the addition of Gaussian noise with variance d. We summarize the training process in Alg. 1, and the sampling process in Alg. 2. both of which are straightforward to implement. In practice, the algorithms mean that we train the neural net deblur with noise-injection regularization, and sampling consists of alternating deblurring and adding noise. We further parametrize pa (u,, k) with a skip connection such that s (ig [ug; k) u, + fo (Uz, k)), which stabilizes training. The motivation is that seek to take small step backwards in differential equation. With the skip connection, the loss functions in Eq. (9) and Eq. (10) resemble the denoising score matching objective, except that are not predicting the denoised version of ug, but a less blurry u
Prior.
We can use any standard density estimation technique for the prior distribution p(ux) since the blurred images are effectively very now-dimensional. We use Gaussian kernel density estimate with variance 8², which is a reasonable estimate if the blurred images at level K are cow-dimensional and close enough each other. We obtain samples by taking a training example, blurring it with
4

---

Published as a conference paper at ICLR 2023
Algorithm Loss function for a single data point
Algorithm Sampling
Ug Sample from training data
+ K
> Start from terminal state
A r Sample from {I,..., K}
u p(uk)
Sample from the blurry prior
ug f F((x)uo)
Eq. (2) > see Eq. (2) Training noise
while k > 0 do
u f- Fft
E NO, 821)
Sampling noise
NN I2)
u - uo u, k) EEk D Reverse step + noise k - 1-1 end while
ûg f un EE
> Perturb image
Loss c [ kt)
Us 1
F(Kx), and adding noise with variance 8², Using kernel density estimate means that the term Lx is constant but also tricky to calculate efficiently for log-likelihood evaluation due to the high- dimensional integral and multiple components in the kernel density estimate. In App. A.3, we provide a further variational upper bound on LK that can be evaluated without numerical integration.
Asymptoties. While the model introduces the desirable explicit multi-scale behaviour, we also have to drop some other established results related to diffusion models, such as theoretical guarantees about Gaussian reverse transitions being optimal in the limit of infinite steps. To provide more intuition, we point out the following connection to the early score-based generative modelling work (Song& Ermon, 2019; 2020): In the limit K + oc, the loss function Ls becomes equivalent to the denoising score matching loss with noise level a, and 5 = V2a, the sampling procedure is equivalent to running Langevin dynamics sampling with certain step size on given blur level. Generation then happens by slow annealing of sampling toward less blurry distribution. In practice, we take directed steps backwards in the heat equation instead and do not limit to v /2o, but the result gives intuition for why a Gaussian transition is good choice and first guess at the correct ratio of 8/o (full details in App. A.4).
2.2 IMPLICIT COARSE-TO- CA-RSE-TO-FING GENERATION IN DIFFUSION MODELS
The frequency behaviour of natural images clarifies connections and differences between the new model and standard diffusion models. It also explains and characterises the well-known phenomenon that, in practice, diffusion models tend to create informative content in a coarse-to-fine fashion. The power spectral density (PSD) of natural images obeys an approximate power law 1/f", where often 2 (van der Schaaf & van Hateren, 1996; Hyvärinen et al., 2009). When displayed on a log-log scale, the power spectral density is thus
Frequency
10
10
Power spectral density
approximately a straight line. In App. A.5, we show that if we add isotropic Gaussian noise, the PSD of the noise and the PSD of the original image are
PSD of added noise
additive in expectation. Thus, the highest frequencies get drowned out by the noise while the lower frequencies stay intact. When continuing the process, the noise masks more frequencies until the lowest frequencies have disappeared, as visualised in the red-coloured PSDs in Fig. 4. Thus, in the reverse process, the diffusion model generates frequencies starting from the coarse-graines structure and progressing toward fine details. App. B.6 contains more details on the PSD calculations. Concurrently, the implicit spectral inductive bias was also noted in Kreis et al. (2022).
10
Figure 4: The 1/f" power spectral density in natural images induces an implicit coarse-to- fine inductive bias in diffusion models.
The result also shows differences between our model and standard diffusion models. While the frequency content in standard diffusion models is implicitly removed by drowning it out in noise, we do it explicitly by decaying the highest frequencies faster than the lower ones, as noted in Eq. (2). The noise level a sets a floor for the frequency components. Since the frequencies decay at rates corresponding to the heat equation, our process results in explicit range of effective resolutions.
5

---

Published as a conference paper at ICLR 2023
3 3
3 3 3
CIFAR-I0
18.96
LSUN CHURCIES 125 128
FID 45.06
8
8
APHQ 250x 266
43.39
3
3
3
3
FFHQ 250
FID 64,91
2
2
2
(a) Examples of generative sequences
(b) Hierarchical generation process
Figure 5: (a) Generation sequences for different data sets. Generation starts from flat image and adds progressively more detail. The FID scores are next to the sequences. (b) The process is stochastic, and any given image in the image can progress towards multiple directions as shown on the right. More uncurated generated samples in App. D.I.
EXPERIMENTS
We showcase generative sequences, quantitative evaluation, and analyse the noise hyperparameters and 6. We then study emergent properties, starting with the overall colour and other features becoming disentangled. Next, contrary to standard diffusion models, interpolations in the full latent un smooth. We also show that the forward heat process induces structure the function learned by the neural net. Finally, we show that the model can generalise just from the first 20 MNIST digits.
Architecture and hyperparameters. Similarly many recent works diffusion models (Ho et al., 2020; Song et al., 2021d; Nichol & Dhariwal, 2021; Dhariwal & Nichol, 2021), we use a U-Net architecture (Ronneberger et al., 2015) with residual blocks and self-attention layers in the low-resolution feature maps. We list the architectural details for different data sets in App. B. We choose o 0.01 (data scaled [J, although the model is not particularly sensitive to the value, as shown in App. C.1. We use K = 100 iteration steps MNIST (LeCun et al., 1998) 200 steps on CIFAR- 10 (Krizhevsky. 2009), AFHQ (Choi al., 2020), and FFHQ (Karras et al., 2019), and 400 on LS UN-CHURCCHSS (Yu et al., 2015). The time 4 in in the inference process is spaced logarithmically from near zero to Lバ = in m/n/// where 0 B _max is the effective length-scale of blurring the end of the process as described in the beginming of Sec. 2. We set it to half the width of the image in all experiments unless mentioned otherwise. For CIFAR-10, we set to 24 pixel-widths. We do not add noise the last step of sampling since that cannot increase image quality.
Generative sequences. Fig. 5a showcases the generative sequences for the data sets. Generation starts from blank image and progressively adds more fine-scale structure. Since the model is trained to reverse the heat equation, it effectively redistributes the original image mass to a random image with the split the sequence into two at specified time steps. The large-scale structure gets determined in the beginning, and successive bifurcations lead to smaller and smaller changes in the output image. We present uncurated samples from all five data sets in App. D.I.
average colour. We visualize the stochasticity of the process in Fig. 5b, where
Quantitative evaluation. We evaluate the FID scores (Heusel al., 2017) on the chosen data sets with IHDM, and list them on the right side of Fig. 5a. While the results are not yet as good as state-of-the the-rrr diffusion models and GANs, we find the image quality promising, In particular, the CIFAR-10 FID of DDPM (Ho et al., 2020) was 3.17 in the original paper, and the current state-of-the art methods (Sauer et al., 2022) have FID scores of 1.85, compared to our 18.96. For qualitative comparison, we refer the reader to Fig.1. 1. in (Ho et al., 2020) and our Fig. 19. In App. C.3, also look at the marginal log- likelioodo values and note that their optimal values do not correspond to optimal FID values when we vary the sampling noise parameter 8.
The importance of a. We already noted that a non-zero a is essential for the probabilistic model to be sensible. In App. C.5, we show empirically that with 0, the model fails even MNIST, producing random images. An intuition is that directly trying to reverse the heat equation without any regularization is unstable. In that sense, a non-zero regularizes the reverse process.

---

Published as a conference paper at ICLR 2023
The effect of noise 8. As noted in Sec. 2.1, we can expect the optimal ratio of 5/or to be larger than one. In Fig. 6, we see that 6 = 0 mainly sharpens the prior image into a shape. As we increase 5 above o, more and more detail appears until the images degenerate into noise. The optimal ratio 5/0 is approximately from 1.25 to 1.3, and 1 1.25 works as good default value on all data sets. In App. C.I, we show that the optimal ratio of 5/ar does not depend on the absolute value of a. We provide thorough d sweeps in App. D.3. Deterministic sampling is possible in standard diffusion models through the probability flow ODE (Song et al., 2021d) the DDIM (Song al., 2021a) formalisms, but here it seems that the noise is key factor in producing the information content.
Disentangling colour and shape. We show that the overall colour and other characteristics of the generated image can
Increasing sampling noise
6=0.0 4=m=0.01
8=0.012
6=0.125
6=0.1
8=0.0135
become disentangled with the model. Fix- ing the noise steps and only changing the prior uk. the process carves out similar image with different average colours, vi- sualized in Fig. 7a for 1 128x. x 128 FFHQ model with B,maa 128.
Smooth interpolation. We can also the latent U1:K to interpolate between generated images in a perceptually smooth way, as shown in Fig. 7b. We use linear interpolation on the input state ug and a spherical interpolation on the noise (see App. D). As noted in Ho et al. (2020), the corresponding trick results in non-smooth interpolations with a standard diffusion model (DDPM, Fig. their Appendix). We showcase the result with a DDPM in Fig. 7b, where the interpolation passes through features that are not present in either endpoint. A connection to previous work is the StyleGAN architecture (Karras et al., 2019), where explicit modulation of resolution scales resulted in smoother interpolations. Our latent similarly hierarchical, with different steps corresponding to different resolutions.
Figure 6: When the sampling noise parameter 8=0, the model effectively sharpens the blurred image, with no new details added. As 5 increases over the training noise o, the results become more fine- greeggaanedd and detailed, and finally noisy. Here, the noises are sampled once and scaled with 5 for easier comparison.
Inductive bias on the learned function. The heat forward process sets inductive bias and encourages structure on the function the neural network tries to approximate. While we use the DCT- elementary approach would have been grid-
based approach to simulate the forward process, based finite difference approximation. It yields the following Fuler step for the reverse heat equation:
0
(1
1
0) 1)
u xx, y, l. di) ze u(x y. 1)
u(x, y, dr.
(11)
Here, (as, E Il are locations on the discrete pixel grid of the image, and 's' is discrete convolution with the given sharpening kernel that corresponds to the negative Laplace operator. Since small
Generation input state u.
JHDM
DDPM
Uo
(a) Disentanglement of colour and other features
(b) Interpolations
Figure (a) By setting the noise added during the generative process to constant value and changing the starting image, which only contains information about the colour, the process creates highly similar faces with different overall colour schemes. (b) By also interpolating the noise steps in addition to the starting image. we obtain smooth interpolations between any two images in model (IHDM). In standard diffusion model (DDPM), the corresponding interpolation is non-smooth, passing through features that not present in either endpoint, similarly as in Ho et al. (2020).
7

---

Published as a conference paper at ICLR 2023
input gradient
input gradient
Generation
THDM
DDPM
NN
NN
IHDM
Diffusion
Figure 8: Left: The input gradients towards the end of the process for the first channels of the pixels highlighted in the image for our model (IHDM). Middle: The same for a standard diffusion model (DDPM). Right: The input gradients for pixel at the center of the image with respect to number of generation steps. The images correspond to the generation step in question. The colours are scaled to ssmllgg scale with linear cut-off at 0.1 on the left and middle and 0.01 on the right.
updates along the ideal reverse heat equation are approximately convolutions, this should be reflected in the learned function. We probe into the network by visualizing the input gradients in Fig, 8 for our model and a denoising diffusion probabilistic model (DDPM) for comparison. The learned functions are circularly symmetric and localized in the sense that perturbations at far-away pixels do not affect the output. contrast, the dependence more global and complex for DDPM. This reveals the well-localized nature of the IHDM compared to the DDPM (see also Fig. 13 in App. C.2 for the correlation structure in samples pre-conveeeence during training).
Few-shot learning. Since the inductive bias of the generative process sets prior natural images, the model can be highly data efficient. This is showcased in App. C.2 by training a standard DDPM and IHDM on the first 20 digits of MNIST. While DDPM either fails to produce convincing samples or overfits the data, IHDM can produce meaningful generalisation with only 20 data points.
4 RELATED WORK
Diffusion models (Sohl-Dickstee et al., 2015) have seen fast development since the first papers on score-based generative modelling (Song& & Ermon, 2019; 2020). Score-based models were later shown to be connected with the original diffusion probabilistic models by (Ho et al., 2020). A reverse SDE formalism unified the framework in Song al. (2021d) (later extended in Song et al., 2021b; Huang et al., 2021). Dhariwal & Nichol (2021) obtained state-of-the-a performance on ImageNet. Theoretical developments include the works by De Bortoli et al. (2021); Kingma et al. (2021). Excellent performance has been shown in other domains, such audio (Chen et al., 2021a; Kong et al., 2021; Chen et al., 2021bb. Recent ideas introduced for diffusion models also include training diffusion models different levels of resolution and cascading them together to improve the model performance (Dhariwal& & Nichol, 2021; Saharia et al., 2021; Ho et al., 2022; Ramesh et al., 2022: Saharia al., 2022). The difference is that we consider the ansolution-increaing model instead of a gerformance-boosting addition, and all computational steps increase the resolution.
basis of our
Explicitly utilizing the hierarchy of resolutions in natural images has resulted in improved per- formance, e.g., by training a stack of upsampling GAN layers that join to create a single image generator (Denton et al., 2015). Other famous examples are the progressive GAN (Karras al., 2018) and later the StyleGAN architectures (Karras et al., 2019; 2020; 2021; Sauer et al., 2022). Resolution based hierarchies in VAES have brought them to rival GANs and other ofate-of-the-eee models on different benchmarks (Razavi et al., 2019; Vahdat & Kautz, 2020; Child, 2021). Aside from these works, the architectures of other standard GANs and VAFs have been such that they start from low-resolution feature maps and increase the feature map resolution through upsampling layers.
Multi-scale ideas in the context of autoregressive models have also been proposed, starting with the multi-scale pixeIRNN model (van den Oord et al., 2016). Possibly the closest one to model is the work by Reed et al. (2017), where the authors factorize the joint distribution as a subsampling pyramid such that generation starts from pixelated image and progresses towards high- regh-resolution version. Menick & Kalchbrenner (2019) suggest a similar method based on resolution and bit-depth Previously, upscaling.Previously generative models, including diffusion models, have been utilized for image
upscaling.
8

---

Published as a conference paper at ICLR 2023
deblurring (Kupyn et al., 2018: 2019; Asim et al., 2020; Whang et al., 2021), super-resolution (Ledig
et al., 2017; Sujjadi et al., 2017; Dahl et al., 2017; Parmar et al., 2018: Chen et al., 2018; Saharia et al., 2021; Chung et al., 2022), and other types of inverse problems (Kawar et al., 2021; Chung et al., 2021; Jalal et al., 2021; Song et al., 2021e; Chung & Ye, 2022: Kawar et al., 2022). While our model aaauurrnng/uppe-rssllttti
the main difference to these works is that instead of using pre-existing generative model to solve the inverse problem, we do the exact opposite and create new generative model that directly reverses the heat equation with a simple MSE loss. Thus, our goal is not to do, e.g., deblurring in itself, but to do unconditional generative modelling.
effectively performs
Parallel work. Concurrently, Lee et al. (2022) incorporate Gaussian blur into standard diffusion mod- els. with a difference being that their work generalises the standard diffusion framework to include blur along with increasing noise, while we foeus on creating a generative model that explicitly reverses the heat equation (blur process), and step out of the standard Markovian forward framework in the process. Another concurrent work is Daras et al. (2022), where the authors derive generalised score-matching objective that allows incorporating Gaussian blur into the model training and sampling. We view these works as complementary: Lee et al. (2022) and Daras et al. (2022) show that it is possible to improve diffusion model performance by applying blur, whereas a blurring process by proposing a model that generates images using deblurring in a highly explicit way. Work in combining the inductive biases provided by our model with the flexibility of standard diffusion could be a valuable direction for future research. Bansal et al. (2022) consider deterministic blurring and other operations as cold diffusions, and show an intriguing result that method similar to the standard diffusion model training or sampling routines can be effectively used to approximately invert different deterministic operations deterministically, that is, to perform a type of conditional generation. They also propose to use a Gaussian mixture model prior for unconditional generation. The main difference is that their method maps given blurry image deterministically to one possible solution. In contrast, our method produces distribution of images. Hoogeboom & Salimans (2022) look into bridging inverse heat dissipation and denoising diffusion to have them meet in the middle.
investigate the inductive bias brought by
DISCUSSION AND CONCLUSIONS
We have proposed a new approach for generative modelling by explicitly reversing the heat equation, with the goal of exploring inductive biases in diffusion-like models. An intriguing point of view on the model is that simply alternating a type of regularized deblurring and adding noise results in generative model, without much need for hyperparameter tuning. We showed useful properties such as smooth interpolation, latent disentanglement, and data efficiency, highlighting the potential of the idea. We believe that our work a first step in this direction, and that our results will allow future rescarchers to better reason about inductive biases in related generative models.
Potential future directions include more research into the probabilistic model formulation and its statistical properties, which could allow to reason about how to improve the model. Based on the experiments, it appears that the inductive bias of the generative process effectively regularises the model compared to diffusion models. While IHDM seems set smoothness prior images, diffusion models are free to even overfit slightly to parts of the data distribution if necessary. Ways to loosen this regularisation could be fruitful direction of research. is also possible that the used U-Net architecture has been optimized for standard diffusion models and is not idcally suited to our method. Research into the neural network could be valuable in improving the model.
The central idea here is also generalizable to other domains whenever a natural gaarse-rainnin operator exists on the data. For instance, the heat equation can be defined straightforwardly on ID audio data and on graphs, one could use the graph Laplacian to define a heat dissipation process based on Newton's 's law of cooling. This would also be in spirit with the geometric deep learning frame- work (Bronstein et al., 2017), where the Laplacian operator also plays prominent role. Finally, a broader point is that our work opens up the potential for designing other types of generative sequences. While our heat dissipation process is arguably natural for images, others can be considered.
9

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
We acknowledge funding from the Academy of Finland (334600, 339730, 324345) and the computa- tional resources provided by the Aalto Science-IT project and CSC IT Center for Science, Finland. We thank Paul Chang, Riccardo Mereu, Zheyang Shen, Valerii Iakovlev, Pashupati Hedge, and Ella Tamir for useful comments, and Shreyas Padhy for inspiring the data-efficiency experiment. also thank Emiel Hoogeboom for discussions towards the end of the project.
10

---

Published as a conference paper at ICLR 2023
REFERENCES
Muhammad Asim, Fahad Shamshad, and Ali Ahmed. Blind image deconvolution using deep generative priors. IEEE Transactions on Computational Imaging, 6:1493-1506, 2020.
Jean Babaud, Andrew P Witkin, Michel Baudin, and Richard O Duda. Uniqueness of the Gaussian kernel for scale-space filtering. Transactions on Pattern Analysis and Machine Intelligence, pp. 26-33, 1986.
Arpit Bansal, Eitan Borgnia, Hong-Min Chu, Jie S Li, Hamid Kazemi, Furong Huang, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Cold diffusion: Inverting arbitrary image transforms without noise. arXiv preprint arXiv:2208.05 09392, 2022.
Kristian Bredies and Dirk Lorenz. Mathematical Image Processing. Springer, 2018.
Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geometric deep learning: Going beyond Euclidean data. IEEE Signal Processing Magazine, 34(4):18-42, 2017.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron Weiss, Mohammad Norouzi, and William Chan. Wave- grad: Estimating gradients for waveform generation. In International Conference on Learning Representations (ICLR), 2021a.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J. Weiss, Mohammad Norouzi, Najim Dehak, and William Chan. WaveGrad 2: Iterative refinement for text-to-speech synthesis. In Proceedings of Interspeech, pp. 3765-3769, 2021b.
Yu Chen, Ying Tai, Xiaoming Liu, Chunhua Shen, and Jian Yang. Fsrnet: End-to-end learning face super-resolution with facial priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 2492-2501, 2018.
Rewon Child. Very deep VAEs generalize autoregressive models and can outperform them on images. In International Conference on Learning Representations (ICLR), 2021.
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and lung-Woo Ha. StarGAN v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8188-8197, 2020.
Hyungjin Chung and Jong Chul Ye. Score-based diffusion models for accelerated MRI. Medical Image Analysis, pp. 102479, 2022.
Hyungjin Chung, Byeongsu Sim, and Jong Chul Ye. Come--loser-difiie-faasee Accelerating conditional diffusion models for inverse problems through stochastic contraction. arXiv preprint arXiv:2/ 2021.
Hyungjin Chung, Eun Sun Lee, and Jong Chul Ye. MR image denoising and super-resolution using regularized reverse difffusion. arXiv preprint arXiv:2203.12621, 2022.
Giuseppe Da Prato and Jerzy Zabezyk. Stochustic Equations in Infinite Dimensions, volume 45 of Encyclopedia of Mathematics and its Applications. Cambridge University Press, 1992.
Ryan Dahl, Mohammad Norouzi, and Jonathon Shlens. Pixel recursive super resolution. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 5439-5448, 2017.
Giannis Daras, Mauricio Delbracio, Hossein Talebi, Alexandros G Dimakis, and Peyman Milanfar. Soft diffusion: Score matching for general corruptions. arXiv preprint 2xXiv:2009005442 2022.
Valentin De Bortoli, James Thornton, Jeremy Heng, and Amaud Doucet. Diffusion schrüdinger bridge with applications to score-based generative modeling. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pp. 17695-17709. Curran Associates, Inc., 2021.
Emily L Denton, Soumith Chintala. Arthur Szlam, and Rob Fergus. Deep generative image models using a Laplacian pyramid of adversarial networks. In Advances in Neural Information Processing Systems (NeurIPS), volume 28, pp. 1486-1494. Curran Associates, Inc., 2015.
11

---

Published as a conference paper at ICLR 2023
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pp. 8780-8794. Curran Associates, Inc., 2021.
Ian Goodfellow, Jean Pouget-Abadics Mehdi Mirza, Bing Xu, David Warde-Farles Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems (NIPS), volume 27, pp. 2672-2680. Curran Associates, Inc., 2014.
John R Hershey and Peder A Olsen. Approximating the Kullback Leibler divergence between gaussian mixture models. In 2007 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), volume 4, pp. IV-317. IEEE, 2007.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. Advances in Neural Information Processing Systems (NeurIPS), 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pp. 6840-6851 Curran Associates, Inc., 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23(47):1-33, 2022.
Emiel Hoogeboom and Tim Salimans. Blurring diffusion models. arXiv preprint arX5v:2209.05557. 2022.
Chin-Wei Huang, Jac Hyun Lim, and Aaron C Courville. A variational perspective on diffusion- generative models and score matching. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pp. 22863-2222 22876. Curran Associates, Inc., 2021.
Aapo Hyvärinen, Jarmo Hurri, and Patrick O Hoyer. Natural Image Statistics: A Probabilistic Approach to Early Computational Vision, volume 39. Springer Science & Business Media, 2009.
T Tijima. Observation theory of two-dimensional visual patterns. Technical Group on Automata and Automatic Control, IECE, Japan, 1962.
Ajil Jalal, Marius Arvinte, Giannis Daras, Eric Price, Alexandros G Dimakis, and Jon Tamir. Ro- bust compressed sensing MRI with deep generative priors. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pp. 14938-14954. Curran Associates, Inc., 2021.
Jari Kaipio and Erkki Somersalo. Statistical and Computational Inverse Problems. Springer Science & Business Media, 2006.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In International Conference Learning Representations (ICLR), 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4401-4410, 2019.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8110-8119, 2020.
Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pp. 852-863, Curran Associates, Inc., 2021.
Bahjat Kawar, Gregory Vaksman, and Michael Elad. Snips: tically. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pp. 21757- 21769. Curran Associates, Inc., 2021.
Solving noisy inverse problems stochas-
12

---

Published as a conference paper at ICLR 2023
Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. arXiv preprint arXi::2201.1199. 2022.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pp. 21696-21707. Curran Associates, Inc., 2021.
Jan J Koenderink. The structure of images. Biological Cybernetics, 5065):63--770, 1984.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations (ICLR), 2021.
Karsten Kreis, Ruiqi Gao, and Arash Vahdat. Tutorial on denoising diffusion-based generative modeling: Foundations and applications. CVPR, 2022.
Alex Krizhevsky, Learning Multiple Layers of Features from Tiny Images. MSc thesis, University of Toronto, Canada, 2009.
Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych, Dmytro Mishkin, and Jifí Matas. DeblurGAN:
Blind motion deblurring using conditional adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1183-8192, 2018.
Orest Kupyn, Tetiana Martyniuk, Junru Wu, and Zhangyang Wang. DeblurGAN-v2: Deblurring (orders-of- -faenitu-- faster and better. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 8878-8887, 2019.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to 1998.
document recognition. Proceedings of the IEEE,
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, 7ehan Wang, et al. Photo-realistic single image super-resolution using generative adversarial network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4681 1-469,, 2017.
Sangyun Lee, Hyungjin Chung. Jachyeon Kim, and Jong Chul Ye. Progressive deblurring of diffusion models for coarse-to-fine image synthesis. arXiv preprint urXiv:2207.11 2022.
Tony Lindeberg. A computational theory of visual receptive fields. Biological Cybernetics, 107(6): 589-635, 2013a.
Tony Lindeberg. Invariance of visual operations at the level of receptive fields. PLoS One, 8(7): c66990, 2013b.
Jacob Menick and Nal Kalchbrenner. Generating high fidelity images with subscale pixel networks and multidimensional upscaling. In International Conference Learning Representations (ICLR), 2019.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In Proceedings of the 38th International Conference on Machine Learning (ICML), volume 139 of Proceedings of Machine Learning Research, pp. 8162-8171. PMLR, 2021.
Gaurav Parmar, Richard Zhang, and Jun- Yan Zhu. On aliased resizing and surprising subtleties in gan evaluation. In CVPR, 2022.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In Proceedings of the 35th International Conference on Machine Learning (ICML), volume 80 of Proceedings of Machine Learning Research, pp. 4052-4061 PMLR, 2018
Silvia L Pintea, Nergis Tömen, Stanley F Goes, Marco Loog, and Jan C van Gemert. Resolution leamn ing in deep convolutional networks using scale-space theory. Transactions on Image Processing, 30:8342-8353, 2021.
13

---

Published as a conference paper at ICLR 2023
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
Ali Razavi, Aaron den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with VQ-VAE-2. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, pp. 14866-14876. Curran Associates, Inc., 2019.
Scott E. Reed, Aäron van den Oord, Nal Kalchbrenner, Sergio Gomez Colmenarejo, Ziyu Wang, Yu- tian Chen, Dan Belov, and Nando de Freitas. Parallel multiscale autoregressive density estimation. In Proceedings of the 34th International Conference on Machine Learning (ICML), volume 70 of Proceedings of Machine Learning Research, pp. 2912-2921. PMLR, 2017.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer- assisted Intervention, pp. 234 241. Springer, 2015.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. arXiv:2104.07636 2021.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic -ik-te-- diffusion models with deep language understanding. arXiv preprint 1rX11:2205.11487, 2022.
Mehdi SM Sajjadi, Bernhard Scholkopf, and Michael Hirsch. Enhancenet: Single image super- resolution through automated texture synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4491-4500, 2017.
Axel Sauer, Katja Schwarz, and Andreas Geiger. StyleGAN-XL: Scaling StyleGAN to large diverse datasets. arXiv preprint arXiv:2202.00275. 2022.
Jaseha Sick-Dic,,,ein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Proceedings of the 32nd International Confer- ence on Machine Learning (ICML), volume 37 Proceedings of Machine Learning Research, pp. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations (ICLR), 2021a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances Neural Information Processing Systems (NeurIPS), volume 32, pp. 11918-1 1930. Curran Associates, Inc., 2019.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models.
In Advances Neural Information Processing Systems (NeurIPS), volume 33, pp. 12438-12448 Curran Associates, Inc., 2020.
Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. In Advances in Neural Information Processing Systems (NeurlPS). volume 34, pp. 1415-1428. Curran Associates, Inc., 2021b.
Yang Song, Liyue Shen, Lei Xing, and Stefano Ermon. Solving inverse problems in medical imaging with scsre-based generative models. arXiv preprint arXiv:21 2021c.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations (ICLR), 2021d.
Arash Vahdat and Jan Kautz. NVAE: A deep hierarchical variational autoencoder. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pp. 19667-19679, Curran Associates, Inc., 2020.
14

---

Published as a conference paper at ICLR 2023
Aäron den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In Proceedings of the 33nd International Conference on Machine Learning (ICML), volume 48 of Proceedings of Machine Learning Research, pp. 1747-1756. PMLR, 2016.
Arjen van der Schaaf and JH Hateren. Modelling the power spectra of natural images: Statistics and information. Vision Research, 36(17):2759-2770, 1996.
Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G Dimakis, and Peyman Milanfar. Deblurring via stochastic refinement. arXiv preprint urXiv:2112.02477 2021.
Andrew P Witkin. Scale-space filtering. In Readings in Computer Vision, pp. 329-332. Elsevier, 1987.
Daniel Worrall and Max Welling. Deep scale-spaces: Equivariance scale. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, 7366-7378. Curran Associates, Inc., 2019.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. LSUN: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint urXiv::506. 03365, 2015.
15

---

Published as a conference paper at ICLR 2023
Appendix
Table of Contents
Derivation of Method Details
17
A.1 Numerical Solution of the Heat Equation with Neumann Boundary Conditions
Using Discrete Cosine Transform
A.2 Derivation the Variational Lower Bound
18
A.3 Variational Upper Bound Lx
20
A.4 Limit of K and Convergence to Denoising Score Matching and Langevin Dynamics
20
A.5
Analysis for the Power Spectral Density in the Diftusion Forward Process
21
B Experiment Details
22
B.I Neural Network Architecture and Hyperparameters..
22
B.2
Training Time Computational Resources
23
Score Calculation
23
B.4 Hyperparameters Related the New Generative Process
23
B.5 Schedule on 4
23
B.6 Calculation of Power Spectral Densities
24
C Additional Experiments
26
C.I Robustness to a and 8
26
Data Efficiency/ Few-Shot Generalization
26
C33 Optimal with Respect Marginal Log- Likelihood vs.
27
C.4 Evaluation of the Prior in Terms of Overlap with the Test Set
27
C.5 The Importance of Non-Zero Training Noise
27
C.6 Comparison with Gaussian Blur Implemented with Convolutional Filter
29
D Sample Visualizations
30
D.1 Additional Samples
31
Interpolations
42
D.3 Effect of Sampling Noise 5
45
D.4 Disentanglement of Colour and Shape Fixing the Sampling Noise
51
D.5 Nearest Neighbours Sampled Images
52
16

---

Published as a conference paper at ICLR 2023
A DERIVATION OF METHOD DETAILS
A.1 NUMERICAL SOLUTION OF THE HEAT EQUATION WITH NEUMANN BOUNDARY CONDITIONS USING THE DISCRETE COSINE TRANSFORM
We expand the presentation of the approach taken in Sec. 2 and provide some further details on the derivation that highlights various appealing aspects of our approach. The material here is not novel, but is included for completeness.
We use the partial differential (PDE) model
Ot u( v.t) = Au(x, y.t,,
(12)
where u R22 x X + R is the idealized, continuous 2D plane of one channel of the image, and A 2 is the Laplace operator. Rather than discretizing the operator by finite-difference scheme (see Eq. (11) for discussion in the main paper), we take an alternative approach where we solve the heat equation in the function space by projecting the problem onto the eigenbasis of the operator. The benefits are good numerical accuracy and scalability to large images. The workflow as follows: (i) Rewrite the PDE an evolution equation. (ii) Choose boundary conditions (Neumann, i.e, the image "averages out' as / + oo). (ii)) Solve the associated eigenvalue problem (eigenbasis of the operator) which in this case results in cosine basis. (iv) The image
regular grid, so can use the Discrete Cosine Transform (DCT) for projection. This also means that there is a cut-off frequency for the problem that makes finite-dimensional.
Evolution equation Following the steps written out above, the PDE model in Eq. (12) can be formally written in evolution equation form as
u u(x, y. t) = F(t) u(x, y. to):
(13)
where F(t) exp[(t- (o) A] is an evolution operator given in terms of the operator exponential function (see, e.g., Da Prato & Zabczyk, 1992). It is worth noting that instead of the Laplacian could also consider more general dissipation processes.
operators to describe more complicated
Choice of boundary conditions We choose to Neumann boundary conditions (Qu/ae Bu/0y 0) with zero-derivatives at boundaries the image bounding box. This means that as t - oo, the image will be entirely averaged out to the mean of the original pixel values in the image. Formally:
a
a
(0, v,t)
u y,
dr a
Oz
Boundary conditions
(14)
Dy u(x 0,t) 0
(xx H,t) =
Dy
u(z, y, to) f(z v).
Initial condition,
(15)
where W is the width of the image and H is the height. We could choose some other boundary conditions as well, in many cases without loss of generality.
Solve associated eigenvalue problem For the (negative) Laplace operator, which positive definite and Hermitian, the solutions to the eigenvalue problem
-Ø (z. v) (x v).
for (I. v) Ω
(16)
(x, v) (0 v) O,
for (x, an,
(17)
0r
Oy
yields orthonormal eigenfunctions (-).) with respect to the associated inner product, meaning that the corresponding operator is diagonalizable. If Ω is rectangular domain and consider the problem in Cartesian coordinates, the eigenbasis (solution to the eigenvalue problem above under the boundary conditions) turns out to be (separable) cosine basis:
TTE W
TTT
Φnm (x, ~ cos
COS
(18)
H
n²
m
2
Am.
2
(19)
W2
H2
17

---

Published as a conference paper at ICLR 2023
Writing out the result of the evolution operator F(t) expl(t to) A] explicitly,
Anme ( 12-00-t cos
Whd W
any
u( v,
(20)
H
===0 m=0
which is a Fourier series where An,m are the coefficients from projecting the initial state u(x, y, to) to the eigenfunctions (I, v).
Leverage the regularity of the image we consider the pixels in the image to samples from the underlying continuous surface u(z, y) that lic regular grid, the projection onto this basis can be done with the discrete cosine transform (ū = yuu u = (u)(u)). As the observed iiiit-resoluttin image has natural cut-off frequency (Nyquist limit due to 'distance' between pixel centers), the projection onto and from the cosine basis can be done with perfect accuracy due to the Nyquist- Shannon sampling theorem. Formally, a finiteddimensional Laplace operator can be written out the eigendecomposition A A VAVT, where A is diagonal matrix containing the negative squared frequencies ( W + H' and is the discrete cosine transform projection matrix. The evolution equation, and our numerical solution to the heat equation, can thus be described by the finiteddimensional evolution model (in image <> Fourier space):
m
u(t) = F(t) ((0) exp (VAV t) u(0)= V exp(At)V u(0) + ū(t) exp(At)ū(O), (21)
where F(I) E N is the transition model and u(0) the initial state. F(I) is not expanded out in practice, but instead we use the DCT and inverse DCT, which are O(N log N) operations. As A is diagonal, the Fourier-space model is fast evaluate. The algorithm in practice is summarized Python snippet in Alg. 3.
Algorithm 3 Python code for calculating the forward process
import numpy np from scipy. fffpack import dct, idct def heat_eq_forwwaddtu, t):
# Assuming the image u is an (KxK) numpy array K = shapeel-11
freqs frequencies_squareed [r, +*2 freqs [None, **2
Linspace (0,K-1,K)/E
u_proj det (u, axis-0, norm=/ortha")
u_proj - dct (u_proj, axis-l, norm-' norm='ortho! ortho')
u projj = np.exp( frequencies_squaee squared t) proj
u reconstucted idet (u_proj, axis=0, norm=' ortho')
12 u. reconstucted idst (u_ tu_reconstucted, axis=1, norm=' ortho')
13 return u_rcconstucted
A.2 DERIVATION OF THE VARIATIONAL LOWER BOUND
This section contains derivation for the variational bound in more detail than what presented the main text. It is mainly intended for readers not already familiar with the diffusion model mathematics. Recall the definitions for the generative Markov chain and the inference distribution:
K
K
Reverse process Generative model
Po (o = p( (u))
II
po (u (1)) = p(ux) II (u po k), 12)
1=
K Nu, F(tx)uo, II).
Forward process / Inference distribution q(u1K (u)
I
g(u
un)
k=1
A=1
18

---

Published as a conference paper at ICLR 2023
Taking the negative of the evidence lower bound, get
Po (P(():k)
- log (un) & E g(uik uu)
log
(22)
Quu:KK
un)
p ((K)
I
Pel P uR)
Bgguu ub)
log
(23)
I (uu Up)
P6 (u(k
K
(u
B (m))
log
log
log pu (1)
(24)
quu Ub)
[Ug-
Uu)
k=
log pa (ux)
po Ug-1
(u))
= (u))
log
log pe up
[1))
(25)
quu uu)
q(u=-1
uo)
= uo) / q(ux (n)
g(ux Uo)
duk
po(ux)
K
Boluquky((e11111 mo) / q(uu=1
g(us-1 un)
un) log
du
Po Pu up
Eg (b) pogpo un)
(26)
o - (w) DxI.((uxx un) p(ux)]
K
E (1 DuL uo) p(u 1 [ux]]
km
E [ un) log u1).
(27)
un) || [ uk)]. the time steps k - 1 have already been integrated over and the term is dependent only on u, when Ug is constant. The term D uo) p(ux)) is not dependent on any of the variables and log pe (uo μ,) only on u1. We can proceed in two ways: Either (i) add dummy integrals over the already marginalized over dimensions get Eq. (7) in
In the KL terms D u
the main text, or (ūi) marginalize out all redundant expectation values to explicitly get the final loss function. We first look (i):
E Eg(unx-1 uo) / aau'k uo)du's D [ uu) p(u))
Not dependent u'g
B (oo) / qquuk_1 un)du' / D un)) p(u
[1k]]
Not dependent U
[u mn) Jog /uj)
(28)
Eglunn x (o)
DKL uo) p[]x]] + > DKulg(u:-11 uo) pe pu uR]]
T.K
2=
log po ( in)
(29)
Lo
which is the formula presented in the main text. On path (ii), marginalizing redundant integrals get the loss function, we get
D up)) pp(u))]++ +
Σ Eyglus (u)
DKL(q(uk-1 (o)
p [x)]]
Eggu, (u)) log (o (n))
Lx
2
Tn-1
Lo
(30)
19

---

Published as a conference paper at ICLR 2023
The first term is constant. As all distributions are defined as Gaussians with diagonal covariance matrices, we get
02 N N + 82 yus, (uz, mo)
E (u uo) [ = 2 82
| k) - F(tx-1)uu
2N log
(31)
Eg(u, uo)]Lo) Beglu / unn) - loggo (u ui)]
(32)
1 Bo (ou)
|| - woll + +Nlogg(vv log(ōv (
(33)
262
Taking a Monte Carlo estimate of the expectations by sampling once from uy or uj and passing through the neural network pea arrive at our final loss function.
A.3 VARIATIONAL UPPER BOUND ON LK
In the main text, we noted that Lx = Dki. (xxl[q(xx [p))p((x is not trivial to evaluate if we define p(ux) be kernel density estimator over the training set, since evaluation of p(ux) is heavy due to the large amount of components. For each spatial location in the integral, we would need re-evaluate its distance to all blurred training data points, and this is highly inefficient. We can, however, provide a further variational upper bound, similarly to (Hershey & Olsen, 2007):
[x [u)) [[u]] = Eq(ux mo)llog quu [u)) -loo (u
(34)
1
E Eggux uo)
[log (ux [uo) logg Nr
Σ N(ux uik, o²)
(35)
where Ny is the training set size and u'x is an example from the training set. The first term is simply the negative entropy of Gaussian. For the second term, introduce variational parameters di
N such that Σ Φi 1:
Nr log N 1=1
1
N(ux [ux: 0²)
-Bggux un) flog
N(ux uuk 6²)] =-Bouuu
-Bq(uk uu)
(36)
N
1i
[N(ux 02)
Eulux uo) Σ de log
(37)
Nr
Σ di [Bylux MmjllogNNuus uix 83)] log NT
(38)
= Σ Φi [-H (a(ux uo).NNuk (d(uuxuo)).N(uuz uig 82)) - log Ny - log dx)
(39)
Note that the cross-entropy between Gaussians has an analytical formula that can be precomputed for all training set - test set sample pairs. Minimizing the upper bound w.r.t. di. we get
e- DDulouu up) M (8))
(40)
E, Dal(s(ux (an (ux (2))
Now, we just need to evaluate the KI. divergences between data points in the training set and the test set, and then we can use those to calculate the upper bound efficiently, without need to integrate over and evaluate p(ux) multiple times in a very high-dimensional space. Calculation of the KL divergences amounts to calculating the distances of blurry data points in the training and test sets.
While the calculations are straightforward to implement practice, one needs to be a bit careful when doing the computations numerically. In particular, we can use the log-sum-exp trick to estimate log Φi in a numerically stable way, and Φ; is obtained from the log di values.
A.4 LIMIT OF K - oo AND CONVERGENCE TO DENOISING SCORE MATCHING AND LANGEVIN DYNAMICS
In this section, we show in detail the result that in the limit + oo, the loss function L₁. converges to the denoising score matching loss function and the sampling process becomes equivalent Langevin
20

---

Published as a conference paper at ICLR 2023
dynamics sampling. Let's start with our loss for the k:th level:
D un) (ux-u uz)] ~ Betu. uo) Ho (uz : k) - F(f2-1) uollz
(41)
Bu un) fouu k) (F((x-1) ug u
(42)
where denotes that overall multiplicative and additive constants have been removed. Now note that if we let K and redefine (uu: k) (ue, k)o², then F(tx-1) + F(tx) and Eq. (42)
will approach
Bygue uo) [o k) (F(4k-1) U1 -0:00
u
(43)
+ Begu (u)) Ilf k) (F(t&) un -u:)l
(44)
Bgg(u uo) [ [uk k)o² (F(t+) un uz)l
(45)
= Eginn (um) fl (ug, F(4x) un
r² 112.
(46)
which is equivalent to the denoising score matching loss with a Gaussian kernel of variance o² on a data set blurred out with the matrix F(tx), up arbitrary scaling. Now, optimizing this results in an estimate of the score V log pk (u), where ps (u) is the kernel density estimate of the data at level k, that fo(u, k) R V log pk (u)o². We can then construct the Langevin SDE that has stationary distribution p&(u)
du V (u)dt + 22WW
(47)
where (t)) is a standard Brownian motion. Note that here we overload notation slightly: t is the time dimension of the SDE, not the time dimension of the heat equation. An Maryyaaa discretized step along the SDE yields
Au Vlogpa V2Alz, Z
(48)
~ At(u) + V2Atz
(49)
fo(u)
o² Al V2A/z.
(50)
Now if we choose the step size be At 0², we get the following update
Au (u) 22oz where ~ N(O
(51)
which is exactly equivalent to sampling update step with 8 = V2o R 1.4lo.
Note that this result is not meant preliminary statistical analysis of our method in an asymptotic limit, helping reasoning with the model and showing theoretical connections to other ideas in the diffusion model literature. In particular, do not in practice simply perform this type of Langevin dynamics with a distribution pk (u) and slowly shift it towards Po (u), but instead take directed steps backward in the heat equation. Thus, the zs /2a also simply provides an intuitive rough scale for ö, and is not necessarily the value we want use in practice.
derivation for the model itself, but instead to provide a
A.5 ANALYSIS FOR THE POWER SPECTRAL. DENSITY IN THE DIFFUSION FORWARD PROCESS
Here we analyse explicitly the PSD behaviour in the diffusion forward process. In particular, the expected value of the PSD of a noised image equals to the PSD of the original image added with the noise variance. We define the PSD for individual frequency components PSD(u), Ivv'u] where v, is the projection vector to the 1th frequency in the DCT/DFT basis. These individual components compose the PSD of the image, PSD(u).
21

---

Published as a conference paper at ICLR 2023
Without loss of generality, we consider diffusion process where do not scale the original image u and simply add noise e with covariance I The expected PSD of the original image plus noise is:
B [PS((u + e)d = BEelv (u =)1=
(52)
[ee[(v u)² + ( + (v, e)a]
(53)
Iv.'u² + 2(v, u)(v, E.(e) + Bellv
(54)
PSD(u): + v. Eg|es Il:
(55)
PSD(()) + v. ?
(56)
= -PSD(u),+ +
(57)
meaning that the PSD of the image and the variance of the noise add together. This gives a formal explanation for the idea that isotropic noise effectively drowns out the frequency components in the data with lower PSD than the variance of the noise. Note that in the 1D plot Fig. 4, the equal frequency components have been averaged out, and this is visualised in more detail in App. B.6. The result does not change in that case aside from replacing PSD(u); with the mean of multiple PSD values that correspond to equal frequencies.
As further curiosity, also point out that a single PSD component is distributed as the sum of normally distributed random variable and chi-squared variable:
v. (E + u)/² (v. u)² + + u) (vIEE) + (VTTE)
(58)
~N(0077)
Here we the fact that 1D orthonormal projection of isotropic zero-mean Gaussian random variable is a simple ID Gaussian.
B EXPERIMENT DETAILS
B.1 NEURAL NETWORK ARCHITECTURE AND HYPERPARAMETERS
Similarly to many recent works on diffusion models (Ho et al., 2020; Song et al., 2021d; Nichol & Dhariwal, 2021; Dhariwal Nichol, 2021), we use a U-Net architecture for estimating the transitions, and in particular parametrize μ4 [ug, k) U1 u fo k), where fe fe(u,, k) corresponds to the U-Net, as explained in the main text. Following (Dhariwal & Nichol, 2021), we use self-attention layers at multiple low-resolution feature maps. Otherwise, the architecture follows the one in (Ho et al., 2020), and we do not, e.g., use multiple attention heads or adaptive group normalization (Dhariwal & Nichol, 2021). We use two residual blocks per resolution layer in other models than the MNIST and CIFAR-10 models, where we use 4. We use 128 base channels, GroupNorm layers in the residual blocks and dropout rate of 0.1, applied in each residual block. The downsampling steps in the U-Net were done with average pooling and upsampling steps with nearest- neighbuu interpolation. The time step information is included using a sinusoidal embedding that is added the feature maps in each residual block.
Table I lists the values of other hyperparameters that we used on different data sets. They are mostly based on the recent work with diffusion models (Ho et al., 2020; Nichol & Dhariwal, 2021; Dhariwal & Nichol, 2021), and have not been optimized over. For the lower resolution data sets, we used a higher learning rate of 10-1 or 2- 10-4, and a lower learning rate of 2- 10 for the higher resolutions for added stability, although we did not sweep over these values. An EMA rate 0.9999 was also used instead of 0.999 for 256 x 256 images, instead of the 0.999 that was used for other data sets. We used the Adam optimizer with default hyperparameters 31 = 0.9, 33 = 0.999 and ε = 10 8 We also use gradient norm clipping with rate 1.0 and learning rate warm up by linearly interpolating it from 0 to the desired learning during the first 5000 steps. We do not add noise on the final step of the generative process, that cannot increase the output quality. Other details included in the code release.
We used random horizontal flips AFHQ, and data augmentation on the other data sets.
22

---

Published as a conference paper at ICLR 2023
Table
Neural network hyperparameters.
Layer multipliers
Base channels
Learning rate
Self-attention resolutions
Batch
# Res-
Data
Resolution
EMA
size blocks
MNIST
28x28
(1,2,2)
128
le-4
7x7
0,999
128
4
CIFAR-10
32x32
2,,2)
2e-4
8x8, 8x8, 4xx4
0,999
128
FFH
256
(1,2,3,4.5
128
2e-5
64x64, 32x32, 16x
0,9999
32
2
FFHQ
128x
(,,2,3,4,5)
128
2e-5
32x32, 16x 16, 8x8
0.999
32
2
AFHQ
x
3,,2,
128
2c-5
64x64, 32, 16x
0.9999
32
2
AFHQ
64x64
3,,2,
128
lc-4
16x 8x8
0,999
128
LSUN CHURCHES
128x 128
3,.2.3.4.5)
128
2e-5
32x32, 16x 16x 8
0.999
32
2
B.2 TRAINING TIME AND COMPUTATIONAL RESOURCES
We NVIDIA A100 GPUs for the experiments, with GPUs for 256x256 resolution models and one GPU for all others. We use the Pytorch automatic mixed precision functionality for a higher Per-GPU batch size. On CIFAR-10, steps. On the 256x256 images, 100,000 iterations takes about 40 hours, and we used 800,000 iterations on FFHQ and 400,000 iterations on AFHQ. On LSUN-Churches, 100,000 training steps takes 11 hours, and training was continued for one million iterations. On the smaller 64x64 AFHQ data set, we trained for 100,000 iterations, taking a total of 14 hours.
trained for 400 000 iterations, taking 7 hours per 100,000
B.3 FID SCORE CALCULATION
To calculate FID-scores
used clean-fid (Parmar et al., 2022), where the generated images and reference images are scaled to x99x299 resolution with bicubic interpolation before passing them to the InceptionVs V3 network. We used 50,000 samples to calculate the scores for all other data sets than the 256x256 256 sets, where we used 10,000 samples. Using lower amount of samples results in the values being slightly overestimated, which is not much of issue since the FID scores very close to state-of-the-art values. We used the training set to calculate the reference statistics on L.SUN-CHURCHIE and CIFAR 10, and the entire data sets for FFHQ and AFHQ
B.4 HYPERPARAMETSR RELATED TO THE NEW GENERATIVE PROCESS
The different hyperparameters related to the new generative process are listed in Table 2. Early on during experimentation, we noticed that a 0.01 for the training noise seems to work well, and use that for all experiments unless mentioned otherwise. We have not done an extensive study on the optimal value. The number of iteration steps K was set to 200 for most experiments, except for the LSUN CHURCHES data set, where noted that K = 400 increased sample quality slightly. Otherwise, in contrast to findings on diffusion models (Ho et al., 2020), we found in early experimentation that increasing the number of steps well above 200 did not seem result in trivial improvements in sample quality. On MNIST, we used 100 steps. Although a of 0.0125 = 1.25 o seemed work well sweeping over the FIDs obtained with different values (results visualized in App. C.3). On AFHQ 256x256, we set it to 0.01275 because visual inspection showed slightly improved results. The maximal effective blurring length-scale. 0B max. was set to half the size of the images in most experiments, although on MNIST and CIFAR- we used slightly higher values. We noticed during early experimentation that moving B,maa from the entire image width to half the width resulted in better image quality, although the information content present in the half-blurred image intuitively not much different from the information content in the fully averaged out image. We also study how a max affects the value of the prior overlap term Dxr. [q [ux un) p(())] ]] App. C.4 on CIFAR-10 10, providing justification to not having mama entire image width.
default value on all data sets, we tuned it 0.01325 on CIFAR-10 by
B.5 SCHEDULE ON 4
In all experiments, we used a logarithmic spacing for the time steps ty. where tx = o m2x/2 and o min/2 0.5"/2, corresponding to sub-pixel-size blurring. Effective averaging sizes k K-k + log B,x Kー $11) B,minn other levels were then interpolated with B,A exp(log B,min K-T
23

---

Published as a conference paper at ICLR 2023
Table 2: Hyperparameters related the generative process.
Data
Resolution
JB..max
MNIST
28x
100
20
0.01
0.0125
CIFAR-I0
32x32
200
24
0.01
0.01325
FFH
x56x256
128
0.0125
FFH
128x
200
128
0.01
0,0125
AFHQ
256
200
128
0.01
0.01275
AFHQ
64x64
200
32
0.01
0.0125
LSUN CHURCHES
128x128
64
0.0125
for an even spacing on a logarithmic axis. We can view the dB schedule in two ways: First, corresponds constant rate of resolution decrease, in the sense that "B.ttl constant. Second, an we can explicitly visualize the rate of effective dimensionality decrease by looking at the frequency components in the discrete cosine transform as we increase dg = V2f. When they pass well below the o² 0.01 line, the frequency components become indistinguishable from noise in the forward process g(ux uo) Nuux F((a) uo I²1). This is done in Fig. 9, where we see that with a logarithmic spacing on о (and tş), the amount of remaining frequencies decreases at an approximately constant rate, although slows down somewhat towards the end.
B.6 CALCULATION OF POWER SPECTRAL DENSITIES
We define the power spectral density of a frequency as the squared absolute value of the DCT coefficient for that frequency. So start by taking the 2D DCT of the image to get the 2D frequency coefficients. Then we square those values to get the corresponding PSDs. To get the 1D plots PSD) with respect frequency, e-g., in Fig. This is visualized Fig. 10 for an example image. We use the orthogonal version DCT. Aside from DCT, we could also use the discrete cosine transform (DFT).
take the average PSDDs over equal-frequency contours.
24

---

Published as a conference paper at ICLR 2023
10a
0.00
10a
Frequency coefficients
100
0.73
10a
Froquency
PSD
1,46
10
10
2.18
10
0.01
10 4
>2.91
10
10
10
10
Frequency
Effective blurring scale, dB
Figure 9: Squared frequency components in the discrete cosine transform of the example image as function of the effective blurring width dBg As the heat equation progresses, the more frequency components become indistinguishable from the training noise a. Each vertical slice on the plot the right corresponds to ID PSD, such the equal-frequency contours in the full 2D DCT.
the left. The values averages over
0.00
-27
1.1
-18
Frequency
Frequency
-9.5
log PSD
2
2
3.3
-0.6
4.4
3.4
0
2
3
Frequency
Frequency
Figure 10: Left: The partitioning of frequencies in the 2D Discrete Cosine Transform. Right: The PSD from the discrete cosine transform an example image, with the partitions. The one-dimensional power spectral densities, e.g., in Fig. 4, are obtained by averaging the PSDs along the equal-frequency contours.
25

---

Published as a conference paper at ICLR 2023
C ADDITIONAL EXPERIMENTS
C.I ROBUSTNESS TO AND 8
In this section, we empirically investigate the relationship between the parameters á and and their robustness different choices. We ran models on CIFAR-10 with o (0.005, 0.0075, 0.01, 0.0125, 0.015, 0.02) and calculated FID scores for different values, which are shown in Fig. 11. Note that (i) The model is quite robust the choice of o, as long as is not too close to zero. (ti) The parameter of interest here is 8/a instead of the absolute value of 5, and optimal FID values are obtained approximately ó 1.3 a for all choices a, while other nearby values, such as 8 = 1.25
work also. The overall pattern seems to also be that the model becomes somewhat less sensitive to the choice of 8 with increasing a. We also give example images from the different models in Fig. 12 with å 1.3 o, showing that visual differences between the samples are rather small.
140
o =0.005
a =0.0075
120
=0.01
o =0.0125
FID- 10000000
100
a =0.015
0.02
80
60
40
20
1.1
1.15
1.2
1.25
1.3
1.35
1.4
1.45
Noise scale ratio, 8/a
Figure 11: FID-scores for CIFAR-10 models trained with different values, varying as function of Note that the optimal values are reached with the same 8/a ratio, which is about 1.3. The FID scores here calculated with 10,000 samples, and are slightly overestimated compared Fig. 14. 0.0075
0.005
0.01
0.0125
= 0.015
0.02
Figure 12: Example images from the different CIFAR-10 models trained with different o and 8 1.3 a. Visual differences are qualitatively not large.
C.2 DATA EFFICIENCY / SEOT GENERALIZATION
As said in the main text, the explicit inductive biases introduced in the model allow it to effectively generalize beyond the data set even in very low data regimes. As an extreme example of this, we train
26

---

Published as a conference paper at ICLR 2023
the model and diffusion model with the first 20 MNIST digits. As shown in Fig. 13, the diffusion model either fails to produce plausible digits or completely overfits the training data. IHDM, however, generalizes to new digit shapes and does not overfit noticeably. Intuitively, the multi-resolution nature of the model provides a very explicit way to generalise on image data: The model can combine learned features on different resolution scales form new images and meaningful variation with even a few data points.
C.3 THE OPTIMAL 5 WITH RESPECT TO MARGINAL LOG-LIKELIHOON VS. FID
We point out that the optimal NLL scores with respect to ó do not correspond to the optimal 6 for FID values. We plot the negative per-sample ELBO and FID values in Fig. 14, where we see that the lowest NLL achieved close to a 0.01, whereas the lowest FID score is got somewhere near 0.01325, which is our chosen value for 6, listed in Table 2. To get an intuition to the result, consider the Lg terms in the loss ELBO:
1 Eg D (q (o) po (us-us ux]]]
(59)
1 1 0² N N+ Eotuul uom l k)
8) σ
F F(tx-1) un
+ 2N log (60)
82
Without the MSE term in the KI. divergences LE, the optimal values for 8 is always a, can be seen by straightforward differentiation of the o² N and 2NV log terms. The inclusion of the MSE term nudges the optimal ó to a higher value, but if the MSE loss is not very high, then it does not get nudged by a lot. Changing the value of a would likely change the picture, but appears that our model is in a regime where z2 NN and 2N log 6 marginal log- likelihood maximizer, the interplay of these terms could be studied further to get to hyperparameter regime where the optimal FID values and NLL values are obtained simultaneously.
a dominate the NLL scores. To improve our model as a
C.4 EVALUATION OF THE PRIOR IN TERMS OF OVERLAP WITH THE TEST SET
Overlap between the blurred out train and test sets, measured by Lx, can be used prerequisite measure of how we can expect the model to generalize beyond the train set, since we use the training data to draw samples from the prior u(u ). If B,max O then there is almost no overlap at all and the generative process will amount to just memorization of the train set. On the other hand, if the data set it fully averaged out at the end of the forward process, then we expect that the using samples from the blurry train set as the prior to have high overlap with the blurry test set since both distributions are essentially low-dimensional. To showcase the situation with our CIFAR-IO model, plot the average Lą values the test set with respect to B,maa- We that it decreases as the maximal effective length-scale is increases, and does not change much moving from 24 to 32. Thus, there does not seem be reason to believe that increasing d max would result in much better generalization.
C.5 THE IMPORTANCE OF NON-ZERO TRAINING NOISE
As noted in the main paper, the training noise o is necessary for the model to be defined in a sensible way. Mechanistically. it also acts as a regularization parameter: Training the neural network to directly solve the exact reverse heat equation, that is, estimate the extremely ill-conditioned inverse F(tx) does not work. To showease this, Fig. 16a shows samples from a model trained on MNIST with ø = O. The produced images are essentially random patterns. Figure 16b shows the generative process and visualizes neural network input gradients during the process, both for 5 = 0, and with a non-zero amount of sampling noise. Starting from the flat prior, the image quickly blows up into a random pattern. Note that the input gradients are similar to the ones seen in Fig. 8 for our model, but the signs are the opposite. Intuitively, the model has learned a generic sharpening filter where the response of the output increases more as the image gets less blurry. This is very unstable, and small errors in the reverse steps are amplified. With a non-zero o,, the model becomes more robust and is forced to take the training data distribution into account.
27

---

Published as a conference paper at ICLR 2023
Full training set, 20 first MNIST digits
5491133 3
2
6
0 5 8 9
DDPM results
105
Validation loss
104
500
1,000
1,500
2,000
2,500
Traping steps
DDPM samples
3 7 1 6
4
8
6
5
6
9
9
6
7
9
3
3
3 4 0 σ
3
3
1
3
5
4
3
6
6 0 6 2
9 3 3 9
8
!
4 3
4 9 9
2 f
6
9 4 6
3
8
3
1
2 2 J
5
7
4
6
6
8
8
Failed samples
Overfitting
IHDM results
20
Validation loss
10
500
1. 1,000
1,500
2,000
Training steps
3 7
9
3 8
9
8
f
7
IHDM samples
3
3
3
8
3
9
P
3
J
i
f
P
3
D
3 1 0
l
/
3
4
6
9
0 16414441
3
9
>
3
P
0
d
8
3
2
b
9
3
6
P
9
3
/
97 0
3
3
J
9
P
3
7
3
0
7
3
2
3 5
8
22267
3
9
1 (
1
0
3
0
Figure 13: Experiment results on few-shot learning. Top: The entire training set that consists of the 20 first MNIST digits. Middle: The evaluation loss for denoising diffusion probabilistic model and generated samples at different points during training. The model either fails to produce plausible digits or overfits the training data. Bottom: The evaluation loss and generated samples for IHDM. The model does not overfit to the training data and is able to produce meaningful variation from just the 20 training examples.
28

---

Published as a conference paper at ICLR 2023
-10
1.5
80
Negative ELBO
60
FID value
Optimal 6
Optimal
0.5
0.7
0.8
0.9
1
1.1
1.2
1.3
Sampling noise scale, 8
-10-2
Figure 14: The per-sample negative ELBO and FID values function of our CIFAR-10 model.
-104
0.8
Lx
0.6
0.2
0
10
15
20
25
30
Maximum effective blurring width, d
Figure 15: Average values of the LK term the CIFAR- 10 test set, with respect to the maximum effective blurring width d maxx Lx effectively measures how large is the overlap between the averaged test distribution and the averaged out train distribution, which is used the prior distribution in the model.
C.6 COMPARISON WITH GAUSSIAN BLUR IMPLEMENTED WITH A CONVOLUTIONAL FILTER
We also experimented with implementing the forward heat dissipation process, or blur, using convolutional filter with sampled Gaussian blur. This is a reasonable approach as well, although somewhat computationally slower and does not expose directly the intuitions about frequency decay or the heat equation boundary conditions, as the DCT-based approach does.
To test this, trained a convolutional filter-based model on CIFAR-I0 with otherwise the same parameters as our standard CIFAR-10 one. We use a convolutional kernel size (2*N- -11--1) 11) x N 11), where N is the width and height of the image in pixels, guaranteeing that all pixels pixels with large enough blur widths. We then fill the kernel with samples from the Gaussian pdf of different blur standard deviations g. We use padding the image edges. The method achieves
affect all
29

---

Published as a conference paper at ICLR 2023
Generation
Input gradient
Generated sample
0.5
0
-0.5
Generation
Input gradient
0.012
Generated sample
0.5
0
-..5
(a) Generated samples
(h) Sample trajectory and input gradient visualization
Figure 16: Failure of the model when 0. (a) Generated samples from a model trained on MNIST with the training noise parameter = O. (b) Generative trajectory and neural network input gradients for the pixel highlighted in red. Note that the input gradients resemble the ones seen Fig. 8 for our model, but the signs are exactly the opposites. The colours are scaled to symlog scale with linear cutoff at 0.002.
FID score 22.44 as opposed 18.96 with the DCT-hased method, indicating that the DCT-hased method may have edge, although the difference is minor.
third approach could be to use a finite difference based approximation to the heat equation and use standard numerical solver to simulate the differential equation forward in time. This should work as well in principle, but the problem is that large blur widths og and thus long simulation times
t zi translate into lots of sequential computations. This means that the method would be very slow especially for blurring out larger images.
D SAMPLE VISUALIZATIONS
In this section, start by showcasing uncurated samples the different data models on in App. D.I. We then showcase the finding that interpolating the noise and starting image results in smooth interpolations in the output image in our model in App. D.2. In App. D.3 illustrate the behaviour of the å parameter in more detail, and in particular point out that 5 = 1.25 x and resolutions. In App. D.4 provide further
trained
results in good image quality across data examples of the result where the overall colour and other features of images can become disentangled our model. Finally, App. D.5, we plot example Euclidean nearest neighbours of samples from our model, showcasing that the generated samples are not just approximations of training set images.
In the interpolation results, the prior states ug are interpolated linearly. The noises are interpolated with spherical interpolation sin(Ø);ı + cos(11/2 - 6)22, where and Vi are the noise vectors and [o, J]. The reason is that when we sample two random high-dimensional standard Gaussian noise vectors, they are, with high probability, approximately orthogonal to each other, with approximately equal magnitudes. In a linear interpolation between two orthogonal, equal magnitude vectors, the magnitude of the interpolated vector will decrease half-way. In Fig. we that decreasing the magnitude of the sampling noise has a systematic qualitative effect on the results, and really we want the magnitude remain constant during interpolation. This is achieved with spherical interpolation,
30

---

Published as a conference paper at ICLR 2023
where the vector is moved along the surface of hypersphere between the two orthogonal vectors 1/1 and V2-
D.1
ADDITIONAL SAMPLES
0
0
4
7
7
5
8
1
5
7
9
6
7
6
2
7
9
4
0
7
8
9
9
6
4 O 0 53 3
3
6
6
4
8
/
4
0
5
5
8
2
6
9
5
4 8
/
9
9
8
8
6
3
7
3
5
5
006000000 5
6
6
0
/ 9
8
4
8
9
9
6
|
6
2
9 0
9 4
/
5
0
1
5
8
3
9
7
8
3
5 0 2 3
9
0
3
5
3
3
9
8
4
0
6
2
/
0 1
/
9
7
3
0
0
9
2
0
57005444
5
4
5
6
2
3
7
6
2
/
4
5 9
4 2
9
29
2
8
3
8
5
3
1
6
0
9
0
0
9
6
/
9
5
9
7
9
4
8
8
4
6
9
6
7
Figure 17: Uncurated samples on MNIST.
Initial state
Generated samples
9
3
7
9
4
4
9
8
7
9
4
4
8
9
4
9
6
8
6
6
6
2
6
0
6
2
0
6
0
0
9 7
3
6
3
3
6
9
9
9
2
4
7
9 7 /
7
4
6
5
3
8
9
6
4
9
5
8
8
4
9
9
3
3
/
8
/
7
/
9
9
8
5
0
0
0
0
6
3
8
0
0
0
6
0
0
8
3
0
0
O
0
5
0
3
9
3
8
6
6
8
2
0
2
2
0
2
4
6
0
3
0
8
8
6
6
8
3
9
4
3
6
3
5
4
9
2
2
8
0
0
3
5
8
8
6
Figure 18: Uncurated samples on MNIST, with shared initial states ug-
31

---

Published as a conference paper at ICLR 2023
Figure 19: Uncurated samples on CIFAR-10. FID 18.96.
32

---

Published as a conference paper at ICLR 2023
Initial state
Generated samples
Figure 20: Uncurated samples on CIFAR-10, with shared initial states uk--
33

---

Published as a conference paper at ICLR 2023
shutterstor
shutt
Shiute
Figure 21: Uncurated samples 128x 128. FID 45.06.
34

---

Published as a conference paper at ICLR 2023
Initial state
Generated samples
Shiltterstock
shitters
Figure 22: Uncurated samples on L.SUN-CHURCHER
128x 128, with shared initial states ug-
35

---

Published as a conference paper at ICLR 2023
Figure 23: Uncurated samples on AFHQ 64x64, FID 14.78.
36

---

Published as a conference paper at ICLR 2023
Initial state
Generated samples
Figure 24: Uncurated samples on AFHQ 64x64, with shared initial states uk-
37

---

Published as a conference paper at ICLR 2023
Figure 25: Uncurated samples on FFHQ 256x256. FID 64.91.
38

---

Published as a conference paper at ICLR 2023
Initial state
Generated samples
Figure 26: Uncurated samples on FFHQ 256x256, with shared initial states
39

---

Published as a conference paper at ICLR 2023
Figure 27: Uncurated samples on AFHQ 256x256. FID 43.49.
40

---

Published as a conference paper at ICLR 2023
Initial state
Generated samples
Figure 28: Uncurated samples AFHQ 256 256, with shared initial states ug-
41

---

Published as a conference paper at ICLR 2023
D.2 INTERPOLATIONS
Figure 29: Interpolations between two random images on LSUN-CHURCIIES 128x 128.
42

---

Published as a conference paper at ICLR 2023
Figure 30: Interpolations between two random images on FFHQ 256x256;
43

---

Published as a conference paper at ICLR 2023
Figure 31: Interpolations between two random images on AFHQ 64x64. 44

---

Published as a conference paper at ICLR 2023
D.3 EFFECT OF SAMPLING NOISE 8
0 0.00
0.005
= 0.0075
6 0.01
0.0105
= 0.011
8 0.0115
0.012
8 0.0125
0.013
8 = 0.0135
0.014
0.0145
0.015
Illustration of the effect of the sampling noise parameter ó on CIFAR-10 model. For each column, the sampling noise added during the generative process is sampled only once, and scaled with the different ô values the different rows allow for easier comparison. We highlight 5 = 0.01, before which changes are slow, and a good default value = 1.25 x that works well across data sets.
Figure 32:
45

---

Published as a conference paper at ICLR 2023
0 = 0.00
0.005
8 0.0075
8 = 0.01
8 == 0.0105
0.011
8 0.0115
= 0.012
0.0125
8 0.013
8- 0.0135
= 0.014
8 0.0145
0.015
Illustration of the effect of the sampling noise parameter on our For each column. the sampling noise added during the generative process is sampled only once, and scaled with the different ó values on the different rows to allow for casier comparison. We highlight 5 0.01, before which changes are slow, and good default value 5 1.25 that works well across data sets.
Figure 33:
model.
46

---

Published as a conference paper at ICLR 2023
0 = 0.00
8 0.005
8 0.0075
8 = 0.01
8 = 0.0105
8 = 0.011
6 0.0115
8 0.012
8 = 0.0125
8 = 0.013
6 = 0.0135
8 0.014
8 0.0145
0.015
Illustration of the effect of the sampling noise parameter 5 on AFHQ 64x64 model. For each column, the sampling noise added during the generative process is sampled only once, and scaled with the different ê values on the different rows allow for easier comparison. We highlight 8 = = 0.01, before which changes are slow, and a good default value ò = 1.25 x a that works well across data sets.
Figure 34:
47

---

Published as a conference paper at ICLR 2023
5 = 0.00
8 0.005
2
8 0.0075
8 = 0.01
5 = 0.0105
8 = 0.011
5 = 0.0115
= 0.012
8 = 0.0125
8 0.013
8 0.0135
8- 0.014
8 0.0145
8 = 0.015
Figure 35: Illustration of the effect of the sampling noise parameter 5 on our FFHQ model. For each column, the sampling noise added during the generative process is sampled only once, and scaled with the different ô values on the different rows allow for easier comparison. We highlight 5 == a = 0.01, before which changes are slow, and a good default value d = 1.25 x a that works well across data sets. Best seen zoomed in.
48

---

Published as a conference paper at ICLR 2023
8 0.00
8 0.005
8 0.0075
8 = 0.01
5 = 0.0105
8 = 0.011
5 = 0.0115
0.012
8 = 0.0125
8 0.013
8 0.0135
8 0.014
8 0.0145
8 0.015
Figure 36: Illustration of the effect of the sampling noise parameter 5 our AFHQ 256 x 256 model. For each column, the sampling noise added during the generative process is sampled only once, and scaled with the different ó values on the different rows allow for easier comparison. We highlight 5 = a = 0.01, before which changes are slow, and a good default value d = 1.25 x a that works well across data sets. Best seen zoomed in.
49

---

Published as a conference paper at ICLR 2023
0.00
9
9
9
9
0
6
6
0
9
0
0
9
0
9
O. 0.005
9
8
7
9
6
6
9
8
9
0
6
8
0
8
8 0. 5.0075
9
8
7
9
6
6
9
8
9
0
6
7
0
8
0.01
3
5
7
9
6
6
8
8
9
0
6
7
6
3
8 0.0105
3
57
9
6
8
110067
9
0
6
3
8 0.011
9
3
579668810
6
8
8900676
3
8 0.0115
3
57
8
3
8 0.012
3
57
6
8
619676
9
0
6
3
6 0.0125
3
57 4 a
6
8 8
0
6
6
3
8 0.013
3 4574
6 8 5
0
6 6
6 3
6 0.0135
8 = 0.014
3 5 (
3
5 = 0.0145
65946685(904)
3
3 5
= 0.015
3 5 2
3
0.02
Figure 37: Illustration of the effect of the sampling noise parameter on our MNIST model. For each column, the sampling noise added during the generative process is sampled only once, and scaled with the different 8 values on the different rows to allow for easier comparison. Interestingly, the model performs well with wide range of on MNIST, including with deterministic sampling. Intuitively, this may be because the MNIST data set does not contain any small-scale details apart from the edges of the digits, and seen on other data sets, the model able to reshape the prior image mass into simple shapes with = 0. We add results from 5 = 0.02 (not shown on other data sets) to show that the results do start degenerating with high enough sampling noise on MNIST as well.
50

---

Published as a conference paper at ICLR 2023
D.4
DISENTANGLEMENT OF COLOUR AND SHAPE BY FIXING THE SAMPLING NOISE
Result Prior uk
Result Prior uk
Result Prior uK
Result Prior uk
Result Prior uk
Result Prior uk
Figure 38: Disentanglement of colour and shape FFHQ 128 x 128, where d max 128. The noise steps added during the process are fixed, and only the initial state p(ux) is changed, resulting in the model carving out images with very similar characteristics, but with different average colours. The sample quality is somewhat lower on this model than the regular FFHQ model, but it illustrates the effect.
51

---

Published as a conference paper at ICLR 2023
D.5 NEAREST NEIGHBOURS ON SAMPLED IMAGES Generated image
Nearest neighbours
Figure 39: Samples on FFHQ vs. their Fuclidean nearest neighbours the training data.
52

---

Published as a conference paper at ICLR 2023
Generated image
Generated image
Nearest neighbours
Nearest neighbours
Figure 40: Samples on AFHQ 64x64 vs. their Euclidean nearest neighbours on the training data.
53

---

Published as a conference paper at ICLR 2023
Generated image
Nearest neighbours
sshutteistock
shattes tetterstckk
shu
Figure 41: Samples on LSUN- SSUN-CHURCHES 128x 128 vs. their Euclidean nearest neighbours on the training data.
54


Reviews:
[{"review_id": "Pc2rVUPoIQ", "reviewer": "Reviewer_kPGb", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new generative model by using inverse heat dissipation. The proposed model is motivated by diffusion models and the empirical success of coarse-to-fine modelling. Specifically, the authors propose to synthesize images by iteratively inverting the heat equation, which is basically a PDE that locally erases fine-scale information. Furthermore, they theoretically interpret the noise-relaxed solution of the forward heat equation as a variational approximation in a diffusion-like latent variable model. Extensive theoretical and experimental analyses have been conducted to demonstrate the effectiveness of the proposed model. ", "summary_of_the_review": "This paper proposes a new generative model which is both theoretically and technically sound. Extensive mathematical deviations of method details have also been provided making it a solid paper. Despite the above-mentioned weaknesses, I believe that this paper is still deserved to be published in ICLR after some necessary revisions. The proposed method is interesting and shows up some valuable potential for future research. \n\nHowever, the quality of synthesized images of the proposed method is not as good as the state of the art, and there is no comparison between existing methods and the proposed one. Otherwise, I could give it a higher score. \n", "strength_and_weaknesses": "Strengths:\nThis paper proposes a new type of generative model based on inverse heat dissipation, which possesses a different mechanism compared to existing diffusion models. Moreover, a solid and extensive mathematical derivation of method details has been provided in the Appendix. The proposed method shows emergent qualitative properties not seen in existing diffusion models, which has also been proved by experimental results.\n\nWeaknesses:\nFrom the qualitative results shown in this paper, it seems to me that existing SOTA generative models (e.g., StyleGAN, IMAGEN, DALLE) clearly outperform the proposed method. Yet, since the paper opens a new way for designing generative models, it is acceptable for me that the quality of synthesis results obtained by the proposed one is not as good as SOTA ones. However, there is no qualitative and quantitative performance comparison between the existing methods and the proposed one. Please explain the reasons. In fact, no matter whether the performance is good or not, it is still necessary to compare the proposed method against existing approaches. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NCm4ctRNg2b", "reviewer": "Reviewer_ZA3r", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to use heat dissipation to define a forward process (blurring) in the diffusion model. It uses eigendecomposition to decompose the Laplace operation and uses a variational model, similar to DDPM to learn the generative process. Gaussian noise is added to ensure the diversity of the generated images.\n\nOverall using different blurring ops is an interesting exploration direction.", "summary_of_the_review": "Please see above. I will adjust my score based on the rebuttal.", "strength_and_weaknesses": "Strength:\n1. Overall the idea is valid and interesting;\n2. This paper provides several interesting analysis such as PSD analysis.\n3. Lots of experiments are conducted to show interesting properties of the model.\n\nWeakness:\n1. There is not much supportive evidence to demonstrate the unique advantage of using heat dissipation. This greatly reduces the novelty of the paper. I believe there should be some important undiscovered advantages to using this approach. As the effective resolution is changing --- can it save the inference time?\n\n2. There are several numerical approach to deal with the Laplace ops: 1. the eigendecomposition used in this paper; 2. directly using Gaussian blurring; 3. grid-based finite difference; To me there seems no significant difference between these three approaches. Although author mentioned numerical stability and some gradient analysis, I would say it is worth really running those models to exam the outcomes.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9CxP8czTPV", "reviewer": "Reviewer_dZbG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a variant of the diffusion-based model with \"inverse heat dissipation\" (equivalent to continuous de-blurring Gaussian blur). Compared to the standard diffusion model, inverting the heat equation -- a PDE locally erases fine-scale information -- makes the generation consider multi-scale structures naturally. The paper also shows empirical evidence on emergent properties when applying IHDM, such as disentanglement of color and shape.", "summary_of_the_review": "Overall, based on the review above, I think the methodology and the proposed method are technically reasonable and novel, which can be a good contribution toward a better understanding of diffusion-based models beyond Gaussian noises. However, as the concerns raised in the weakness, the poor visual performance and lack of proper comparison/justification with baselines decrease the actual impact.", "strength_and_weaknesses": "Strength:\n-- An interesting study is one of the first several works applying blurring transformations in the diffusion process. Unlike existing works, the paper directly derives the formulation from the heat equation SDE and analyzes the inductive bias compared to the standard diffusion models.\n\nWeakness:\n-- The empirical results are relatively much worse than standard diffusion models. The paper seems to hide the quantitative comparisons (only small captions in the figure). However, the visual quality is also over-smoothed compared to the baselines. Although it is brought by the change of diffusion process, it is hard to justify \"why we want to apply heat equation in generative models in the first place,\" considering there is less merit in doing so.\n-- Moreover, there are also existing works that try to incorporate Gaussian blurring in the diffusion process. For example, \n\nLee, Sangyun, Hyungjin Chung, Jaehyeon Kim, and Jong Chul Ye. \"Progressive deblurring of diffusion models for coarse-to-fine image synthesis.\" arXiv preprint arXiv:2207.11192 (2022).\n\nSimilar to this submission, their method applies a stochastic blurring process in diffusion models. While having differences, their quantitative and qualitative results are much better and match the same performance of standard diffusion models. I think comparing more carefully than citing them solely and indicating solutions to improve the current method is necessary. \n\n-- Noise scheduling is less clear than the standard diffusion models, especially since this paper involves two hyperparameters, \"sigma\" (forward) and \"delta\" (backward), which, however, are unclear how to choose both of them and how to adapt with the blurring schedule (Gaussian kernel)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4UxFpK3g594", "reviewer": "Reviewer_D1k3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to combine heat dissipation forward process with gaussian diffusion.", "summary_of_the_review": "This is an interesting work proposing a novel approach to generative modeling of images based on heat dissipation. I am learning towards acceptance.", "strength_and_weaknesses": "\\+ This is an interesting topic and it improves our understanding of what is possible in generative modeling.\n\n\\+ The experiments are well-analyzed.\n\n\\- The main text of the paper contains excessive details which make it harder to read.\n\n\\- Experimental results do not show that the proposed method is a promising direction, at least in terms of the sample quality.\n\nQuestions/minor comments:\n* \"Their generative process does not explicitly consider the inductive biases of natural images, such as their multi-scale nature\" - Imagen (https://arxiv.org/abs/2205.11487) does use upsamplers, some of its predecessors also do.\n* This work is not just heat dissipation, but rather a combination of heat dissipation with gaussian noising - this needs to be highlighted early in the paper. \"noise-relaxed solution of the forward heat equation\" in the abstract sounds rather obscure, just saying that it's a combination with gaussian diffusion would be more clear.\n* \" We obtain samples by taking a training example, blurring it with F(tK), and adding noise with variance δ2\" - does it mean that the prior is non-parametric, so we would need the training set during the generation time? That sounds like a limitation which should be mentioned early on?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster