Spontaneous symmetry breaking in generative diffusion models
Authors: Gabriel Raya
Luca Ambrogioni
Venue: NeurIPS 2023 poster (2023)
Keywords: generative models;diffusion models;score-based generative models; symmetry-breaking
TL;DR: This paper uncovers spontaneous symmetry breaking in diffusion models, indicating the existence of a phase transition in their generative dynamics.

Abstract: Generative diffusion models have recently emerged as a leading approach for generating high-dimensional data. In this paper, we show that the dynamics of these models exhibit a spontaneous symmetry breaking that divides the generative dynamics into two distinct phases: 1) A linear steady-state dynamics around a central fixed-point and 2) an attractor dynamics directed towards the data manifold. These two "phases'' are separated by the change in stability of the central fixed-point, with the resulting window of instability being responsible for the diversity of the generated samples. Using both theoretical and empirical evidence, we show that an accurate simulation of the early dynamics does not significantly contribute to the final generation, since early fluctuations are reverted to the central fixed point. To leverage this insight, we propose a Gaussian late initialization scheme, which significantly improves model performance, achieving up to 3x FID improvements on fast samplers, while also increasing sample diversity (e.g., racial composition of generated CelebA images). Our work offers a new way to understand the generative dynamics of diffusion models that has the potential to bring about higher performance and less biased fast-samplers.

Spontaneous Symmetry Breaking in Generative Diffusion Models
Gabriel Rayal.2 Luca Ambrogioni
JJeronnmuus Academy of Data Science Tilburg University Raddboud University
A Donders Institute for Brain, Cognition and Behaviour
g raya@ejas................aaaaaa nl, 1. ambrogion @donders iotoroogii@@onnersss ru.nl
Abstract
Generative diffusion models have recently emerged as a leading approach for generating high-dimensional data. In this paper, we show that the dynamics of these models exhibit a spontaneous symmetry breaking that divides the generative dynamics into two distinct phases: 1) linear steady-state dynamics around a central fixed-point and 2) an attractor dynamics directed towards the data manifold. These two "phases" are separated by the change in stability of the central fixed- point, with the resulting window of instability being responsible for the diversity of the generated samples. Using both theoretical and empirical evidence, we show that an accurate simulation of the early dynamics does not significantly contribute to the final generation, since early fluctuations are reverted to the central fixed point. To leverage this insight, we propose Gaussian late initialization scheme, which significantly improves model performance, achieving up 3x FID improvements fast samplers, while also increasing sample diversity (e.g., racial composition generated CelebA images). Our work offers a new way to understand the generative dynamics of diffusion models that has the potential to bring about higher performance and less biased fast-samplers
Introduction
In recent years, generative diffusion models (Sohl-Dickstein et al., 2015), also known score-based diffusion models, have demonstrated significant progress in image (Ho et al., 2020; Song et al., 2021) 1) sound (Chen et al., 2020; Kong et al., 2020; Liu al., 2023) and video generation (Ho al., 2022; Singer et al., 2022). These models have not only produced samples of exceptional quality, but also demonstrated a comprehensive coverage of the data distribution. The generated samples exhibit impressive diversity and minimal mode collapse, which are crucial characteristics of high-performing generative models (Salimans et al., 2016; Lucic et al., 2018: Thanh- -uuh and Tran, 2020). Diffusion models are defined in terms of a stochastic dynamic that maps simple, usually Gaussian, distribution into the distribution of the data, In an intuitive sense, the dynamics of a generated sample passes from a phase of equal potentiality, where any (synthetic) datum could be generated, to a denoising phase where the (randomly) "selected" datum is fully denoised. we shall see in the rest of this paper, form of spontaneous symmetry breaking. As stated by Gross (1996), "The
this can be interpreted secret of nature is symmetry, but much of the texture the world due to mechanisms of symmetry breaking." Surprisingly, the concept of spontancous symmetry breaking has not yet been examined in the context of generative modeling. This is particularly noteworthy given the importance of spontaneous symmetry breaking in nature, which accounts for the emergence of various phenomena such as crystals (breaking translation invariance), magnetism (breaking rotation invariance), and broken gauge symmetries, common theme in contemporary theoretical physics.
The concept of spontaneous symmetry breaking is strongly connected with the theory phase transitions (Stanley, 1971; Donoghue et al., 2014). In high energy physics, experimental evidence of
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

dX,
Vxu(XI,7
fddt
Odw
(A) Symmetry breaking in 1D dimision modal
(1) Symmetry braaking in GelehA HQ 256rx258
Figure I: Overview of spontaneous symmetry breaking in generative diffusion models. a) Symmetry breaking in a simple one-dimensional problem with two data points (-1,1). The figures on the top illustrate the potential at different time points, while the bottom figure displays the stochastic trajectories. The red dashed line denotes the time of the spontaneous symmetry breaking (computed analytically). The red arrows represent fluctuations around the fixed-point of the drift. b) Symmetry breaking in a real dataset. The top figures show ID sections of the potential of trained diffusion models (CelebA HQ) at different times. The potential is evaluated along circular interpolating paths connecting two generated samples (bottom figure).
particle collisions and decay appears to respect only a subset of the symmetry group that characterizes the theory weak and electromagnetic interactions. This mismatch, which is responsible for the non- null mass of several particles, is thought to be due to the fact that the potential energy of the Higgs field has infinitely many minima, each of which corresponding to a subgroup of the symmetries the standard model (Englert and Brout, 1964; Higgs, 1964; Anderson, 1963; Nambu and Jona-Lasinio 1961). Therefore, while the overall symmetry is preserved the potential, this is not reflected in our physical experiments been arbitrarily "selected". The Higgs potential often described as a "Mexican hat", since it has an unstable critical point at the origin and a circle of equivalent unstable points around it (see Figure 2). Spontaneous symmetry breaking phenomena are also central to modern statistical physics as they deseribe the thermodynamics of phase transitions (Stanley, 1971). For example, ferromagnetic material at low temperature generates a coherent magnetic field in one particular direction since all the atomic dipoles tend align the field. On the other hand, higher temperature the kinetic energy prevents this global alignment and the material does not generate a macroscopic magnetic field. In both cases, the laws of physics are spherically invariant and therefore do not favor any particular directions. However, at low temperature a direction is selected among all the equally likely possibilities, leading to an apparent breaking of the physical symmetries of the system. The global symmetry can then only be recovered by considering an ideal ensemble of many of these magnets, each aligning along one of the equally possible directions.
experience world where only one of these equivalent Higgs state has
In this paper, using both theoretical and experimental evidence, we show that the generative dynamics of difffusion models is characterized by a similar spontaneous symmetry breaking phenomenon. In this case, the symmetry group does not come from the laws of physics but it is instead implicit in the dataset. For example, translational invariance can be implicit in the fact that translated versions of similar images are equally represented in naturalistic dataset. During the early stage of the generative dynamics, each particle reflects all the symmetries since its dynamics fluctuates around a highly symmetric central fixed-point. However, after a critical time, this central fixed- point becomes unstable, and each particle tends towards different (synthetic) datum with arbitrarily "selected" features, with the global symmetry being apparent only when considering the ensemble of all generated data. An overview of spontaneous symmetry breaking in diffusion models summarized Figure Our code can be found at https://github.com/ http://hwww...ooooaaaaaayymmeetm breaking diffusion satissiiioooolllsooeee models
Figure 2: Mexican hat potential

---

2 Preliminaries
Notation. We denote random variables using upper-case letters and their value using lower-case letters. Additionally, vector-valued variables are denoted using boldface letters. The forward process denoted as s)), where Yo represents the data and Y. denotes noisy state. The generative process is denoted as (X,,t), with s and = 7 - & representing forward and generative time, respectively, and T as the time horizon. We will always use the standard integration from to T' (different to Song et al. (2021)) for the short hand notion of the Itō integral equation. Since we focus on the generative part, we use W. and W, to denote Brownian motion associated to the inference and generative SDEs. For ease of notation, we assume an additive SDE, so g only depends on time.
Continuous diffusion models. The stochastic dynamics of a particle Yu ~ p(y: 0), starting at time s = 0, are described as the solution to the Itô SDE: dY. = f(Y s)ds + (s))W,,, s. where and g are the drift and diffusion coefficient chosen properly such that the marginal density will (approximately) converge to spherical Gaussian steady-state distribution as s + T. We can express the marginal density time s as
/ k(y. yo: O)p(yo: 0)dyo
ply. 3)
JRD
where k(y,s; s; s')) p(y.0) 0) by starting from the tractable p(y. 7)), we can employ a "backward" SDE that reverses this process (Anderson, 1982), whose marginal density evolves according to p(y. s), reverse in time,
) is the transition kernel that 'solves' Eq. 2 below. To generate samples from
dx, g2 xx IX,
f(x,, {)] dt 9T t)dW,
The score function V log p(x, T t) directs the dynamics towards the target distribution ply.0) and can be reliably estimated using a denoising autoencoder loss (Vincent, 2011: Song et al., 2021). Ornstein-Uhlenbbekk process. In the rest of the paper, we will assume that the forward process follows a (non-stationary) Ornstein -uhhenbbeh dynamics: dY, = =
ds / /B(s) an instance of Variance Preserving (VP) diffusion (Song et al., 2021) wherein the transition kemel (s) -]
This
Jo ((r)ar It is easy to see that this kernel reduces to an unconditional standard spherical normal for + oo while it tends to delta function for 0.
can be written in closed form: kyy, s; Vo 0) N (3y: 0.Yo. (1 02)1), with 0g
3 Theoretical analysis
For the purpose of our analysis, is convenient to re-express the generative SDE in Eq. 2 in terms of a potential energy function
dx, = V. T -f)dt + 9(T - -tddW,
(3)
where
u(x, 5) = -g'(s) log p(z, s) + / s(z,s) dz.
(4)
where the line integral can go along any path connecting O and æ. Given a sequence of potential functions u(æ s), we can define an associated symmetry group of transformations
= {g RRR ++ R. u(g(z), 8) = u(æ s), 8 E R x E ER?).
(5)
In words, is the group of all transformations of the ambient space R. that preserves the probability measure of the training set at all stages of denoising.
We define path of fixed points as #(t) T RD such that Vu(ā((), = 0, E R+. These are points of vanishing drift for the stochastic dynamics. The stability of the path can be quantified using the second partial derivatives, which can be organized in the path of Hessian matrices H (w,T" - t). A fixed-point is stable when all the eigenvalues of the Hessian matrix of the potential that point are positive, while it is a saddle or unstable when least one eigenvalue is negative. Around a stable path of fixed-point, the drift term can be well approximated by a linear function: Veu(z,T t) H (x 2) From this we can conclude that, along a stable path, the dynamics is locally characterized by the quadratic potential
1 - --((t)) H()(t, t) z(t))
ū (xx T t)
(6)
3

---

-26x
10
Criscal urea
6200
-05
2
(a)
(b)
Figure 3: Bifurcation analysis of the generative dynamics of one-dimensional diffusion model. (a) Geometric visualization of bifurcation of fixed points through the intersection straight line and a hyperbolic tangent at a value Or d.. (b) Bifurcation diagram obtained by numerically solving the celf-consistency equation Eq. 10, demonstrating the bifurcation at the critical value 0z. The vector field is given by the drift term (i.e. the gradient of the potential) in the generative SDE.
The blue, orange and green lines denote the three paths of
The associated symmetry group & is generally only subgroup of the global symmetry group G. We say that the dynamics exhibit a bifarcation when there are at least two fixed-points paths that overlap for some values of In this case, usually a stable fixed point loses its stability after a critical time te and 'splits' into two more stable paths. As we will see, this is the core of the spontaneous symmetry breaking phenomenon. Each of the branched stable paths only preserve a sub-group of the overall symmetry, while the full symmetry is still present when taking all stable paths into account.
3.1 Spontaneous symmetry breaking in one-dimensional diffusion models
We start by considering a very simple one one-dimensional example with a dataset consisting of two points y-1 and y/1 = - group that preserves the potential is comprised by identity and the transformation g(x) = zz Up to terms that are constant in x, the potential is given by the following expression:
sampled with equal probability. In this case, the symmetry
(x-er 1)
(e+9p o
u(x, t) = - -) - -- log
20 9 tE
21
which can be obtained from Eq. 1. Figure la illustrates the evolution of the potential (top) and the corresponding one-dimensional generative process (bottom). For values of f, the gradient vanishes at since the potential is symmetric under the transformation g(x) = The stability of this fixed-point can be established by analyzing the second derivative:
82
3( 1) ( 20%-1-11 -
dr T=
(67 - 1)
where 0r- is a monotonic function of ranging from 0 to 1. The second derivative is positive to critical value 9. and negative afterwards. This implies that the fixed-point at the origin loses its stability when fr-r 0c. We can find this critical value by setting the second derivative equal to zero and solving for fc.. which gives
0. - - 0.6436
(9)
When 0x-c 8c. the origin is not the only fixed-point of the system. All fixed ekedtppinnt be found by solving the self-consistency equation:
Or-
(0 + 1)x" -20 tanh
(10)
02
This equation strikingly similar to the Curie-Weits equation of state, which describes magnetization under the mean-field approximation (Täuber, 2014). Solving this equation corresponds to finding

---

the intersections between a straight line and an hyperbolic tangent. From Figure 3a, it is clear that there are three solutions for f 0g and only the zero solution otherwise. This corresponds to bifurcation into two paths of fixed points that converge to the values of the data-points for 0r-t -> 1.
We can now describe the spontaneous symmetry breaking phenomenon. The potential u(x, t) is invariant under the transformation g(x) = -r for all values of 0r However, for Or > 0.. the individual particles will be 'trapped' in one of those new stable paths of fixed-points, locally breaking the symmetry of the system. From the point of view of generative modeling, this spontaneous symmetry breaking corresponds the selection of a particular sample among all possible ones. This selection almost exclusively depends on the noise fluctuations around the critical time. In fact, fluctuations for & < te are irrelevant since the process mean reverting towards the origin. Similarly, when t > t.. fluctuations will be reverted towards the closest fixed-point However, fluctuations are instead amplified when f. le since the origin becomes unstable, as illustrated by the red arrows in Figure la.
In supplemental section A.2, we provide detailed calculations pertaining this one-dimensional example. Additionally, in section A.3, we generalize our investigation to models of arbitrarily high dimensionality by considering a hyper-spherical data distribution.
3.2 Theoretical analysis of spontaneous symmetry breaking for arbitrary normalized datasets
We can now move to a more realistic scenario where the data is comprised by finite number N of data-points (y1,-... yN embedded in R2 Assuming iid sampling, the most general symmetry group in this case is given by all norm-preserving transformations of R&D that map data-points into data-points, Up to constant terms, the potential is given by
11--0- -10,1 07
11001 log Σ
u(xx,) = -st)
e
(11)
where the sum runs over the whole dataset. The fixed-points of this model can be found by solving the following self-consistency equation:
1 1+
07 x²
Σ Wy (x) :0==;=)0)0))
(12)
20T
E, (r=)))))))
-r-- 11/11211-644)) While this is a very general case, we can still prove the existence of a spontaneous symmetry breaking the origin under two mild conditions. First of all, we assume the data-points to be centered: 2, y = 0. Furthermore, we assumed that the data-points are normalized so as to have a norm r: |,, = r Vj Under these conditions, which can be easily enforced on real data through normalization, it straightforward to see from Eq. 12 that the origin is minimum of the potential for all values of t. While we cannot evaluate all the eigenvalues of the Hessian matrix at the origin closed-form, we can obtain a simple expression for the trace of the Hessian (i.e. the Laplacian of the potential):
where wy (r)
(
D
(D + D++2)07
Pul==0==300 =
(13)
(0² (07_.- 112
which switches sign when 0y is equal to 0" V +
However, in this case we cannot conclude that this is the point of the first spontaneous symmetry breaking since the Laplacian the sum of all second derivatives. which are not necessarily all equal. Nevertheless, we do know that all second derivatives are positive at the origin for - 0. since the forward dynamics has Gaussian steady-state distribution. Therefore, from the change of sign of the Laplacian we can conclude that at least one second derivative at the origin changes sign, corresponding to change in stability and the onset of a spontaneous symmetry breaking with 0* 0.. In supplemental section A.4, provide detailed calculations pertaining to this model.
5

---

Experimental evidence of spontaneous symmetry breaking in trained diffusion models
In this section, we present empirical evidence demonstrating the occurrence of the spontaneous sym- metry breaking phenomenon in diffusion models across a range of realistic image datasets, including MNIST, CIFAR-10, CelebA 32x32, Imagenet 64x64 and CelebA 64x64. We trained diffusion models in discrete time (DDPM) with corresponding continuous time Variance-Preserving SDE for each dataset with a time horizon of T 1000 and evaluated fast samplers using Denoising Diffusion Implicit Models (DDIMs) (Song et al., 2020) and Pseudo-numerical Methods for Diffusion Models (PNDM) (Liu al., 2022). For more detailed information about our experimental implementation, please refer to supplemental material section E.
4.1 Analyzing the impact of a late start initialization on FID performance
To investigate how the bifurcation affects generative performance, we performed an exploratory analysis of Fréchet Inception Distance (FID) scores for different diffusion time starts. The objective of this analysis to examine the possible effects of starting the generative process a late diffusion time, since our theory predicts that performance will stay approximately constant up to first critical time and then it will sharply decrease. We explicitly define a "late start" as changing the starting point of the generative process. For instance, instead of starting at 7'' = 1000, generative performance for various starting points, ranging from ssaart 50 to 8start = T, using starting state Xy NN, I). In particular, since the distribution is not longer (O. I) for any 8 << T, a decrease in performance can be expected. However, as shown in Figure 4, denoted as "DDPM-1000", there is indeed initial phase in which the generative performance of the model remains largely unaffected by this distributional mismatch, and in some cases, better FID scores were obtained. Remarkably, as predicted by our framework, in all datasets performance after a certain initialization time threshold, with an apparent discontinuity in the second derivative of the FID curve. We provide additional results in supplemental section B.
evaluate the model's s
see sharp degradation in
400
430
M405
CONN-ES
300
19674
000
220
Dillstonn time
Ofh
(a) MNIST
(c) Imagenet64
DC 00/99-000 1000
DCPM 1C49
FSDR
3000
230
100
2500
250
200
Eittidioo time
(b) CIFAR10
(d) CelebA64
Imagenet late generation
Figure 4: Analysis of the model's performance, as measured by FID scores, for different starting times using three different sampling methods: the normal DDPM sampler with decreasing time steps from T = 1000 to O, and fast sampler DDIM and PSDM for 10 and 5 denoising steps. The vertical line corresponds to the maximum of the second derivative of the FID curve, which offers a rough estimate of the first bifurcation time. (e) Illustrates samples generation on Imagenet64, while progressively varying the starting time from 1000 to 100.

---

Dataset
n
gl-DDPM
DDPM
Dataset
n
glsDDIM
DDIM
Datasel
n
gh-PNDM
PNDM
6.75
2.44
4.46
5.02
14.36
MNIST
6.95
13.25
MNIST
6.95
13.25
MNIST
5
5.11
21.22
11.92
42.63
11.92
42.63
38.23
154,89
28.77
43.35
15.98
19.79
5.90
8.35
CIFARIO
42.46
84.82
CIFARIO
5
26.36
44.61
CIFARIO
5
9.55
13.77
$7.03
146.95
42.31
109.37
34.20
103.11
10
11,05
26.79
10
7.27
11.37
10
2.88
4.92
CelebA32
14.79
40.92
CelebA32
10.83
23.45
CelebA32
4.2
6.61
18.93
59.75
16.24
45.34
28.60
235.87
10
57.31
63.68
10
36.25
38.21
10
27.9
28.27
Imagenet64
75.11
99.99
Imagenet64
52.11
Imagenet64
33.35
34.86
91.69
145.71
76.92
126.3
50,92
70,58
10
23.79
36,66
10
15.82
19.37
10
6.80
8.03
CelebA64
31.24
48.38
CelebA64
22.06
28.51
CelebA64
9.26
10.26
37.05
62.18
29.96
50,304
51.72
171.75
(a) DDPM
(b) DDIM
(c) PNDM
Table 1: Summary of findings regarding image generation quality, as measured by FID scores. The performance of the stochastic DDPM sampler (a) is compared to the deterministic DDIM (b) and PNDM (c) samplers in the vanilla case, well as our Gaussian late start initialization scheme denoted as "gls". Results are presented for 3, 5, and denoising steps (denoted as "n") across diverse datasets.
4.2 Empirical analysis of the potential function in trained diffusion models
The analysis of the FID curves offers indirect evidence of the existence of a critical time that demarcates the generative dynamics of diffusion models trained on real datasets. In order to obtain direct evidence, we need to study the potential associated trained models. Unfortunately, studying the stability of the initial fixed-point in trained model is challenging given the high dimensionality and the fact that its location is not available analytically. To reduce the dimensionality, we analyzed how the potential changes across variance-pree preseecvigreeeiiggggg interpolating curves that connect generated images. We defined an interpolating curve as x(a,t) t) = (as x (t) + sin(a: (())) where æ and (t) are two sampled generative paths. Up to constant terms, the potential along the path ((a,t) t) = u( t) can be evaluated from the output of the network (i.e. the score) using the fundamental theorem of line integrals (see supplemental section B.2.1). An example of these potential curves is shown in Figure 1b (top), as predicted by the theory, the potential has a convex shape up time and then splits into bimodal shape. A visualization of the average potentials for several datasets is provided in the supplemental section B.2.2. Interestingly. this pairwise splits happens significantly later than the critical transition time observed in the FID analysis. This suggests that, in real datasets, different spontancous symmetry breaking phenomena happen at different times, with an early one causing a major distributional shift in the overall dynamics (as visible from the FID curves) and later ones splitting the symmetry between individual data-points.
Improving the performance of fast samplers
In the following, we leverage the spontaneous symmetry breaking phenomenon in order to improve the performance of fast samplers. Since the early dynamics is approximately linear and mean- reverting, the basic idea is to initialize the samplers just before the onset of the instability. This avoids wasteful utilization of denoising steps in the early phase while also ensuring that the critical window of instability has been appropriately sampled. Unfortunately, while we can prove the existence of spontancous symmetry breaking in large family of models, we can only determine its exact time in highly simplified toy models. We therefore find an "optimized" starting time empirically by generating samples for different starting times (with a fixed number of equally spaced denoising steps) and selecting the one that gives the highest FID scores. From our theoretical analysis it follows that, while the distribution of the particles can drift from Ao /) before the critical time point, it generally remains close to a multivariate Gaussian distribution. Leveraging these insights, we propose a Gaussian initialization scheme to address the distributional mismatch caused by a late start initialization. This scheme involves estimating the mean and covariance matrix of noise-corrupted
7

---

(a) DDPM S= 48D-48.38
(c) gls DDPM n=5, FID=31.24
(b) DDPM 1= FID=36.66
(d) gls-DDPM 1= FID=23.79
Figure 5: Comparison of stochastic DDPM samplers on CelebA64 with varying denoising steps. Subfigures (a) and (c) represent the generative model performance for denoising steps, while (b) and (d) showcase the results for 10 denoising steps. The DDPM was initialized with the common standard initialization point Sstart 800 for 5 steps and Sstart = 900 for 10 steps. Notably, our Gaussian late start initialization (gls-DDPM) with Sstart 400 for both 5 and 10 denoising steps demonstrates significant improvements in FID scores and diversity, leveraging spontaneous symmetry breaking in diffusion models.
dataset at the initialization time and utilizing the resulting Gaussian distribution as the initialization point. By using this fitted Gaussian as the starting point, we can obtain an initial sample and run the denoising process from there instead of using a sample from (O, T). We refer to this method as "Gaussian late start" (gls) and present results for stochastic dynamics in Table 1a and deterministic dynamics in Tables 1b and Ic, using DDIM and PNDM samplers respectively. In experiments on several datasets, we found that in both cases, the Gaussian late start always increases model performance when compared with the baseline samplers. The performance boost is striking in some datasets, with a 2x increase in CelebA 32 for 10 denoising steps and a 3x increase for 5 denoising steps. As evidenced by the results showcased in Figure 5, gls-DDPM boost performance over vanilla DDPM for 10 and 5 denoising steps. For more detailed information on fast samplers, please refer to the supplemental section C. This section also includes extended results for higher number of denoising steps, as detailed in Table 6. Figure 19 provides empirical evidence of the Gaussian nature of the initial distribution via the Shapiro-Win Wilk test, which remains valid until the critical point.
5.1 Diversity analysis
Diffusion models that can quickly generate high-quality samples are highly de- sirable for practical applications. to a reduction in the diversity of generated samples. For instance, the analysis shown in Figure using the DDIM sampler with only 5 denoising steps on the CelebA64 dataset revealed a decrease in "race" diversity, with a bias towards the dominant race (white) and reduction in the coverage of other races. Our theoretical analysis suggests that achieving high generative diversity relies on reliably sampling a narrow temporal window around the critical time, since small perturbations during that window are amplified by the insta- bility and thereby play central role in the determination of the final samples. This suggests that our Gaussian late initialization should in- crease diversity when compared with the vanilla fast-sampler, since
However, the of
fast samplers can lead
Figure 6: Sample entropy as function DDIM denoising steps.
8

---

12:55
(a) Training set
(b) 1000
(c) gls-DDIM-05
(d) DDIM-0S
Figure 7: "Race" diversity analysis on CelebA64 over 50,000 generated samples by (c) gls- gls-DDIM and (d) DDIM samplers with 5 denoising steps. Results obtained on (a) training set and (b) DDPM using 1000 denoising steps are provided for reference. Corresponding samples obtained by each set are shown on top of the pie charts.
the Gaussian approximation is reliable prior to the change of shape in the potential. Indeed, our proposed late initialization method (gls-DDIM-05) was able to significantly improve sample diversity, even for a small number of denoising steps, e.g., 3, 5, and 10 sampling steps (see Figure 6). For comparison, we also provided the diversity obtained in the training set and from typical sampling of DDPM with 1000 sampling steps (DDPM- 1000). In supplemental section D, we also provided full analysis of attributes such as age, gender, race, and emotion by running facial attribute analysis over 50,000 generated images using the deep face library (Serengil and Ozpinar, 2020).
6 Related work
Efficient sampling in diffusion models. In recent years, research has been focused accelerating the slow sampling process in diffusion models, resulting in two categories of efficient samplers: learning-free and learningbbased. Among the learning-free samplers, Denoising Diffusion Implicit Models (DDIM) (Song et al., 2020) and Pseudo Numerical Diffusion Models (PNDM) (Liu et al., 2022) have gained considerable attention. DDIM introduced non-Markovian class of diffusion processes to achieve faster sampling, while PNDM proposed an enhancement to traditional numerical methods through the use of pseudo numerical methods. Both DDIM and PNDM implicitly employ a late start initialization using linearly spaced open interval running from 1 to T, with steps determined by the required number of denoising steps, and consequently not including T, but use the sample at dr as the starting point. The choice of the starting point has been proposed based on empirical results, and to the best of our knowledge, our work is the first to theoretically explore the relevance of this starting point and systematically suggest way choose it.
Diffuusion and spontaneous symmetry breaking in optimal control theory. The concept of spontaneous symmetry breaking has been studied in optimal control theory, with the work of Kappen (2005), who showed the existence of spontaneous symmetry breaking in both stochastic and deterministic optimal control problems. Spontaneous symmetry breaking in the cost as function of time implies a "delayed choice" mechanism for stochastic control, where taking a decision far in the future is sub-optimal due to high uncertainty. Instead, the optimal policy is to steer between the targets until the critical time one should aim for one of the targets. For T < Le (far in the past) it is best to steer towards x 0 (between the targets) and delay the choice which slit to aim for until later, when the uncertainty is less close to the terminal state. The relationship between optimal control theory and diffusion models has recently been explored by the work of Berner et al. (2022), who demonstrated that the evverse-bakwwar SDE is directly related to the Hamilton- lacobi-Bellman equation for the time-inverse log-density through a Hopf-Cole transformation. This theoretical link suggests that the spontaneous symmetry breaking phenomena that we have identified here are analogous to delayed choice phenomena in optimal control.
Encoding symmetries in generative models. Symmetries plays a central role in any area of physics, revealing the foundational principles of nature and enabling the formulation of conservation laws (Gross, 1996; Noether, 1971). A substantial body of deep learning literature has capitalized on this inherent property of nature by encoding equivariance to symmetry transformations constrains into deep neural networks (Cohen and Welling, 2016; 6; Worrall and Brostow, 201 88 Bogatskiy et al.,

---

2020; Cesa et al., 2021; Weiler et al., 2021) allowing to construct more efficient and physically interpretable models with minimized learnable parameters. Recently, this attribute has garnered attention in the realm of generative models, spurring interest in integrating both internal (Gauge) and external (space-time) symmetries (Kanwar et al., 2020), particularly when sampling symmetric target densities (Köhler et al., 2020). While the focus has been in incorporating symmetries in normalizing flows (Boyda et al., 2021; Köhler et al., 2020; Rezende et al., 2020; Satorras et al., 2021), there is already an increase interest in encoding symmetries in diffusion models Xu et al. (2022); Hoogeboom et al. (2022). Our work, along with previous research, acknowledges the importance of the existence of symmetry groups for generative modeling. However, the key distinction in our approach lies in emphasizing the relevance of spontaneous symmetry breaking for particle generation rather than encoding symmetry constraints.
Comparison with other generative models. While diffusion models differ conceptually from other deep generative approaches like Variational Autoencoders (VAEs) (Kingma and Welling, 2014; Rezende et al., 2014), GANs (Goodfellow et al., 2014), autoregressive models (Van Den Oord et al., 2016), normalizing flows (Dinh et al., 2014; Rezende and Mohamed, 2015), and Energy-Based Models (EBMs), they all share certain operational traits. Specifically, all these models models--xplicitty or implicitly-start from an isotropic equilibrium state and evolve toward a data-equilibrium state. Beyond this, an underlying theme across these generative models is the idea of pattern formation, marked by a transition from a high-symmetric state to a low-symmetric state. This commonality suggests the potential for leveraging the concept of symmetry breaking for performance improvement. For instance, implementing symmetry breaking mechanisms in GANs could address the well-known problem of mode collapse, thereby enhancing their generative performance. This idea is especially relevant for EBMs, which, like diffusion models, strive to learn an energy function that assigns lower values to states close to the data distribution. Consequently, the phenomenon of spontaneous symmetry breaking could offer key insights for refining the generative capabilities of all these models.
7 Discussion and conclusions
In this work, we have demonstrated the occurrence of spontaneous symmetry breaking in diffusion models, phenomenon of significant importance in understanding their generative dynamics. Our findings challenge the current dominant conception that the generative process of diffusion models is essentially comprised of single denoising phase. Instead, we have shown that the generative process is divided into two (or more) phases, the carly phase where samples are mean-reverted to a fixed point, and the "denoising phase" that moves the particle to the data distribution. Importantly, our analysis suggests that the diversity of the generated samples depends on a point of "critical instability" that divides these two phases. Our experiments demonstrate how this phenomenon improve the performance of fast samplers using delayed initialization and a Gaussian approximation scheme. However, the algorithm we proposed should be considered as proof of concept since the full covariance computation does not scale well to high-resolution images. Nevertheless, these limitations can be overcome either by using low-rank initialization or by fitting the Gaussian initialization (ower-dimensional space of Fourier frequencies. More generally, our work sheds light on the role that symmetries play the dynamics of deep generative models. Following this research direction, existing theoretical frameworks, such as optimal control and statistical physics, can be leveraged to make advancements in this field towards more efficient, interpretable, and fair models.
be leveraged to
Broader Impact
Our research has important implications for advancing deep generative models, which are essential for understanding complex phenomena like human creativity, molecular formation and protein design. However, the real-world usage of generative diffusion models can pose several risks and reinforce social biases. We hope that the insights and results offered in our work will have a positive social outcome by helping to create less biased models.
8 Acknowledgements
We would like to thank Eric Postma for his helpful discussions and comments. Gabriel Raya was funded by the Dutch Research Council (NWO) as part of the CERTIF-AI project (file number 1 17998).
10

---

References
Anderson, D. (1982). Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):31-3226.
Anderson, P. W. (1963). Plasmons, gauge invariance, and mass. Physical Review, 130(1):439.
Berner, J., Richter, L., and Ullrich, K. (2022). An optimal control perspective on diffusion-basee generative modeling. arXiv preprint arXiv:2211.0f306
Bogatskiy, A., Anderson, B., Offermann, J., Roussi, M., Miller, D., and Kondor. (2020). Lorentz group equivariant neural network for partiele physics. International Conference on Machine Learning.
Boyda, D., Kanwar, G., Racanière, S., Rezende, D. Albergo, M. S., Cranmer, K., Hackett, D. C., and Shanahan, E. (2021). Sampling using SU(n) gauge equivariant flows. Physical Review D, 103:074504.
Cesa, G., Lang, L., and Weiler, M. (2021). A program build e (n)-equivariant steerable cnns. International Conference on Learning Representations.
Chen, N., Zhang, Y., Zen, H., Weiss, J., Norouzi, M., and Chan, W. W. (2020). Wavegrad: Estimating gradients for waveform generation. arXiv preprint arXiv:2009.007/3.
Cohen, T. and Welling, M. (2016). Group equivariant convolutional networks. In International conference on machine learning, pages 2990-2999, PMLR.
Dinh, L., Krueger, D., and Bengio, Y. (2014). Nice: Non-linear independent components estimation. arXiv preprint arXiv:1410.85111
Donoghue, J. F., Golowich, E., and Holstein, B. R. (2014). Dynamics of the standard model. Cambridge university press.
Englert, and Brout, (1964). Broken symmetry and the mass of gauge vector mesons. Physical Review Letters, 13:321-323.
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative adversarial nets. In Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N., and Weinberger, K., editors, Advances in Neural Information Processing Systems, volume 27, Curran Associates, Inc.
Gross, D. J. (1996). The role of symmetry in fundamental physics. Proceedings of the National Academy of Sciences, 93(25):1423) 14256-14259.
Higgs, P. W. (1964). Broken symmetries and the masses of gauge bosons. Physical Review Letters, 13:58-5599.
Ho, J., Jain, A., and Abbeel, P. (2020). Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems.
Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., and Fleet, J. (2022). Video diffusion models. arXiv preprint a33442200..03558.
Hoogeboom, E., Satorras, V. G., Vignac, C., and Welling, M. (2022). Equivariant diffusion for molecule generation in 3d. International Conference on Machine Learning.
Kanwar, G., Albergo, M. S., Boyda, D., Cranmer, K., Hackett, D. C., Racanière, S., Rezende, J., and Shanahan, P. E. (2020). Equivariant flow-based sampling for lattice gauge theory. Physical Review Letters, 125:121601.
Kappen, H. J. (2005). Path integrals and symmetry breaking for optimal control theory. Journal of Statistical Mechanics: Theory and Experiment, 2005(11):P1100) P111111..
Kingma, D. P. and Welling, M. (2014). Auto-encoding variational bayes. In Bengio, Y. and LeCun, Y., editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings.
11

---

Köhler, J., Klein, L., and Noé, F. (2020). Equivariant flows: exact likelihood generative learning for symmetric densities. International Conference on Machine Learning.
Kong, Z., Ping, W., Huang. J., Zhao, K., and Catanzaro, B. (2020). Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:200909761..
Liu, H., Chen, Z., Yuan, Y., Mei, X., Liu, X., Mandic, D., Wang, W., and Plumbley, M. (2023). Au- dioldm: Text-to-audio generation with latent diffusion models. arXiv preprint arXiv:2301. 12503.
Liu, L.,, Ren, Y., Lin, Z., and Zhao, Z. (2022). Pseudo numerical methods for diffusion models manifolds. arXiv preprint arXiv:2202.09777.
Lucic, M., Kurach, K., Michalski, M., Gelly, S., and Bousquet, O. (2018). Are gans created equal? large-scale study. Advances Neural Information Processing Systems.
Nambu, Y. and Jona-Lasinio, G. (1961). Dynamical model of elementary particles based on an analogy with superconductivity. Physical review, 122(1):345.
Noether, E. (1971). Invariant variation problems. Transport Theory and Statistical Physics, 1(3):186- 207.
Rezende, and Mohamed, S. (2015). Variational inference with normalizing flows.
International
conference on machine learning, pages 1530-1538. PMLR.
Rezende, D. J., Mohamed, S., and Wierstra, D. (2014). Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning, pages 1278 1286. PMLR.
Rezende, D. J., Papamakarios, G., Racaniere, S., Albergo, M., Kanwar, G., Shanahan, P., and Cranmer, K. (2020). Normalizing flows on tori and spheres. International Conference on Machine Learning.
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X. (2016). Improved techniques for training gans. Advances in Neural Information Processing Systems.
Satorras, V. G., Hoogeboom, E.,, and Welling, M. (2021). ). E (n) equivariant graph neural networks. In International Conference on Machine Learning, pages 9323-9332.
Serengil, S. and Ozpinar, A. (2020). Lightface: A hybrid deep face recognition framework. In Innovations in Intelligent Systems and Applications Conference.
Singer, U., Polyak, A., Hayes, T., Yin, X., An, J., Zhang, S., Hu, Q., Yang, H., Ashual, O., Gafni, O.. al. (2022). Make-a-viden Text-to-video generation without text-video data. arXiv preprint MXXi::2009.1492
Sohl-Dickstein, J., Weiss, E., Muheswaranathan, N., and Ganguli. S. (2015). Deep unsupervised learning using nonequilibrium thermodynamics. International Conference on Machine Learning.
Song. J., Meng, C., and Ermon, S. (2020). Denoising diffusion implicit models. arXiv preprint DrXiv:2010.025500
Song. Y., Sohl- Bohl-Dicksteen, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, (2021). Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations.
Stanley, H. E. (1971). Phase transitions and critical phenomena, volume Clarendon Press, Oxford.
Täuber, U. C. (2014). Critical dynamics: a field theory approach to equilibrium and non-equilibrium scaling behavior. Cambridge University Press.
Thanh-Tume Tung, H. and Tran, T. (2020). Catastrophic forgetting and mode collapse in gans. In International Joint Conference Neural Networks, pages 1-10.
Van Den Oord, A., Kalchbrenner, N., and Kavukcuoglu, K. (2016). Pixel recurrent neural networks.
In International conference machine learning, pages 1747-1756. PMLR.
12

---

Vincent, P. (2011), A connection between score matching and denoising autoencoders. Neural Computation, 23(7):1661-1674
Weiler, M., Forré, P., Verlinde, E., and Welling, M. (2021). Coordinate independent convolutional networks-isometry and gauge equivariant convolutions on riemannian manifolds. arXiv preprint arXiv:2106.06020.
Worrall, D. and Brostow, G. (2018). Cubenet: Equivariance to 3d rotation and translation. European Conference Computer Vision, pages 567-584.
Xu, M., Yu, L., Song, Y., Shi, C., Ermon, s., and Tang, (2022). Geodiff: geometric dif- fusion model for molecular conformation generation. International Conference on Learning Representations.
13


Reviews:
[{"review_id": "CA4RE6Oerh", "reviewer": "Reviewer_sN4H", "summary": "The authors examine denoising diffusion models and their dynamics through the lense of spontaneous symmetry breaking.  They characterize the dynamics of the denoising process into two stages: in the beginning, samples orbit a central fixed point and after a critical amount of time denoising of a ‘selected’ sample begins.  They use insights gained from theory as well as practical examples to develop a novel fast sampling method they call “Gaussian late start” (GLS).\n", "questions": "This may be a silly question, but I was wondering if the authors thought if the datasets clustered and the sampler was initialized using the statistics of each cluster — do you expect the samples drawn to be similar to the cluster whose statistics were used to initialize the sampler?", "limitations": "As noted, GLS as is, is slightly impractical, but there is room for future research to develop fast approximate methods in the same vain.  ", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This was a very well written paper.  The authors motivate the issue very well, and they additionally illucidate the concept of spontaneous symmetry breaking in a way that is easy to understand.  \n\nOne of the biggest strengths, was the foundational example laid out in Sec. 3.1 — this example helped fill in a lot of gaps I had up until that point, and made me really appreciate the authors characterization of the denoising dynamics into two separate stages.  \n\nThe experiment of section 4.1 followed by its extension in section 4.2 do appear to give credence to thinking of the denoising dynamics as having a spontaneous symmetry breaking characteristic.  This also leads in nicely to the Gaussian late start sampling method, which is both intuitive, practical, and appears to work well.  \n\nOverall, the paper presents a compelling idea, motivates it through practical toy examples as well as theory, and then provide a novel efficient way of sampling.", "weaknesses": "It seems like this is not a problem constrained solely to diffusion models used for image generation, but for any data modality.  In that case, it might be good to empirically demonstrate  this for another data modality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We wish to thank the reviewer for the positive and insightful review. We are happy to hear that you found the one-dimensional example as insightful, as it certainly shaped our understanding of this fascinating phenomenon. To further solidify the intuition, in the updated manuscript we will also add a figure visualizing the fixed-points and showing how the score changes after the bifurcation (see Figure 2 in the attached pdf). \n\nIn the following, we will address some of your questions.\n\n\n### Point 1: Other modalities\nQ: *“It seems like this is not a problem constrained solely to diffusion models used for image generation, but for any data modality. In that case, it might be good to empirically demonstrate this for another data modality.”*  \n\nA: We agree that the phenomenon should not be limited to image datasets. The choice of the experiments mainly reflected our field of expertise. We also agree that it would be useful to see results in other modality, such as audio and language, and we are planning to pursue these research  directions in the future.\n### Point 2: Cluster initialization\nQ: *“This may be a silly question, but I was wondering if the authors thought if the datasets clustered and the sampler was initialized using the statistics of each cluster — do you expect the samples drawn to be similar to the cluster whose statistics were used to initialize the sampler?”*  \n\nA: Interesting point. Prior to the symmetry breaking, all clusters are merged into a single mode (around a fixed-point). Therefore, at least if we are talking about stochastic samplers, the initial value will not correlate with the final cluster assignments. The situation is different for deterministic samplers. In this case, the initialization scheme will likely work. However, we believe that it risks inducing significant overfitting. \n"}, {"review_id": "vpWmfDGX1R", "reviewer": "Reviewer_J7iR", "summary": "The authors of this work propose a new method to accelerate the sampling of diffusion models. First, the authors define as fixed points of the reverse process, points where the drift function is 0. Then, the authors claim that if two paths of stable fixed points interact, then it is the noise that can make the system jump from one path to the other. The authors argue that there are phase transitions happening in the sampling of diffusion models: until a certain time, the stable paths interact and hence the system jumps between potential paths. After a point in the reverse diffusion, there are no more intersections in the paths. Even if we don't initialize exactly in the path, the stability ensures that the differential equations are mean-reverting and will correct any introduced errors. Hence, the authors propose to start the sampling from a point in the middle of the diffusion and accelerate the sampling.", "questions": "* It seems to me that the method can be used *together* with other approaches for accelerating sampling (since it is only changing the initial point). It would strengthen the paper if the authors included experiments to show this.\n* Can the authors comment on whether we can find fixed points given a trained diffusion model?\n* I am puzzled by what happens to the story that this paper builds once we think about the deterministic samplers. In such settings, stable points would correspond to no movement at all. If the system becomes mean-reverting at some point, wouldn't that mean that we stop moving once we reach a fixed point? \n* It would strengthen the paper to include comparisons with EDM and other more recent sampling methods.\n* It would also help to include results for higher NFEs.", "limitations": "The authors adequately addressed the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The introduced framework is very interesting and novel. The example with the two Dirac functions is very pedagogical and its' generalization to a uniform distribution over many discrete points follows naturally. The experimental evidence supports that this phenomenon is indeed happening in trained diffusion models. The proposed method surpasses the baselines for a low number of function evaluations. The paper is well-written and the approach can be implemented easily. I expect that the paper will be of great interest to the research community and the audience of NeurIPS.", "weaknesses": "The theoretical analysis has some simplifying assumptions that it is not clear to what extent they alter what happens in practice. Specifically, the authors analyze uniform distributions over Dirac functions. The distributions observed in practice have a more complicated structure and it is not clear how we can argue anything about the fixed points of a model given access to its (learned) score-function.\n\nA weakness of the approach is that to start the sampling from some intermediate diffusion time, one needs to estimate the parameters of a multivariate Gaussian. This can be a particularly challenging task and it doesn't scale well for high dimensions because of the quadratic parameters needed for the covariance. The authors acknowledge this limitation.\n\nExperimentally, the authors mostly consider what happens in the low NFEs regime. More detailed comparisons would be useful. If the method performs poorly (relative to the baselines) for higher NFEs, it is important to acknowledge it. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We wish to thank the reviewer for the detailed and insightful review. In what follows, we will address the main issues and questions.\n\n### Point 1: Analytically tractable models\nQ: *“The theoretical analysis has some simplifying assumptions that it is not clear to what extent they alter what happens in practice. Specifically, the authors analyze uniform distributions over Dirac functions.”* \n\nA: The Z1 symmetry breaking on the on-dimensional model has been chosen due to its pedagogical value and the fact that it highlights all the main conceptual points. However, this form of binary symmetry breaking has relevance in many situations, for example, if the data spans two topologically separated manifolds. Of course, in the real model the Z1 symmetry will be a small subgroup of the overall symmetry group.\n\n### Point 2: Studying the fixed-point in trained models\n Q: *\"The distributions observed in practice have a more complicated structure and it is not clear how we can argue anything about the fixed points of a model given access to its (learned) score-function.” “Can the authors comment on whether we can find fixed points given a trained diffusion model?”*\n\nA: In theory, having access to a fully learned score function allows us to analyze its fixed points, which can be found by gradient descent. Specifically, this could be used to find the original fixed point, to determine its stability by evaluating the Hessian, and consequently to detect the onset of a symmetry breaking in a real-world model. However, unfortunately, this analysis is not going to be reliable in actual trained models since the fixed-point itself is outside the training range, and the shape of the potential in that region is not properly trained. This is due to the, perhaps paradoxical, fact that the fixed points have vanishingly low probability of being visited, as the samples concentrate in a fixed-variance annulus around them (see reply to Reviewer FFjV, Point 3). However, what ultimately matters is the shape of the potential, which we have studied directly in trained models along variance-preserving arcs connecting pairs of data points, where we found the expected change of shape (see Figure 1 and in the Supplementary Material see Figures 13-15). \n\n### Point 3: Scalability of the Gaussian late initialization\nQ: *\"A weakness of the approach is that to start the sampling from some intermediate diffusion time, one needs to estimate the parameters of a multivariate Gaussian. This can be a particularly challenging task and it doesn't scale well for high dimensions because of the quadratic parameters needed for the covariance. The authors acknowledge this limitation.\"*\n\nA: Indeed, in fact the GLS as used in the paper is meant as a proof of principle to showcase the importance of an appropriate initialization in order to preserve diversity. So said, it is rather straightforward to make the approach scalable either by PCA, Fourier analysis or, even better, by replacing the Gaussian initialization with another more efficient model such as a VAE. \n\n### Point 4: Performance for a large number of samples\nQ: *“Experimentally, the authors mostly consider what happens in the low NFEs regime. More detailed comparisons would be useful. If the method performs poorly (relative to the baselines) for higher NFEs, it is important to acknowledge it.”*  *“It would also help to include results for higher NFEs.”*\n\nA: Certainly, it is important to present results on higher NFEs. Thank you for pointing that out. In light of your feedback, we have incorporated extra experiments for 20, 50, and 100 denoising steps. You can view these results in Table 1 on the attached PDF. Notably, we observe the same behavior obtained at low NFEs remains at higher NFEs. Consequently, our gls scheme enhances performance over fast samplers, like DDIM and PNDM, as well as the standard DDPM. \n\n### Point 5: Deterministic samplers\nQ: *“I am puzzled by what happens to the story that this paper builds once we think about the deterministic samplers. In such settings, stable points would correspond to no movement at all. If the system becomes mean-reverting at some point, wouldn't that mean that we stop moving once we reach a fixed point?”*\n\nA: This is a very interesting point. Since the deterministic samplers are designed to exactly track the marginals of the stochastic sampler, a change in the stochastic dynamics will also appear in the deterministic case. However, the symmetry breaking looks very different from the point of view of the deterministic samplers. Specifically, in the deterministic ODEs, the correction term makes the drift vanish prior to the symmetry breaking. Therefore, in a deterministic sampler the first phase is ‘irrelevant’ not because of the mean-reverting dynamics, but just because there is no significant temporal evolution prior to that point. Therefore, with good approximation every point is a fixed-point in the first phase of the deterministic ODE. This can be clearly seen in Fig. 7(k) and Fig 16 (c,f,i,l,o)  in the Supplementary Material.\n\n\n\n"}, {"review_id": "K5A8GL9e5m", "reviewer": "Reviewer_kBdj", "summary": "The authors investigate theoretically and empirical when (in terms of diffusion timestep) certain symmetries in a diffusion process are broken, corresponding to when \"choices\" are made about which qualitative features a generated data point should have. Their findings suggest that there is no symmetry breaking occuring early in the process. They follow this up by showing that the early dynamics can be replaced by a sample from a multivariate Gaussian without meaningful degradation of the perceptual quality of generated data.", "questions": "See weaknesses (mainly the first one)", "limitations": "sufficiently addressed", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- This perspective on the diffusion process is novel as far as I know.\n- Understanding when in the diffusion process different symmetries are broken is likeliy to be very helpful for future work designing better diffusion-based models.\n- Their technique for better fast sampling yields surprising large improvements. Although they acknowledge that it does not immediately scale to high-dimensional data, it provides good support of their analysis.", "weaknesses": "- There is a fairly large gap between the simple example in Section 3.1, where we could reason analytically about when the symmetry is broken, to the realistic case in Section 3.2, where the only symmetries considered were the identity of each data point. Do the authors think it would be possible or meaningful to do this type of analysis on other features that could be extracted from the data (e.g. the identity of a person in an image)?\n- This analysis of symmetry breaking makes sense for \"discrete\" variables, like cluster index. It does not significantly help to provide insights about when values of \"continuous variables\" like e.g. background colour of an image are selected.\n- The proposed Gaussian initialization improves fast samplers in terms of FID and the authors show is helpful in terms of maintaining the identity of discrete variables, but it is important to note that replacing the early diffusion process with a Gaussian may still lead to large changes in the distribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback. Next, we will address the highlighted concerns and questions.\n\n### Point 1: Gap between theory and experiments\nQ: *“There is a fairly large gap between the simple example in Section 3.1, where we could reason analytically about when the symmetry is broken, to the realistic case in Section 3.2, where the only symmetries considered were the identity of each data point. Do the authors think it would be possible or meaningful to do this type of analysis on other features that could be extracted from the data (e.g. the identity of a person in an image)?”*\n\nA: There is indeed a substantial gap between what we can show theoretically in analytically tractable models with simple symmetry groups and the actual (often approximate) symmetries in real-world datasets. We definitely agree that a substantial amount of work can be done in characterizing the symmetry breaking for more complex symmetry groups (e.g. translational and rotational symmetries) and also, as the reviewer suggested, on symmetry transformation learned directly from a dataset. However, these are significant and complex endeavors that go beyond the scope of this first paper. While we understand the perspective that this might be seen as a weakness, in fact, we do believe this is a significant strength of our paper, as it opens several fascinating research directions.\n\n### Point 2: Discrete and continuous symmetries\nQ: *“This analysis of symmetry breaking makes sense for \"discrete\" variables, like cluster index. It does not significantly help to provide insights about when values of \"continuous variables\" like e.g. background colour of an image are selected.”*\n\nA: In the main text, we focused on discrete symmetry groups as they show the phenomenon in a clear and simple way. However, our symmetry breaking framework can deal with both continuous and discrete symmetry groups. In fact, we discussed the case of the continuous $SO(N)$ symmetry breaking in the Supplementary Material in section A.3. In order to make this point clear, we will move this analysis to the main text. We will also include a new paragraph discussing this point explicitly. \n\n### Point 3: Deviation from Gaussianity\nQ: *\"The proposed Gaussian initialization improves fast samplers in terms of FID and the authors show is helpful in terms of maintaining the identity of discrete variables, but it is important to note that replacing the early diffusion process with a Gaussian may still lead to large changes in the distribution\".*\n\nA: While it is true that the distribution will not be exactly Gaussian, our theoretical and, more importantly, experimental results show that the deviations are minor during the first phase.\nTo further address this point, we performed Gaussianity tests on forward samples as a function of time. Due to Anderson's theorem, both forward and generated samples have the same marginal statistics. The results of these tests across several datasets are graphically illustrated in Figure 1 on the attached pdf.\n\n\n\n\n"}, {"review_id": "ioLhAN4a5E", "reviewer": "Reviewer_FFjV", "summary": "The authors approach the reverse diffusion process in score-based generative modeling from the perspective of spontaneous symmetry breaking, which may explain why generation is minimally affected by \"late initializations\" (e.g. initializing x_t at t=T, but simulating the reverse process starting at some t < T). Motivated by this framework, the authors propose late-start scheme and show improved performance compared to some baselines.", "questions": "See \"Weaknesses\" section.", "limitations": "See \"Weaknesses\" section.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "**Exciting direction of study.** The authors propose an exciting perspective on the diffusion model in terms of spontaneous symmetry breaking in physical systems.\n\n**Initially promising empirical results.** The empirical results appear to be promising. (However, they do not presently provide much in the way of concrete proof that the symmetry breaking framework provides tangible improvements on generation speed or quality, see \"Weaknesses\").", "weaknesses": "**Motivation.** The motivation of the work should be made more clear. If it is to improve generation speed, the model should be compared against fast generation models, such as Model Distillation, Consistency Models, and EDM. If it is to generate better samples, then the model should be compared against standard diffusion models such as [4] or [5].\n\n**Empirical Results.** The empirical results are not very convincing. All models achieve FIDs that are far from state-of-the-art. It is not clear at this stage whether improvements come directly from the proposed approach, and whether the improvements will remain when models are tuned to be more competitive with respect to the state of the art.\n\n**Claims.** The claim (up to 3x FID improvements) feels overstated. The authors do not compare against state-of-the-art fast samplers, such as Model Distillation [1], Consistency Models [2], and EDM [3].\n\n**Soundness.** I have some concerns over the soundness of the claims in the paper.\n- The existence of stable points is only shown for toy examples (e.g. Secs. 3.1 and 3.2). Even in these toy examples, it is not clear to me how the stable points will affect the result of the reverse diffusion process for most initial conditions of the reverse process. Showing the existence of stable points at the origin and the original data points implies only the existence of a finite set of stable points, which are essentially probability zero set of the entire space. Diffusion models are initialized from a standard normal, so x_T can be very far from the origin (or any of the stable points). So is it true that the spontaneous symmetry breaking model is relevant here, even in theory?\n- While the second toy example is more realistic (Sec 3.2), and the authors claim that the assumptions required are mild, I do not believe they hold for any usual image distributions (e.g. MNIST, CIFAR10, ImageNet, CelebFaces, etc.).\n- Finally, the authors connect the theory of symmetry breaking to diffusion modeling with a single piece of empirical evidence: The FID of samples drawn with late start diffusion is relatively stable (up to a critical) point in time t_c. I can see how this is necessary condition to show the existence of symmetry breaking. But how is it sufficient?\n\nIn summary, while the theory of spontaneous symmetry breaking is very interesting and thought-provoking, I am not yet convinced that it is presently a useful perspective in diffusion-based generative modeling.\n\n**Minor edits.**\n\n39: the theory weak and electromagnetic -> the theory of weak and electromagnetic\n\n108: Hessian matrices (of what?)\n\n[1] Progressive Distillation for Fast Sampling of Diffusion Models. https://arxiv.org/abs/2202.00512\n\n[2] Consistency Models. https://arxiv.org/abs/2303.01469\n\n[3] Elucidating the Design Space of Diffusion-Based Generative Models. https://arxiv.org/abs/2206.00364\n\n[4] Denoising Diffusion Probabilistic Models. https://arxiv.org/abs/2006.11239\n\n[5] Score-Based Generative Modeling through Stochastic Differential Equations.https://arxiv.org/abs/2011.13456", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are thankful to the reviewer for the comprehensive and insightful feedback. In what follows, we'll address the main concerns and questions. \n\n### Point 1: Motivation\nQ: *“The motivation of the work should be made more clear. If it is to improve generation speed, the model should be compared against fast generation models, such as Model Distillation, Consistency Models, and EDM. If it is to generate better samples, then the model should be compared against standard diffusion models such as [4] or [5].”*\n\nA: We acknowledge that we were not entirely clear in the original manuscript and we will change the text accordingly. Our primary goal is to demonstrate that spontaneous symmetry breaking is a near-universal consequence of the dynamics of generative diffusion models. Our result establishes a direct connection between a dominant class of generative models in machine learning and a large body of theory and numerical methodologies developed in physics, which can be leveraged to improve our understanding and mastery of generative AI. \n\nAs we argue in the general reply to all reviewers, the goal of our empirical study on fast samplers is to show how this theoretical understanding can be leveraged in practice and it can indeed lead to rather impressive results. However, we believe that the level of optimization needed to optimize SOTA methods goes beyond the scientific scope of this paper.\n\n\n### Point 2: Comparison with distillation models\nQ: *“the model should be compared against fast generation models, such as Model Distillation, Consistency Models, and EDM”*\n\nA: Distillation models aim at skipping the generative dynamics in order to obtain one or few step sampling, often using a re-trained model. Since our aim is to study the generative dynamics, we decided to not focus on this class of methods. So said, it is worth noting that distillation models still rely on the DDIM sampler in their training process, a sampler that we have employed and improved upon in our paper. \n\n### Point 3: Relevance of the fixed-points\nQ: *“The existence of stable points is only shown for toy examples (e.g. Secs. 3.1 and 3.2). Even in these toy examples, it is not clear to me how the stable points will affect the result of the reverse diffusion process for most initial conditions of the reverse process. Showing the existence of stable points at the origin and the original data points implies only the existence of a finite set of stable points, which are essentially probability zero set of the entire space. Diffusion models are initialized from a standard normal, so x_T can be very far from the origin (or any of the stable points). So is it true that the spontaneous symmetry breaking model is relevant here, even in theory?”*\n\nA: This is a very insightful and subtle point. Indeed, the reviewer is entirely correct in pointing out that, at least in the early stage, the fixed-point in itself does not seem to be very relevant. What actually matters is the shape of the potential around the fixed-point. The main reason to study the bifurcation in the fixed-points is that it implies a dramatic change of shape in the potential, which splits into multiple wells, each representing a subset of the original symmetry group. This change of shape is the hallmark of a spontaneous symmetry breaking, as visualized in Figure 2 in the attached pdf.\n\nAs a secondary point, note that for  $T-t <T- t_c$ the fixed-points are not always isolated. In fact, if the data spans a d-dimensional manifold with uniform probability, for $T-t$ tending to $0$, each point in the manifold becomes a fixed-point. Finally, in a discrete dataset, for $T-t$ tending to zero, the fixed-points will generally acquire a non-zero measure since the magnitude of the score pointing at them tends to infinity. \n\n### Point 4: Assumptions in discrete datasets\nQ: *“While the second toy example is more realistic (Sec 3.2), and the authors claim that the assumptions required are mild, I do not believe they hold for any usual image distributions (e.g. MNIST, CIFAR10, ImageNet, CelebFaces, etc.).”* \n\n\nA: We would like to point out that we did not claim that these assumptions are met in real datasets, but instead that it is straightforward to induce them (at least approximately) by normalization. For example, subtracting the mean (centering) is commonly done as a preparatory step in many algorithms without loss of generality, since the mean can be summed after generating the centered data. In general, the centering assumption is only needed to make sure that the fixed-point is the origin and to make the analysis simpler. Concerning the constraint on the Euclidean norm, it is well known that the Euclidean norm often concentrates around a single value in high-dimension, meaning that the data approximately ‘live’ in a hyper-spherical annulus. Generally speaking, while the constraint is not exactly met, we think that it is relevant for understanding high-dimensional data. \n\n\n### Point 5: Empirical evidence\nQ: *“Finally, the authors connect the theory of symmetry breaking to diffusion modeling with a single piece of empirical evidence: The FID of samples drawn with late start diffusion is relatively stable (up to a critical) point in time t_c. I can see how this is necessary condition to show the existence of symmetry breaking. But how is it sufficient?”* \n\nA: The late start FID curves are not the only empirical result in the paper. Using the trained network, we also directly evaluate the potential on arcs connecting two trajectories and we show that the potential has a single minimum initially and then splits into two separate minima. This can be seen in Figure 1 and in the Supplementary Material in Figures 13-15. These changes in the shape of the potential provide very direct empirical evidence of the symmetry breaking phenomenon in trained models.\n\n"}, {"review_id": "UvT9TjvudK", "reviewer": "Reviewer_Z6e7", "summary": "The paper explores the generative dynamic of diffusion models. It proposes spontaneous symmetry breaking and Gaussian late initialization scheme and they achieve better fidelity and diversity on the generated images.", "questions": "[Q1] In Table 1, why the dataset and denoising step (n) is repeated? You can put them in the first column and it might be enough. So, instead of making subtables, can you make them a single table? \n\n[Q2] In Table 1 and Figure 7 why you didn't include other generative models such as GANs in comparison?\n\n[Q3] Is Table 1 an ablation study to show the effect of the gls? If so, where is the fidelity comparison of the proposed approach?\n\n[Q4] In the discussion of Table 1, you do not discuss the results so as to indicate the spontaneous symmetry breaking. What is the reason for this?", "limitations": "[L1] It is only applicable to the diffusion models. \n\n[L2] The paper ignores the other type of generative models in both related work and experiments. It is required to compare the proposed approach with existing generative models.\n\n[L3] The paper does not discuss the drawback of the proposed approach.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "[S1] The paper supports the claims in both theoretical and empirical.\n\n[S2] The paper successfully convince the proposed approach improves diversity by comparing.", "weaknesses": "[W1] They do not discuss the limitations and drawbacks of their model.\n\n[W2] The other generative models such as VAE, GAN, regression models, etc. do not discuss. At least, it is expected to mention them in the related work section as they claim that the proposed approach has a broader impact on generative models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewers' feedback and are happy to hear that our work supports our claims both theoretically and empirically. In the following, we will address the main issues raised by the reviewer.\n\n### Point 1: Discussion of other generative models\nQ: *“The other generative models such as VAE, GAN, regression models, etc. do not discuss. At least, it is expected to mention them in the related work section as they claim that the proposed approach has a broader impact on generative models.”* \n\nA: In the revised version, we will add a sentence discussing other generative models in the introduction and a more extensive discussion about how to generalize these insights to other models in the discussion section.\n\nWe indeed focused our attention on generative diffusion models. This is due to the fact that these models are radically different from other generative approaches such as VAEs and GANs and it is not straightforward to extend these insights to these other models. However, given the current importance of diffusion models in the generative modeling literature, we believe that the insights developed by carefully analyzing these models can have a broad impact on generative modeling. \n\n### Point 2: Experimental comparison with other generative models\nQ: *“In Table 1 and Figure 7 why you didn't include other generative models such as GANs in comparison?”* \n\nA: Our aim is 1) to demonstrate that spontaneous symmetry breaking phenomena are ubiquitous in the dynamics of generative diffusion models and 2) to show that this insight can lead to faster, higher quality and more diverse sampling in diffusion models.\n\nGiven these aims, it is unclear what we would learn by including other non-diffusion baselines. The relative performance of these models when compared with diffusion models has already been discussed in the literature [1, 2] and it is orthogonal to our claims.\n\n[1] Diffusion Models Beat GANs on Image Synthesis, https://arxiv.org/abs/2105.05233  \n[2] Improved Denoising Diffusion Probabilistic Models, https://arxiv.org/abs/2102.09672 \n\n\n### Point 3: Table 1\nQ: *“In the discussion of Table 1, you do not discuss the results so as to indicate the spontaneous symmetry breaking. What is the reason for this?”* \n\nA: Unfortunately, we are not sure to have understood the question. We show the existence of the symmetry breaking in the FID curves in Fig. 4 and in the splitting of the potential obtained from the model in Fig.1 (right side) and Supps in Fig. 13-14. Table 1 reports the result of the late initialization experiment. As we explain in the paper, the existence of the symmetry breaking predicts that late initialization will produce more efficient fast samples, which is confirmed in Table 1. We hope this clarifies it. \n"}]

Author rebuttal:
Dear reviewers,

We wish to thank you for the detailed analysis of our paper and for all the encouraging and constructive comments. We are extremely grateful for your work, as it will make our manuscript substantially better. Several important questions were raised, and we provided extensive answers under the individual reviews. In particular, we would like to point out to the discussion in the reply to reviewer  FFjV (point 3) about the importance of the fixed points and the analysis of the deterministic samplers in the reply to reviewer J7iR (point 5). Upon acceptance, we will incorporate these discussions in the updated manuscript. 

In what follows, we will outline our conclusions from the reviews, addressing both the strengths and weaknesses of our work as pointed out by the reviewers. We have also included a summary of all the additional analysis that we conducted during the rebuttal period in response to questions and observations raised by the reviewers.

### **Main strengths**  
We are happy to see that all reviewers agreed that our main claim is well supported by strong theoretical and empirical evidence. This is corroborated by the fact that two reviewers (sN4H and kBdj) marked the soundness of the paper as excellent, a distinction that makes us proud, given the ambitious nature of our claims. We are also pleased to see that a majority of reviewers agree that our work is novel and important, as expressed, for example, by the following quotes from reviewer J7iR and kBdj:

> *“I expect that the paper will be of great interest to the research community and the audience of  NeurIPS.”*  (reviewer J7iR) ,

> *“Understanding when in the diffusion process different symmetries are broken is likely to be very helpful for future work designing better diffusion-based models.”* (reviewer kBdj) ,

and even by the more critical reviewer FFjV

> *“the theory of spontaneous symmetry breaking is very interesting and thought-provoking”* (reviewer FFjV) .


### **Main weaknesses**
The main concern, most prominently expressed by reviewer FFjV, is that the improvements in sample quality and diversity due to the Gaussian late initialization scheme might not generalize to state-of-the-art (SOTA) fast-sampling approaches such as progressive distillation and consistency models. 

While we acknowledge that this is a fair criticism, we would like to emphasize that the primary motivation of this paper is not to propose a SOTA fast sampler. The primary goal of the paper is instead to demonstrate that the generative dynamics of diffusion models can be understood as spontaneous symmetry breaking, a phenomenon that is ubiquitous  in physical systems.  This is in itself a very ambitious endeavor as it establishes a direct connection between generative AI and some of the deepest aspects of fundamental physics (e.g. the standard model and the statistical mechanics of critical phenomena), which might in turn lead to substation methodological and theoretical developments in both fields.

The purpose of our sampling experiments is indeed to demonstrate that understanding this phenomenon can directly lead to algorithmic improvements, thereby providing evidence for its usefulness.  However, reaching the boundaries of the state-of-the-art requires a level of optimization that we believe goes beyond the scope of our paper. In this sense, our sampling experiments should be interpreted as a proof of principle and a starting point for future developments. 

### **Summary of new experimental results**
Several questions and observations raised by the reviewers motivated us to run several new experiments for this rebuttal period, which will end up in the revised version of the manuscript.

In the accompanying PDF, we provide expanded analyses addressing our reviewers' questions. 

- Reviewer kBdj concern about the Gaussianity nature of the approximate distribution is addressed in Figure 1, which provides a Gaussianity analysis via the Shappiro-Wilk test. The figures corroborate the validity of a Gaussian distribution up until the symmetry breaking’s critical point, beyond which the distribution rapidly ceases to be Gaussian.
- In answer to the comments made by reviewer J7iR, Table 1 expands on the robustness of our improvements in the context of higher numbers of function evaluations (NFE). Here, we present results for 20, 50, and 100 denoising steps for datasets including MNIST, CIFAR10, and Celeba64. These results confirm that our improvements persist in various NFE regimes.
- Lastly, to address reviewer FFjV concern on the importance of the fixed-points, Figure 2 provides a visualization of the bifurcation of the fixed-points (estimated numerically) in the one-dimensional model, together with a visualization of the vector field provided by the score function. The figure shows how the bifurcation is associated with a dramatic shift in the vector field, which determines the dynamics of the system away from the fixed-points themselves.



Meta-review: Most reviewers appreciate the scientific value of revealing a nature in the working mechanism of diffusion model under the view of a dynamical system, and the practical value of a derived faster sampling implementation. There are also concerns on the assumptions for the analysis, and practicality of the proposed method. The ratings indicate the contributions overweigh the concerns. I hence recommend accept for this submission.
Decision: Accept (poster)