SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(b) DDIM
Dataset	n	gls-PNDM	PNDM
MNIST	10	5.02	14.36
5	5.11	21.22
3	38.23	154.89
CIFAR10	10	5.90	8.35
5	9.55	13.77
3	34.20	103.11
CelebA32	10	2.88	4.92
5	4.2	6.61
3	28.60	235.87
Imagenet64	10	27.9	28.27
5	33.35	34.86
3	50.92	70.58
CelebA64	10	6.80	8.03
5	9.26	10.26
3	51.72	171.75
(c) PNDM
Table 1: Summary of findings regarding image generation quality, as measured by FID scores. The performance of the stochastic DDPM sampler (a) is compared to the deterministic DDIM (b) and PNDM (c) samplers in the vanilla case, as well as our Gaussian late start initialization scheme denoted as "gls". Results are presented for 3, 5, and 10 denoising steps (denoted as "n") across diverse datasets.
4.2 Empirical analysis of the potential function in trained diffusion models
The analysis of the FID curves offers indirect evidence of the existence of a critical time that demarcates the generative dynamics of diffusion models trained on real datasets. In order to obtain direct evidence, we need to study the potential associated to trained models. Unfortunately, studying the stability of the initial fixed-point in a trained model is challenging given the high dimensionality and the fact that its location is not available analytically. To reduce the dimensionality, we analyzed how the potential changes across variance-preserving interpolating curves that connect generated images. We defined an interpolating curve as  $\pmb{x}(\alpha, t) = \cos(\alpha)\pmb{x}_1(t) + \sin(\alpha)\pmb{x}_2(t)$ , where  $\pmb{x}_1(t)$  and  $\pmb{x}_2(t)$  are two sampled generative paths. Up to constant terms, the potential along the path  $\tilde{u}(\alpha, t) = u(\pmb{x}_{\alpha}, t)$  can be evaluated from the output of the network (i.e., the score) using the fundamental theorem of line integrals (see supplemental section B.2.1). An example of these potential curves is shown in Figure 1b (top), as predicted by the theory, the potential has a convex shape up to a time and then splits into a bimodal shape. A visualization of the average potentials for several datasets is provided in the supplemental section B.2.2. Interestingly, this pairwise splits happens significantly later than the critical transition time observed in the FID analysis. This suggests that, in real datasets, different spontaneous symmetry breaking phenomena happen at different times, with an early one causing a major distributional shift in the overall dynamics (as visible from the FID curves) and later ones splitting the symmetry between individual data-points.
5 Improving the performance of fast samplers
In the following, we leverage the spontaneous symmetry breaking phenomenon in order to improve the performance of fast samplers. Since the early dynamics is approximately linear and mean-reverting, the basic idea is to initialize the samplers just before the onset of the instability. This avoids a wasteful utilization of denoising steps in the early phase while also ensuring that the critical window of instability has been appropriately sampled. Unfortunately, while we can prove the existence of a spontaneous symmetry breaking in a large family of models, we can only determine its exact time in highly simplified toy models. We therefore find an "optimized" starting time empirically by generating samples for different starting times (with a fixed number of equally spaced denoising steps) and selecting the one that gives the highest FID scores. From our theoretical analysis it follows that, while the distribution of the particles can drift from  $\mathcal{N}(\mathbf{0},I)$  before the critical time point, it generally remains close to a multivariate Gaussian distribution. Leveraging these insights, we propose a Gaussian initialization scheme to address the distributional mismatch caused by a late start initialization. This scheme involves estimating the mean and covariance matrix of noise-corrupted
(a) DDPM  $n = 5$ , FID=48.38
(b) DDPM  $n = 10$ , FID=36.66
Figure 5: Comparison of stochastic DDPM samplers on CelebA64 with varying denoising steps. Subfigures (a) and (c) represent the generative model performance for 5 denoising steps, while (b) and (d) showcase the results for 10 denoising steps. The DDPM was initialized with the common standard initialization point  $s_{start} = 800$  for 5 steps and  $s_{start} = 900$  for 10 steps. Notably, our Gaussian late start initialization (gls-DDPM) with  $s_{start} = 400$  for both 5 and 10 denoising steps demonstrates significant improvements in FID scores and diversity, leveraging spontaneous symmetry breaking in diffusion models.
(c) gls-DDPM  $n = 5$ , FID=31.24
(d) gls-DDPM  $n = 10$ , FID=23.79
dataset at the initialization time and utilizing the resulting Gaussian distribution as the initialization point. By using this fitted Gaussian as the starting point, we can obtain an initial sample and run the denoising process from there instead of using a sample from  $\mathcal{N}(\mathbf{0},I)$ . We refer to this method as "Gaussian late start" (gls) and present results for stochastic dynamics in Table 1a and deterministic dynamics in Tables 1b and 1c, using DDIM and PNDM samplers respectively. In our experiments on several datasets, we found that in both cases, the Gaussian late start always increases model performance when compared with the baseline samplers. The performance boost is striking in some datasets, with a 2x increase in CelebA 32 for 10 denoising steps and a 3x increase for 5 denoising steps. As evidenced by the results showcased in Figure 5, gls-DDPM boost performance over vanilla DDPM for 10 and 5 denoising steps. For more detailed information on fast samplers, please refer to the supplemental section C. This section also includes extended results for higher number of denoising steps, as detailed in Table 6. Figure 19 provides empirical evidence of the Gaussian nature of the initial distribution via the Shapiro-Wilk test, which remains valid until the critical point.
5.1 Diversity analysis
Diffusion models that can quickly generate high-quality samples are highly desirable for practical applications. However, the use of fast samplers can lead
to a reduction in the diversity of generated samples. For instance, the analysis shown in Figure 7 using the DDIM sampler with only 5 denoising steps on the CelebA64 dataset revealed a decrease in "race" diversity, with a bias towards the dominant race (white) and a reduction in the coverage of other races. Our theoretical analysis suggests that achieving high generative diversity relies on reliably sampling a narrow temporal window around the critical time, since small perturbations during that window are amplified by the instability and thereby play a central role in the determination of the final samples. This suggests that our Gaussian late initialization should increase diversity when compared with the vanilla fast-sampler, since
Figure 6: Sample entropy as function of DDIM denoising steps.
(a) Training set
(b) DDPM-1000
Figure 7: "Race" diversity analysis on CelebA64 over 50,000 generated samples by (c) gls-DDIM and (d) DDIM samplers with 5 denoising steps. Results obtained on (a) training set and (b) DDPM using 1000 denoising steps are provided for reference. Corresponding samples obtained by each set are shown on top of the pie charts.
(c) gls-DDIM-05
(d) DDIM-05
the Gaussian approximation is reliable prior to the change of shape in the potential. Indeed, our proposed late initialization method (gls-DDIM-05) was able to significantly improve sample diversity, even for a small number of denoising steps, e.g., 3, 5, and 10 sampling steps (see Figure 6). For comparison, we also provided the diversity obtained in the training set and from a typical sampling of DDPM with 1000 sampling steps (DDPM-1000). In supplemental section D, we also provided a full analysis of attributes such as age, gender, race, and emotion by running a facial attribute analysis over 50,000 generated images using the deep face library (Serengil and Ozpinar, 2020).
Efficient sampling in diffusion models. In recent years, research has been focused on accelerating the slow sampling process in diffusion models, resulting in two categories of efficient samplers: learning-free and learning-based. Among the learning-free samplers, Denoising Diffusion Implicit Models (DDIM) (Song et al., 2020) and Pseudo Numerical Diffusion Models (PNDM) (Liu et al., 2022) have gained considerable attention. DDIM introduced a non-Markovian class of diffusion processes to achieve faster sampling, while PNDM proposed an enhancement to traditional numerical methods through the use of pseudo numerical methods. Both DDIM and PNDM implicitly employ a late start initialization using a linearly spaced open interval running from 1 to  $T$ , with steps determined by the required number of denoising steps, and consequently not including  $T$ , but use the sample at  $x_{T}$  as the starting point. The choice of the starting point has been proposed based on empirical results, and to the best of our knowledge, our work is the first to theoretically explore the relevance of this starting point and systematically suggest a way to choose it.
Diffusion and spontaneous symmetry breaking in optimal control theory. The concept of spontaneous symmetry breaking has been studied in optimal control theory, with the work of Kappen (2005), who showed the existence of spontaneous symmetry breaking in both stochastic and deterministic optimal control problems. Spontaneous symmetry breaking in the cost as a function of time implies a "delayed choice" mechanism for stochastic control, where taking a decision far in the future is sub-optimal due to high uncertainty. Instead, the optimal policy is to steer between the targets until the critical time one should aim for one of the targets. For  $T < t_{c}$  (far in the past) it is best to steer towards  $x = 0$  (between the targets) and delay the choice which slit to aim for until later, when the uncertainty is less close to the terminal state. The relationship between optimal control theory and diffusion models has recently been explored by the work of Berner et al. (2022), who demonstrated that the reverse-backward SDE is directly related to the Hamilton-Jacobi-Bellman equation for the time-inverse log-density through a Hopf-Cole transformation. This theoretical link suggests that the spontaneous symmetry breaking phenomena that we have identified here are analogous to delayed choice phenomena in optimal control.
Encoding symmetries in generative models. Symmetries plays a central role in any area of physics, revealing the foundational principles of nature and enabling the formulation of conservation laws (Gross, 1996; Noether, 1971). A substantial body of deep learning literature has capitalized on this inherent property of nature by encoding equivariance to symmetry transformations constrains into deep neural networks (Cohen and Welling, 2016; Worrall and Brostow, 2018; Bogatskiy et al.,
2020; Cesa et al., 2021; Weiler et al., 2021) allowing to construct more efficient and physically interpretable models with minimized learnable parameters. Recently, this attribute has garnered attention in the realm of generative models, spurring interest in integrating both internal (Gauge) and external (space-time) symmetries (Kanwar et al., 2020), particularly when sampling symmetric target densities (Köhler et al., 2020). While the focus has been in incorporating symmetries in normalizing flows (Boyda et al., 2021; Köhler et al., 2020; Rezende et al., 2020; Satorras et al., 2021), there is already an increase interest in encoding symmetries in diffusion models Xu et al. (2022); Hoogeboom et al. (2022). Our work, along with previous research, acknowledges the importance of the existence of symmetry groups for generative modeling. However, the key distinction in our approach lies in emphasizing the relevance of spontaneous symmetry breaking for particle generation rather than encoding symmetry constraints.
Comparison with other generative models. While diffusion models differ conceptually from other deep generative approaches like Variational Autoencoders (VAEs) (Kingma and Welling, 2014; Rezende et al., 2014), GANs (Goodfellow et al., 2014), autoregressive models (Van Den Oord et al., 2016), normalizing flows (Dinh et al., 2014; Rezende and Mohamed, 2015), and Energy-Based Models (EBMs), they all share certain operational traits. Specifically, all these models—explicitly or implicitly—start from an isotropic equilibrium state and evolve toward a data-equilibrium state. Beyond this, an underlying theme across these generative models is the idea of pattern formation, marked by a transition from a high-symmetric state to a low-symmetric state. This commonality suggests the potential for leveraging the concept of symmetry breaking for performance improvement. For instance, implementing symmetry breaking mechanisms in GANs could address the well-known problem of mode collapse, thereby enhancing their generative performance. This idea is especially relevant for EBMs, which, like diffusion models, strive to learn an energy function that assigns lower values to states close to the data distribution. Consequently, the phenomenon of spontaneous symmetry breaking could offer key insights for refining the generative capabilities of all these models.
7 Discussion and conclusions
In this work, we have demonstrated the occurrence of spontaneous symmetry breaking in diffusion models, a phenomenon of significant importance in understanding their generative dynamics. Our findings challenge the current dominant conception that the generative process of diffusion models is essentially comprised of a single denoising phase. Instead, we have shown that the generative process is divided into two (or more) phases, the early phase where samples are mean-reverted to a fixed point, and the "denoising phase" that moves the particle to the data distribution. Importantly, our analysis suggests that the diversity of the generated samples depends on a point of "critical instability" that divides these two phases. Our experiments demonstrate how this phenomenon can be leveraged to improve the performance of fast samplers using delayed initialization and a Gaussian approximation scheme. However, the algorithm we proposed should be considered as a proof of concept since the full covariance computation does not scale well to high-resolution images. Nevertheless, these limitations can be overcome either by using low-rank initialization or by fitting the Gaussian initialization on a lower-dimensional space of Fourier frequencies. More generally, our work sheds light on the role that symmetries play in the dynamics of deep generative models. Following this research direction, existing theoretical frameworks, such as optimal control and statistical physics, can be leveraged to make advancements in this field towards more efficient, interpretable, and fair models.
Our research has important implications for advancing deep generative models, which are essential for understanding complex phenomena like human creativity, molecular formation and protein design. However, the real-world usage of generative diffusion models can pose several risks and reinforce social biases. We hope that the insights and results offered in our work will have a positive social outcome by helping to create less biased models.
We would like to thank Eric Postma for his helpful discussions and comments. Gabriel Raya was funded by the Dutch Research Council (NWO) as part of the CERTIF-AI project (file number 17998).
Vincent, P. (2011). A connection between score matching and denoising autoencoders. Neural Computation, 23(7):1661-1674.
Weiler, M., Forre, P., Verlinde, E., and Welling, M. (2021). Coordinate independent convolutional networks-isometry and gauge equivariant convolutions on riemannian manifolds. arXiv preprint arXiv:2106.06020.
Worrall, D. and Brostow, G. (2018). Cubenet: Equivalence to 3d rotation and translation. European Conference on Computer Vision, pages 567-584.
Xu, M., Yu, L., Song, Y., Shi, C., Ermon, S., and Tang, J. (2022). Geodiff: A geometric diffusion model for molecular conformation generation. International Conference on Learning Representations.