SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We start by plotting the function  $\mathbb{E}_{q(\mathbf{x})}[\mathcal{L}(\lambda ;\mathbf{x})]$ , for differences choices of  $n = 1,\ldots,8$ , as shown in Figure 5a. Next, we plot  $\mathbb{E}_{q(\mathbf{x})}\left[\frac{d}{d\lambda}\mathcal{L}(\lambda ;\mathbf{x})\right]$ , for each  $n$ , in Figure 5b. The more bits included in the data, the more information contained by  $\mathbb{E}_{q(\mathbf{x})}\left[\frac{d}{d\lambda}\mathcal{L}(\lambda ;\mathbf{x})\right]$  (measured by the area under each curve). We can further visualize the contribution of each additional added bit to the loss, by subtracting the curve for  $n - 1$  bits from the curve for  $n$  bits, as displayed in Figure 5c. The area under each of these curves is exactly 1 bit. Interestingly, as more bits being added, the contribution to the loss gradually shifts from low-SNR regions to high-SNR regions.
Now the connection between low-bit training (e.g., 5-bit) and the weighted training objective becomes clear: If we want to "shut down" the last three curves in Figure 5c, a way is to have a sigmoidal weighting function that goes down sharply between  $\lambda = 7.5$  and  $\lambda = 10$ . In fact, the 5-bit unweighted loss curve is very similar to the 8-bit loss curve, when using the following weighting function:
$$
w (\lambda) = F _ {\mathcal {N}} \left(\left(- 2 (\lambda - 8. 4)\right)\right) \tag {146}
$$
where  $F_{\mathcal{N}}$  is the CDF of a standard Gaussian. See Figure 5e for the comparison of the 5-bit unweighted loss and the 8-bit weighted loss, and Figure 5d for the visualization of the above weighting function ('5-bit-like weighting'). Interestingly, this weighting function gives much more weight to low noise levels than the weighting functions used in this paper (i.e. the other curves shown in Figure 5d).
J Fourier analysis of Gaussian noise perturbation
Recall Theorem 1 showed that assuming monotonic weightings, the weighted training objective is equivalent to the ELBO with data augmentation, namely Gaussian noise perturbation. One may wonder, why Gaussian noise perturbation can help improve perceptual quality. In this section, we try to answer this question by lifting Gaussian noise perturbation to the frequency domain.
Given a clean natural image  $\mathbf{x}$  and a Gaussian white noise image  $\epsilon$ , the Gaussian-noise-perturbed data is given by  $\mathbf{z}_{\lambda} = \alpha_{\lambda}\mathbf{x} + \sigma_{\lambda}\epsilon$ . Denote the Fourier transform operation as  $\mathcal{F}(\cdot)$ . Since  $\mathcal{F}(\cdot)$  is a linear operation, we have
$$
\mathcal {F} (\mathbf {z} _ {\lambda}) = \alpha_ {\lambda} \mathcal {F} (\mathbf {x}) + \sigma_ {\lambda} \mathcal {F} (\boldsymbol {\epsilon}). \tag {147}
$$
To understand how noise affects the data, we can compare the power spectra of  $\alpha_{\lambda}\mathcal{F}(\mathbf{x})$  and  $\sigma_{\lambda}\mathcal{F}(\epsilon)$  which is defined as the square amplitude of the Fourier transform:  $S(\cdot) = |\mathcal{F}(\cdot)|^2$ . More specifically, given the power spectrum  $S$  of an image, we can take the circular average of  $S$  for the Fourier
(a)  $\mathbb{E}_{q(\mathbf{x})}[\mathcal{L}(\lambda ;\mathbf{x})]$  as a function of  $\lambda$
(b)  $\mathbb{E}_{q(\mathbf{x})}\left[\frac{d}{d\lambda}\mathcal{L}(\lambda ;\mathbf{x})\right]$  as a function of  $\lambda$
(c) Subtraction of the curve for  $n - 1$  bits from the curve for  $n$  bits in Fig.5b.
(d) Weighting functions, with a weighting resembling 5-bit training.
(e) Unweighted 5-bit loss versus weighted 8-bit loss.
Figure 5: Low-bit training analysis. See Section I.1 for description.
components of the same spatial frequency, such that it becomes a function of the spatial frequency. Figure 6 displays the average power spectra of  $\alpha_{\lambda}\mathcal{F}(\mathbf{x})$  and  $\sigma_{\lambda}\mathcal{F}(\epsilon)$  as functions of the spatial frequency, for different  $\lambda$ . As  $\lambda$  goes from high to low, i.e., gradually perturbing the data with more noise, the average power spectrum of  $\sigma_{\lambda}\mathcal{F}(\epsilon)$  (orange curve) first exceeds the one of  $\alpha_{\lambda}\mathcal{F}(\mathbf{x})$  (blue curve) in the high frequency region, and then followed by the low frequency region. That is, the high frequency components of the data are destroyed more quickly than the low frequency components. When the data is perturbed with small amounts of noise, only high frequency details are significantly removed, and most low frequency content remains. Therefore, given that the model capacity is limited, learning data with Gaussian noise perturbation help the model focus more on low frequency components of the data, which often correspond to high level content and global structure that are more crucial to human perception. This analysis can also be applied to understand the forward process of diffusion models.
Gaussian noise perturbation
Figure 6: The circular average power spectrum as a function of spatial frequency, for the scaled clean image and scaled Gaussian noise image. From the rightmost to the leftmost subfigure, more noise is gradually added.
To make the claim even clearer, we can compute the log signal-to-noise-ratio (log SNR)  $\tilde{\lambda}$  between the power spectra of  $\alpha_{\lambda} \mathcal{F}(\mathbf{x})$  and  $\sigma_{\lambda} \mathcal{F}(\epsilon)$ , as a function of  $\lambda$ . Figure 7 shows the average value of that log ratio around different frequency regions. Given a fixed  $\lambda$ ,  $\tilde{\lambda}$  is higher for low frequency regions, so that the data are less destroyed by the noise.
Figure 7: Log-SNR  $(\tilde{\lambda})$  of the power spectra of the scaled clean image and scaled Gaussian noise image, as a function of log-SNR  $(\lambda)$  in the data space. Given a fixed  $\lambda$ ,  $\tilde{\lambda}$  is higher around low frequency regions.
It is important to emphasize that our empirical results, like other deep learning approaches, depend on the choice of hyper-parameters. A change in, for example, the dataset or spatial resolution will generally require re-tuning of optimization hyperparameters, architectural choices and/or weighting functions. Such re-tuning can be time consuming and costly.
L Broader impact
While our work primarily focuses on theoretical developments in the understanding and optimization of diffusion models, the advancements could have broader implications, some of which could potentially be negative. The development of more efficient and effective generative models could, on one hand, propel numerous beneficial applications, such as art and entertainment. However, it is also worth acknowledging the potential misuse of these technologies.
One notable concern is the generation of synthetic media content, for example to mislead. These fraudulent yet realistic-looking images and videos could be used for spreading disinformation or for other malicious purposes, such as identity theft or blackmail.
Regarding fairness considerations, generative models are typically trained on large datasets and could therefore inherit and reproduce any biases present in the training data. This could potentially result
in unfair outcomes or perpetuate harmful stereotypes if these models are used in decision-making processes or content generation.
Mitigation strategies to address these concerns could include the gated release of models, where access to the model or its outputs is regulated to prevent misuse. Additionally, the provision of defenses, such as methods to detect AI generated media, could be included alongside the development of the generative models. Monitoring mechanisms could also be implemented to observe how a system is being used and to ensure that it learns from feedback over time in an ethical manner.
These issues should be considered when applying the techniques we propose. The community should strive for an open and ongoing discussion about the ethics of AI and the development of strategies to mitigate potential misuse of these powerful technologies.
M Samples from our model trained on  $512 \times 512$  ImageNet
Below we provide random samples from our highest-resolution (512x512) model trained on ImageNet. We did not cherry-pick, except that we removed depictions of humans due to ethical guidelines. Samples in Figures 8 and 9 are generated without guidance, while samples in Figures 10 and 11 are generated with guidance strength 4.
Figure 8: Random samples from our 512x512 ImageNet model, without guidance.
Figure 9: More random samples from our  $512 \times 512$  ImageNet model, without guidance.
Figure 10: Random samples from our  $512 \times 512$  ImageNet model, with guidance strength 4.
Figure 11: More random samples from our  $512 \times 512$  ImageNet model, with guidance strength 4.