SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Thus, the sampling step can be interpreted as follows: At each step  $t$ , we start with a noisy image  $x_{t}$  and use the diffusion model to estimate the clean image  $\hat{x_0}$  and the noise  $\hat{z}$  that was added to this clean image  $\hat{x_0}$  to get the noisy image  $x_{t}$ . In order to move to lesser noisy image  $x_{t - 1}$ , one "adds back" lesser noise to the "predicted clean image"  $\hat{x_0}$ . Now one can add back noise in 2 ways, either the noise which was added to the clean image  $\hat{x_0}$  which is  $\hat{z}$  or sample a new uncorrelated noise  $\epsilon_t$ . Infact both of these noise can be added using  $\sigma_t$  as the hyperparameter that weighs the amount of each noise added. This  $\sigma_t$  is placed in the equation such that for any choice of  $\sigma_t$ , the standard deviation of noise added back is  $\sqrt{1 - \alpha_{t - 1}}$ . For  $\sigma_t = 0$ , we only add back the estimated noise  $\hat{z}$  and no uncorrelated noise  $\epsilon_t$  which is infact the DDIM sampling. While for  $\sigma_t = \sqrt{(1 - \alpha_{t - 1}) / (1 - \alpha_t)}\sqrt{1 - \alpha_t / \alpha_{t - 1}}$  we get the sampling method described in DDPM.
Nevertheless, for any choice of  $\sigma_t$ , the sampling method involves a denoising operation which is shown as  $R(x_s, s)$  in Algorithm 1 and adding back noise shown as  $x_{s-1} = D(\hat{x}_0, s-1)$  in Algorithm 1. The only difference between different sampling methods explained in DDPM or DDIM is how one degrades the image back.
A.9 Discussion on Algorithm 2 produces lesser error than Algorithm 1
Consider the case where the degradation is a simple blur operator that removes one frequency from an image every time  $t$  increases by one. We can write  $X = x_{0} + x_{1}\dots x_{T - 1} + x_{T}$ , where each  $x_{i}$  is the Fourier mode of  $X$  representing frequency  $T - i$ . The degradation operator for this blur is  $D(X,t) = x_{t} + \ldots x_{T - 1} + x_{T} = \sum_{i = t}^{T}x_{i}$ , and when  $t = T$  the signal has been blurred into a constant vector.
Suppose we are performing the reverse process, and we begin at step  $t$  with  $X_{t} = x_{t} + \ldots x_{T}$ . Now in order to go from  $X_{t}$  to  $X_{t-1}$ , we first use the diffusion model to predict  $\hat{X} = R(X_{t}, t)$ . This  $\hat{X}$  has an analogous Fourier expansion  $\hat{X} = \sum_{n=0}^{T} \hat{x}_{n}$ , where  $\hat{x}_{0}$  is the highest frequency mode and  $\hat{x}_{T}$  is the DC mode. At step  $t$ , the error  $E_{t}$  can be defined as  $\| X_{t-1} - \hat{X}_{t-1} \|$ .
For the Algorithm 1, in which  $X_{t - 1}$  is given by  $\hat{X}_{t - 1} = D(\hat{X}, t - 1)$ , we can expand the error as follows
$$
\begin{array}{l} E _ {t} ^ {2} = \left\| X _ {t - 1} - \hat {X} _ {t - 1} \right\| ^ {2} \\ = \| \sum_ {i = t - 1} ^ {T} x _ {i} - \sum_ {i = t - 1} ^ {T} \hat {x} _ {i} \| ^ {2} \\ = \| \sum_ {i = t - 1} ^ {T} \left(x _ {i} - \hat {x} _ {i}\right) \| ^ {2} \\ = \sum_ {i = t - 1} ^ {T} \left\| x _ {i} - \hat {x} _ {i} \right\| ^ {2} \\ \end{array}
$$
For the Algorithm 2, in which  $X_{t-1}$  is given by  $\hat{X}_{t-1} = X_t - D(\hat{X}, t) + D(\hat{X}, t-1)$ , we can expand the error as follows
$$
\begin{array}{l} E _ {t} ^ {2} = \left\| X _ {t - 1} - \left(X _ {t} - D (\hat {X}, t) + D (\hat {X}, t - 1)\right) \right\| ^ {2} \\ = \left\| \left(D (\hat {X}, t) - D (\hat {X}, t - 1)\right) - \left(X _ {t} - X _ {t - 1}\right) \right\| ^ {2} \\ = \left\| \left(\sum_ {i = t} ^ {T} \hat {x} _ {i} - \sum_ {i = t - 1} ^ {T} \hat {x} _ {i}\right) - \left(\sum_ {i = t} ^ {T} x _ {i} - \sum_ {i = t - 1} ^ {T} x _ {i}\right) \right\| ^ {2} \\ = \left\| \hat {x} _ {t - 1} - x _ {t - 1} \right\| ^ {2} \\ \end{array}
$$
Hence, we can see that the error incurred at step  $t$  using algorithm 1 is  $\sum_{i = t - 1}^{T}\| x_{i} - \hat{x}_{i}\|^{2}$  which is clearly greater than or equal to error incurred by using the algorithm 2 which is  $\| \hat{x}_{t - 1} - x_{t - 1}\| ^2$ . We
now further break down the analysis based on different scenarios depending on the quality of the reconstruction operator  $R$ :
1. R is perfect.
In this scenario, both Algorithm 1 and Algorithm 2 are indistinguishable, and the error is 0 for both.
2. R is imperfect only for  $\mathbf{x}_{t-1}$  and reconstructs other signals perfectly.
In this scenario, both Algorithm 1 and Algorithm 2 incur the same error, which is  $(x_{t-1} - \hat{x}_{t-1})^2$ .
3. R is imperfect for more than one frequency
This is the realistic scenario. In this case, the error incurred by Algorithm 1 is strictly greater than Algorithm 2.
Thus we prove that for the realistic scenario i.e. when  $R$  is not a perfect reconstruction operator, the error incurred using Algorithm 1 is always greater than the Algorithm 2.
A.10 Empirical comparisons between Algorithm 2 and Algorithm 1
For all the degradations in Section 4, we compare to "Direct Sampling" which is in fact the "one step" reconstruction of the input to Algorithm 2. One of the main contributions of our work is Algorithm 2, which outperforms Algorithm 1 across all degradations. Infact the Algorithm 1 is worse than the one-step generation as well. To demonstrate this concretely we present the FID results in 6, where we can clearly see that FID scores for Algorithm 1 are worse than both the one-step and Algorithm 2. We also show qualitatively how bad the Algorithm 1 is in Figures 16 and 18. In fact, for the case of the celebA dataset, Algorithm 1 fails drastically, while for CIFAR-10 we can see high-frequency signals present in the image generation. Hence, we chose our baseline to be direct generation instead of Algorithm 1.
Table 6: FIDs for blur degradation for Algorithm 1, Algorithm 2 and Direct Reconstruction. This table demonstrates that Algorithm 1 is worse than both the one-step generation and Algorithm 2
Dataset	Direct Generation	Algorithm 1	Algorithm 2
MNIST	5.10	8.24	4.69
CIFAR-10	83.69	97.89	80.08
CelebA	36.37	299.61	26.14
Algorithm 2
Figure 16: Comparison of Algorithm 1 and Algorithm 2. We demonstrate that Algorithm 1 performs much worse than Algorithm 1 and fails completely for CelebA dataset.
Figure 17: Comparison of direct reconstruction with sampling using TACoS described in Algorithm 2 for generation with blur transformation and broken symmetry. Left-hand column is the initial cold images generated using the simple Gaussian model. Middle column has images generated in one step (i.e. direct reconstruction). Right-hand column are the images sampled with TACoS described in Algorithm 2. We present results for both CelebA (top) and AFHQ (bottom) with resolution  $128 \times 128$ .
Figure 18: Comparison of Algorithm 1 (top row) and Algorithm 2 (bottom row) for generation with Method 2 and broken symmetry on  $128 \times 128$  CelebA dataset. We demonstrate that Algorithm 1 fails completely to generate a new image.
Figure 19: Progressive deblurring of selected blurred MNIST and CIFAR-10 images.
Figure 20: Progressive deblurring of selected blurred CelebA images.
Figure 21: Deblurred Cifar10 images