# A Multi-Scale Deep Back-Projection Backbone for Face Super-Resolution with Diffusion Models
## 3 Image Super-Resolution via Repeated Refinement
### Inference Process and Optimization Model  
The inference process \(p\) is the inverse of the process \(q\), which starts from a pure Gaussian noise and goes in the reverse direction of the forward diffusion process. CombiningEquations (2) and (3), and [PERSON] rule [18], we can derive the posterior distribution of \(HR_{t-1}\)\((t=1,2,\dots,T)\) given (\(HR_{0}\), \(HR_{t}\)) as Equation (4):  
\[ \begin{array}{ll}q(HR_{t-1}\mid HR_{0},HR_{t})&=\mathcal{N}\big{(}HR_{t-1}\mid \boldsymbol{\mu},\sigma^{2}\boldsymbol{I}\big{)},\\ \boldsymbol{\mu}&=\frac{\sqrt{\gamma_{t-1}(1-\alpha_{t})}}{1-\gamma_{t}}HR_{0} +\frac{\sqrt{\alpha_{t}(1-\gamma_{t-1})}}{1-\gamma_{t}}HR_{t},\\ \sigma^{2}&=\frac{(1-\gamma_{t-1})(1-\alpha_{t})}{1-\gamma_{t}}.\end{array}  \tag{4}\]  
Equation (4) shows how the \"\(\alpha_{t}\) determines the variance of the noise added at each iteration\" by giving the mean and variance of the noise-added image \(HR_{t-1}\). This formula tells us that the variance of the inference \(p\) at any time \(t\) can be decided with a pre-set sequence of hyperparameters \(\gamma_{t}\). For the process \(p\), \(HR_{t}\) is always known, but \(HR_{0}\) is unknown, which leads to the unknown mean \(\boldsymbol{\mu}\) of \(HR_{t-1}\). But fortunately, according to Equation (3) and properties of the Gaussian distribution, we can use \(HR_{t}\) to represent \(HR_{0}\):  
\[HR_{0}=\tfrac{1}{\sqrt{\gamma_{t}}}\big{(}HR_{t}-\sqrt{1-\gamma_{t}}\boldsymbol {\epsilon}_{t}\big{)}, \tag{5}\]  
where \(\boldsymbol{\epsilon}_{t}\sim\mathcal{N}(0,\boldsymbol{I})\). If we plug Equation (5) into the expression for \(\boldsymbol{\mu}\), we can obtain a new mean expression that only involves \(HR_{t}\) and a normal random variable \(\boldsymbol{\epsilon}_{t}\). Since \(\boldsymbol{\epsilon}_{t}\) is the only unknown factor, we can model it by letting a network \(f_{\theta}\) output a prediction for \(\boldsymbol{\epsilon}_{t}\). Furtherly, we can calculate the \(\boldsymbol{\mu}\) based on \(\boldsymbol{\epsilon}_{t}\), and then obtain the result of \(HR_{t-1}\). Through a series of mathematical derivations and some simplification methods [23], each iteration of iterative refinement takes the form,  
\[HR_{t-1}\leftarrow\frac{1}{\sqrt{\alpha_{t}}}\bigg{(}HR_{t}-\frac{1-\alpha_{t }}{\sqrt{1-\gamma_{t}}}f_{\theta}(LR,HR_{t},\gamma_{t})\bigg{)}+\sqrt{1- \alpha_{t}}\boldsymbol{\epsilon}_{t}\bigg{)}. \tag{6}\]  
In Equation (6), \(f_{\theta}\) represents a network structure for stepwise noise reduction in the process \(p\). The parameters of \(f_{\theta}\) are learned in the training phase throughout the conditional diffusion model, which in turn enables inference to produce better-quality images. \(f_{\theta}\) first up-samples the LR image with interpolation, and then concatenates the up-sampled \(LR\) with \(HR_{t}\) to obtain a richer feature. We will discuss the causes of generation and optimal settings for \(f_{\theta}\) in Section 4. Further, the optimized target model in the training process can be determined by using \(f_{\theta}\) as Equation (7):  
\[\text{min}\ \mathbb{E}_{(LR,HR)}\mathbb{E}_{\epsilon,\gamma_{t}}\parallel f_{ \theta}\Big{(}LR,\sqrt{\gamma_{t}}\ HR_{0}+\sqrt{1-\gamma_{t}}\,\boldsymbol{ \epsilon},\gamma_{t}\Big{)}-\epsilon\parallel_{p}^{p}, \tag{7}\]  
where \(\boldsymbol{\epsilon}\sim\mathcal{N}(0,\boldsymbol{I})\), \((LR,HR)\) are sampled from the training set, \(p\) can take the value 1 or 2, and the distribution of \(\gamma_{t}\) is consistent with the sequence given in Section 3.1. To explain the diffusion model more clearly, we give Figure 1, which shows the entire SR3 operational flow. As shown in Figure 1, all \(Net^{t}(t=1,2,\dots,T)\) constitute a complete \(f_{\theta}\). These sub-networks are abstract representations, and we did not train them separately. Instead, we controlled the network weights at different time steps through the hyperparameter \(\gamma_{t}\). As illustrated in Figure 1, during the training phase, the process of reconstructing an LR image involves several steps.