# A Multi-Scale Deep Back-Projection Backbone for Face Super-Resolution with Diffusion Models
###### Abstract  
Face verification and recognition are important tasks that have made great progress in recent years. However, recognizing low-resolution faces from small images is still a difficult problem. In this paper, we advocate using diffusion models (DMs) to enhance face resolution and improve their quality for various downstream applications. Most existing DMs for super-resolution use U-Net as their backbone network, which only exploits multi-scale features along the spatial dimension. These approaches result in a slow convergence of corresponding DMs and the inability to capture complex details and fine textures. To address this issue, we propose a novel conditional generative model based on DMs called BPSR3, which replaces the U-Net in super-resolution via repeated refinement (SR3) with a multi-scale deep back-projection network structure. BPSR3 can extract richer features not only in depth but also in breadth. This helps to effectively refine the image quality at different scales. The experimental results on facial datasets show that BPSR3 significantly improved both convergence speed and reconstruction performance. BPSR3 has about 1/4 of the parameters of SR3 but achieves a 50.1% improvement in PSNR, a 19.8% improvement in SSIM, and a 15.4% reduction in FID. Our contribution lies in achieving less time and space consumption and better reconstruction results. In addition, we propose an idea of enhancing the performance of DMs by replacing the U-Net with a better network. super-resolution; diffusion models; U-Net; back-projection +
Footnote †: journal: Medical  
images by factors of \(2\times\). They addressed the problem of hallucinating HR facial images from unaligned LR inputs at high magnification factors. Since the introduction of the generative adversarial networks (GANs) [10], SISR based on GANs has gained popularity, as seen in [11; 12; 13; 14; 15; 16; 17]. Recently, however, diffusion models [18] with rigorous mathematical inference have surpassed the previous SOTA GAN in some tasks, such as image generation [19], and image synthesis [20; 21]. Diffusion models are a type of image generation models that simulates the image degradation process by gradually adding noise and learning the distribution of the real image through the denoising process. By adding a small amount of noise at each step, these models achieve better results. Nevertheless, several challenges persist. For instance, the training process of diffusion models is computationally demanding due to the multiple repeated sampling. Moreover, convergence can be slow, and there are instances where the reconstructed output fails to align adequately with the reference image. Several researchers are currently addressing these concerns and solving the drawbacks of the diffusion models mentioned above. [PERSON] et al. [22] proposed an efficient diffusion-based SISR approach that reduces training costs by diffusing the residual between HR and LR images. [PERSON] et al. [23] introduced a generalized conditional generative SISR method that directly generates HR images by using LR image-guided diffusion. However, this method requires stitching LR images and involves a significant computational effort. [PERSON] et al. [24] utilized residual diffusion for SISR, incorporating a CNN in image preprocessing to capture improved pixel neighborhood relations. To tackle a slow convergence, [PERSON] et al. [25] and [PERSON] et al. [26] replaced the U-Net architecture underlying the diffusion model with transformer-based modules. Meanwhile, they introduced an accelerated sampling scheme and achieved promising results. In addition, researchers have proposed accelerated sampling methods [27; 28] to address non-Gaussian and multimodal distributions in the inverse diffusion process. This paper aims to enhance the application of diffusion models in facial image SR by leveraging their strengths in combination. Inspired by the findings from [PERSON] et al. [7], [PERSON] et al. [23], [PERSON] et al. [24], and [PERSON] et al.