HeadsUp! High-Fidelity Portrait Image Super-Resolution

## Abstract

Abstract Portrait pictures, which typically feature both human subjects and natural backgrounds, are one of the most prevalent forms of photography on social media.
Existing image super-resolution (ISR) techniques generally focus either on generic real-world images or strictly aligned facial images ( i.e. , face super-resolution).
In practice, separate models are blended to handle portrait photos: the face specialist model handles the face region, and the general model processes the rest.
However, these blending approaches inevitably introduce blending or boundary artifacts around the facial regions due to different model training recipes, while human perception is particularly sensitive to facial fidelity.
To overcome these limitations, we study the portrait image supersolution (PortraitISR) problem, and propose HeadsUp , a single-step diffusion model that is capable of seamlessly restoring and upscaling portrait images in an end-to-end manner.
Specifically, we build our model on top of a single-step diffusion model and develop a face supervision mechanism to guide the model in focusing on the facial region.
We then integrate a reference-based mechanism to help with identity restoration, reducing face ambiguity in low-quality face restoration.
Additionally, we have built a high-quality 4K portrait image ISR dataset dubbed PortraitSR-4K , to support model training and benchmarking for portrait images.
Extensive experiments show that HeadsUp achieves state-of-the-art performance on the PortraitISR task while maintaining comparable or higher performance on both general image and aligned face datasets.

## 1 Introduction

Image super-resolution (ISR) is an essential computer vision task that aims to recover high-quality, high-resolution images from degraded low-quality counterparts.
Significant advancements in ISR have been driven by collecting high-quality image datasets (Cai et al., 2019; Wei et al., 2020), realistic degradation simulation via the combination of pre-set degradations (Wang et al., 2021c; Zhang et al., 2021), learning from real-world degradation distributions (Wang et al., 2021a; Fritsche et al., 2019; Yuan et al., 2018), and leveraging priors from generative foundation models (Wu et al., 2024b; Wang et al., 2024b; Sun et al., 2024b; Wu et al., 2024a).
Despite general success across various image domains, however, existing models often exhibit notably inferior performance when applied to in-the-wild portrait pictures that contain human faces, an area where human perception is especially sensitive to errors in detail and fidelity.
Portrait images account for a large portion of online photography. Failing to handle both face and background at the same time will cause inconsistency.

To better handle facial images, a variety of methods train *specialist face ISR* models on aligned facial images by leveraging face geometric prior (Yu et al., 2018; Shen et al., 2018; Chen et al., 2018; Kim et al., 2019), reference signals (Zhang et al., 2024; Li et al., 2020a; b; 2018b; Chong et al., 2025), generative prior (Wang et al., 2021b; Yang et al., 2025; Wang et al., 2024c), and quantized codebooks (Wang et al., 2024c; Zhou et al., 2022).
These approaches have substantially improved the reconstruction quality of facial images with better detail and perceptual quality.
However, they are restricted to aligned facial data, limiting their utility for generic portrait photography.
A practical solution to implement portrait ISR is to employ a hybrid fusion strategy (Zhou et al., 2022; Wang et al., 2021c)—using a generalist ISR for backgrounds and a specialist face ISR model for facial regions in a crop-project-restore-blend paradigm.
Unfortunately, this segmented processing approach frequently results in visible boundary artifacts and inconsistencies between facial regions and the surrounding background, significantly degrading the overall perceptual quality of enhanced portraits.

Therefore, we investigate the Portrait Image Super-Resolution problem PortraitISR for short, and propose an end-to-end portrait image super-resolution approach that ❶ achieves seamless portrait ISR without any boundary effects, and ❷ maintains high-quality background and high-fidelity face as the blending-based methods.
A naïve baseline could be a generalist ISR model trained on portrait data.
However, achieving such an integrated model is non-trivial due to several identified challenges: Firstly, while face is usually the most sensitive part with insufficient details and low-fidelity, it only occupies a small portion of the whole image in most circumstances—the small 20% region gets the big 80% importance. Diverse scales, positions, and orientations of faces present in casual captures further worsen this issue.
Secondly, extremely low-quality inputs introduce substantial ambiguity, making precise facial detail reconstruction a particularly challenging ill-posed problem.
Finally, while abundant data is available for general ISR (Li et al., 2023; Agustsson & Timofte, 2017; Wang et al., 2018) as well as the aligned facial ISR (Liu et al., 2015; Karras et al., 2019) individually, a dedicated, high-quality dataset with diverse in-the-wild faces for PortraitISR tasks remains absent.

In response to the above challenges, we propose HeadsUp, an end-to-end framework for high-fidelity, face-aware PortraitISR using a single model.
Specifically, we first propose a face-aware region loss that emphasizes both face perceptual quality and face identity.
Additionally, we design an adaptive face identity module that allows information flow from a reference face image as a promptable identity guidance.
Finally, we construct a large-scale, high-resolution benchmark, PortraitSR-4K, that contains 30k high-quality 4K portrait images, curated and filtered from web-scale data.
Experimental evaluations demonstrate that our proposed approach achieves state-of-the-art results in portrait ISR, surpassing existing methods in terms of perceptual quality and fidelity, while maintaining competitive performance on general ISR benchmarks. In summary, our contributions include:

- •
We study the PortraitISR problem and introduce HeadsUp, a novel end-to-end framework specifically designed for seamless portrait image super-resolution without any need for post-processing blending, producing high-quality outputs without any boundary artifacts.
- •
We propose a face-aware region loss and a reference-guided adaptive face identity mechanism to improve facial restoration quality significantly, which better trains our proposed diffusion model.
- •
We build PortraitSR-4K, the first-of-its-kind, carefully curated, high-resolution (4K) portrait ISR dataset containing 30k images, facilitating future research in portrait ISR tasks.
- •
We have established a benchmark on our proposed PortraitSR-4K, where comprehensive experimental results demonstrate superior performance of HeadsUp over existing ISR and face-specific methods. We have also conducted ablation studies to show the design components of our model.

## 2 Methodology

Figure: Figure 2: Pipeline of HeadsUp. Starting from a pre-trained latent diffusion model, we add a LoRA adapter to the VAE encoder and denoising network. HeadsUp takes as input an LQ image and an optional reference, and denoises for only one step to produce an HQ image. In the training stage, we employ face-specific losses to improve facial restoration quality.
Refer to caption: figs/pipe-v2.png

### 2.1 Preliminary

To make the representation compact, latent diffusion models (LDMs) represent images in a low-resolution latent space. LDM consists of two procedures. The forward procedure gradually introduces Gaussian noise on the latent codes until the noise-added latent codes are subject to a Gaussian distribution. The forward procedure is denoted as $q(x_{t}|x_{t-1})=\mathcal{N}(x_{t},\sqrt{1-\beta_{t}}x_{t-1},\beta_{t}I)$, where $x_{t}$ and $x_{t-1}$ are the latent code at step $t$ and $t-1$. To recover the clean latent code $x_{0}$ from the random noise $x_{T}$, diffusion models progressively remove noise from $x_{T}$, described as $p_{\theta}(x_{t-1}|x_{t})=\mathcal{N}(\mu_{\theta}(x_{t},t),\Sigma_{\theta}(x_{t},t))$, where $\mu_{\theta}$ and $\Sigma_{\theta}$ are learned denoising functions. In practice, we usually learn a neural network $\epsilon_{\theta}$ to predict the noise in $x_{t}$. With DDIM we can jump to any diffusion step $s$ from step $t$ by using $x_{s}=\alpha_{s}x_{0}^{\prime}+\beta_{s}\epsilon_{\theta}(x_{t},t)$, $x_{0}^{\prime}$ is estimated via $x_{0}^{\prime}=\frac{x_{t}-\beta_{t}\epsilon_{\theta}(x_{t},t)}{\alpha_{t}}$. However, one-step diffusion super-resolution models (e.g., OSEDiff (Wu et al., 2024a)), in contrast to generative models, often start from a latent code encoded from LQ images rather than a randomly initialized noise, thus they can usually use fewer denoising steps by directly estimating $x_{0}$ from $x_{t}$ using DDIM.

### 2.2 Portrait Image Super-resolution (PortraitISR)

We follow OSEDiff (Wu et al., 2024a) to formulize the general image super-resolution task as $\hat{\bm{x}}_{H}=\arg\min_{\bm{x}_{H}}{\mathcal{L}}_{\rm{data}}(\Phi(\bm{x}_{H}),x_{L})+\lambda{\mathcal{L}}_{\rm{reg}(\bm{x}_{H})}$, where $\hat{\bm{x}}_{H}$ is the predicted high-quality (HQ) restored image, $\bm{x}_{L}$ the input low-quality (LQ) image, $\Phi$ the degradation function, ${\mathcal{L}}_{\rm{data}}$ the pair-wise supervise term, and ${\mathcal{L}}_{\rm{reg}}$ the regularization term. For portrait images, we separately model the entire portrait image and the face region.
We denote ${\mathcal{P}}$ the portrait image set and ${\mathcal{F}}$ the set of aligned face images. For each portrait image $\bm{x}_{p}\in{\mathcal{P}}$, we can extract its face image and align it with a standard template using an affine transformation, denoted as $\bm{x}_{f}=\Omega(\bm{x}_{p})\in{\mathcal{F}}$, where $\Omega$ is the projection function to transform portrait images to aligned face images. We also introduce a reference facial image $\bm{x}_{r}\in{\mathcal{F}}$ to better guide the model. Thus we introduce the task of portrait image super-resolution (PortraitISR):
given input LQ image $\bm{x}_{L}\in{\mathcal{P}}$ and an optional reference face image $\bm{x}_{r}\in{\mathcal{F}}$ or $\bm{x}_{r}=0$, estimate an HQ image $\hat{\bm{x}}_{H}$ that (1) the enhanced HQ image should be consistent with the input LQ image; (2) the face in the enhanced image should be consistent with both the LQ face and reference face; (3) the entire HQ image should follow the natural image prior. We modify the portrait ISR formula from the general one as follows:

$$ $\hat{\bm{x}}_{H}=\operatornamewithlimits{\rm{argmin}}_{\bm{x}_{H}\in{\mathcal{P}}}~\lambda_{\mathcal{P}}{\mathcal{L}}_{{\mathcal{P}}}(\Phi_{{\mathcal{P}}}(\bm{x}_{H}),\bm{x}_{L})+\lambda_{\mathcal{F}}{\mathcal{L}}_{\mathcal{F}}(\Phi_{{\mathcal{F}}}(\Omega(\bm{x}_{H})),\Omega(\bm{x}_{L}),\bm{x}_{r})+\lambda_{\rm{reg}}{\mathcal{L}}_{\rm{reg}}(\bm{x}_{H}),$ $$

where ${\mathcal{L}}_{{\mathcal{P}}}$ and ${\mathcal{L}}_{{\mathcal{F}}}$ respectively measures the fidelity on the entire portrait image and the face region; ${\mathcal{L}}_{\rm{reg}}$ represents the regularization term the as the general ISR task; $\lambda_{\mathcal{P}}$, $\lambda_{\mathcal{F}}$ and $\lambda_{\rm{reg}}$ is the coefficients to balance the different terms.

Similar to the general ISR task (Wu et al., 2024a), we learn a neural network model $G_{\theta}$ to recover the HQ image. The training data is formulized as triplets $\mathcal{S}=\{(\bm{x}_{L},\bm{x}_{H},\bm{x}_{r})|\bm{x}_{L},\bm{x}_{H}\in{\mathcal{P}},\bm{x}_{r}\in{\mathcal{F}}~\rm{or}~\bm{x}_{r}=0\}$. The learning problem is described as:

$$ $\displaystyle\theta^{*}$ $\displaystyle=\operatornamewithlimits{argmin}_{\theta}\mathbb{E}_{(\bm{x}_{L},\bm{x}_{H},\bm{x}_{r})\sim\mathcal{S}}\left[{\mathcal{L}}(\bm{x}_{H},\bm{x}_{L},\bm{x}_{r},G_{\theta})\right]$ (1) $\displaystyle{\mathcal{L}}$ $\displaystyle=\lambda_{\mathcal{P}}{\mathcal{L}}_{\mathcal{P}}(\bm{x}_{H},G_{\theta}(\bm{x}_{L}))+\lambda_{\mathcal{F}}{\mathcal{L}}_{\mathcal{F}}(\Omega(\bm{x}_{H}),\Omega(G_{\theta}(\bm{x}_{L})),\bm{x}_{r})+\lambda_{\rm{reg}}{\mathcal{L}}_{\rm{reg}}(G_{\theta}(\bm{x}_{L})),$ (2) $$

where ${\mathcal{L}}_{\mathcal{P}},{\mathcal{L}}_{\mathcal{F}}$ and ${\mathcal{L}}_{reg}$ are the loss or regularize terms. In the following section, we will introduce the detailed design of the loss functions and the model.

### 2.3 Face Aware Supervision

As mentioned above, simply training an ISR model on portrait data can face the problem of insufficient supervision on face images and ambiguity for face identities. To tackle these problems, we carefully design the face objectives ${\mathcal{L}}_{\mathcal{F}}$ in Eq. [2](https://arxiv.org/html/2510.09924v1#S2.E2) while the other terms are derived from (Wu et al., 2024a), which includes MSE, LPIPS, and VSD loss. Specifically, the face loss ${\mathcal{L}}_{\mathcal{F}}$ consists of three parts, the face fidelity loss ${\mathcal{L}}_{\mathcal{F}}^{\rm{fid}}$, the reference involved face identity loss ${\mathcal{L}}_{\mathcal{F}}^{\rm{id}}$, and the face adversarial loss ${\mathcal{L}}_{\mathcal{F}}^{\rm{adv}}$, denoted as:

$$ ${\mathcal{L}}_{\mathcal{F}}(\bm{x}_{H}^{\rm{f}},\hat{\bm{x}}_{H}^{\rm{f}},\bm{x}_{r})=\lambda^{\rm{fid}}{\mathcal{L}}_{\mathcal{F}}^{\rm{fid}}(\bm{x}_{H}^{\rm{f}},\hat{\bm{x}}_{H}^{\rm{f}})+\lambda^{\rm{id}}{\mathcal{L}}_{\mathcal{F}}^{\rm{id}}(\bm{x}_{H}^{\rm{f}},\hat{\bm{x}}_{H}^{\rm{f}},\bm{x}_{r})+\lambda^{\rm{adv}}{\mathcal{L}}_{\mathcal{F}}(\bm{x}_{H}^{\rm{f}},\hat{\bm{x}}_{H}^{\rm{f}},\bm{x}_{r}),$ $$

where $\bm{x}_{H}^{\rm{f}}=\Omega(\bm{x}_{H}))$ the cropped and aligned face region in the HQ image, $\hat{\bm{x}}_{H}^{\rm{f}}=\Omega(G_{\theta}(\bm{x}_{L})$ the face image of the model prediction, and $\bm{x}_{r}$ the reference face image.

#### Face Fidelity Loss.

To learn a model that can restore portrait images with a more fine-grained face, we employed a face fidelity loss that is specifically applied on the aligned face region, denoted as

$$ ${\mathcal{L}}_{\mathcal{F}}^{fid}(\bm{x}_{H}^{\rm{f}},\hat{\bm{x}}_{H}^{\rm{f}})=||\bm{x}_{H}^{\rm{f}}-\hat{\bm{x}}_{H}^{\rm{f}}||_{2}+\lambda_{\rm{LPIPS}}{\mathcal{L}}_{\rm{LPIPS}}(\bm{x}_{H}^{\rm{f}},\hat{\bm{x}}_{H}^{\rm{f}}).$ (3) $$

#### Face Identity Loss.

To guide the model to preserve the identity of the face region, we developed an identity loss. We further introduce a reference face image into the system to deal with the ambiguity problem. To model face identity, inspired by (Wang et al., 2024a), we employ an off-the-shelf face recognition model $R:{\mathcal{F}}\rightarrow\mathbb{R}^{d}$ as a feature extractor, then we build a pair-wise identity criterion based on the cosine similarity of their recognition features as follows:

$$ $\varphi(\bm{x},\bm{y})=\frac{\langle R(\bm{x}),R(\bm{y})\rangle}{||R(\bm{x})||_{2}\cdot||R(\bm{y})||_{2}}\in\left[-1,1\right]$ (4) $$

where $\bm{x}$, $\bm{y}$ are two aligned facial images, $\langle\cdot\rangle$ the inner production, and $||\cdot||_{2}$ the L2-norm of vector.
Then we construct the identity loss considering the HQ face $\bm{x}_{H}^{\rm{f}}\in{\mathcal{F}}$, predicted face $\hat{\bm{x}}_{H}^{\rm{f}}\in{\mathcal{F}}$ and the optional reference image $\bm{x}_{r}\in{\mathcal{F}}$ or $\bm{x}_{r}=0$.
The intuition is that the predicted face should always be similar to the HQ face (the GT term). If there is a reference face, the identity loss should encourage the model to take information from the reference face by explicitly taking the identity similarity of the predicted and reference faces into account (the reference term). Further, we weight the terms by the similarity between the HQ face and the reference face. The identity loss is formalized as follows:

$$ ${\mathcal{L}}_{\mathcal{F}}^{\rm{id}}(\bm{x}_{H}^{\rm{f}},\hat{\bm{x}}_{H}^{\rm{f}},\bm{x}_{r})=-\log(\frac{\varphi(\hat{\bm{x}}_{H}^{\rm{f}},\bm{x}_{H}^{\rm{f}})+1}{2})-\log(\frac{\varphi(\hat{\bm{x}}_{H}^{\rm{f}},\bm{x}_{r})+1}{2})\varphi(\bm{x}_{H}^{\rm{f}},\bm{x}_{r}).$ $$

Note that if there is no reference image, we define its similarity with any face to zero, i.e., $\varphi(\bm{x},0)=0,\forall\bm{x}\in{\mathcal{F}}$. In this case, the identity loss will have the GT term only.

### 2.4 One-step PortraitISR Framework

#### Overview.

As mentioned in [2.2](https://arxiv.org/html/2510.09924v1#S2.SS2),
the portrait ISR model $G_{\theta}$ takes as input an LQ image $\bm{x}_{L}\in{\mathcal{P}}$ and an optional reference image $\bm{x}_{r}\in{\mathcal{F}}$ or $\bm{x}_{r}=0$. If the reference image is provided, we additionally introduce a binary mask $M\in\{0,1\}^{h\times w}$ to specify the face location in the LQ image, where $h$ and $w$ are the height and width of the LQ image. The model then predicts the HQ portrait image $\hat{\bm{x}}_{H}$ from the given conditions.
Following (Wu et al., 2024a), we build $G_{\theta}$ as a one-step diffusion model. As it also requires a text prompt as input, we employ a text extractor $T:P\rightarrow\mathcal{T}$ to estimate the corresponding text prompt from the LQ image, where $\mathcal{T}$ is the text set. The portrait ISR procedure is formalized as $\hat{\bm{x}}_{H}=G_{\theta}(\bm{x}_{L},\bm{x}_{r},M,T(\bm{x}_{L}))$.

#### Architecture.

As shown in Fig. [2](https://arxiv.org/html/2510.09924v1#S2.F2), we start from a pre-trained latent diffusion model $\Psi=(E,\epsilon,D)$, where $E$, $\epsilon$, $D$ represent the VAE encoder, the denoiser, and the VAE decoder, respectively. We fixed the decoder and apply a LoRA (Hu et al., 2022) adaption to he encoder and denoiser, denoted as $G_{\theta}=(\tilde{E}_{\theta 1},\tilde{\epsilon}_{\theta 2},D)$, where $\tilde{M}_{\theta}$ means adding a LoRA adapter parameterized by $\theta$ to the module $M$. We omit the subscript $\theta$ in the following. Further, to enable the denoiser with the reference latent and mask as input, we extend its first convolution layer with additional zero-initialized filter channels.
The inference procedure is described as follows: (1) the LQ image and reference image are encoded into latent space $\bm{z}_{L}=\tilde{E}(\bm{x}_{L})$, $\bm{z}_{r}=\tilde{E}(\bm{x}_{r})$. (2) the latents and the resized face mask are concatenated and the denoised for one step $\hat{\bm{z}}_{H}=\frac{\bm{z}_{L}-\beta\tilde{\epsilon}(\bm{z}_{L},\bm{z}_{r},M^{\rm{resize}})}{\alpha}$, where $\alpha$ and $\beta$ are the diffusion scalar and $M^{\rm{resize}}$ is the face mask resized to latent resolution. (3) The denoised latent are decoded into HQ portrait image $\hat{\bm{x}}_{H}=D(\hat{\bm{z}}_{H})$.

## 3 PortraitSR-4K Dataset

While large-scale datasets (Li et al., 2023; Agustsson & Timofte, 2017; Wang et al., 2018; Liu et al., 2015; Karras et al., 2019) have significantly facilitated the ISR and FSR field, the lack of a high-quality portrait image dataset limits the development of PortraitISR approaches. In this work, we propose PortraitSR-4K, a large-scale portrait dataset which consists of 30k high-quality 4K portrait images from the internet.

#### Image Collection

We collected the raw videos from existing datasets, including Laion2B (Schuhmann et al., 2022), Photo Concept Bucket(^1^11https://huggingface.co/datasets/bghira/photo-concept-bucket), and PD12M (Meyer et al., 2024). We selected the images with at least 4K resolution as our raw image candidates, denoted as ${\mathcal{P}}$.

#### Portrait Data Construction.

After collecting the raw image set, we employ a face detector $\phi$ to detect the faces in each of the images. We construct the portrait set $\bm{P}\subset{\mathcal{P}}$ by removing the images that include no face or the face is too small.
We then cropped and aligned the detected faces via affine transformation to form the face image set $\bm{F}$. We construct the reference pairs $\bm{R}$ by employing the identity criteria in Eq. [4](https://arxiv.org/html/2510.09924v1#S2.E4) on each face pairs and collecting the pairs whose similarity is above certain threshold $\gamma$, i.e. $\bm{R}=\{(\bm{x},\bm{y})|\bm{x}\in\bm{F},\bm{y}\in\bm{F},\varphi(\bm{x},\bm{y})>\gamma\}$. The training pairs $\mathcal{S}$ is constructed as $\mathcal{S}=\{(\Phi(\bm{x}),\bm{x},\bm{y})|\bm{x}\in\bm{F},(\Omega(\bm{x}),\bm{y})\in\bm{R}\}$, where the terms in the triplet represents the LQ portrait, the HQ portrait, and the reference face image.

#### Dataset Splitting

We collected 30k portrait data. The training set (PortraitSR-4K-Tr) consists of 27k images, and the testing set (PortraitSR-4K-Te) consists of 3k images. We construct pairs within each subset. For the training set, all valid pairs are recorded, totaling 163k training pairs. For the testing set, each face will have at most one reference face; if multiple similar faces are detected, we simply select the most similar one. Finally, we reserved 190 portrait-reference testing pairs.

Figure: Figure 3: Qualitative Results. While general ISR approaches can achieve good overall quality, they can not produce high-fidelity faces. The blending approaches produce better face fidelity, but suffer from the border effect that causes inconsistency between the face and other regions.
Refer to caption: figs/vis-v3.jpg

## 4 Experiments

### 4.1 Experimental Settings

#### Datasets.

Training: To maintain a fair performance on general ISR tasks, we train our model on a mixture of PortraitSR-4K-Tr, LSDIR (Li et al., 2023), and DIV2K (Agustsson & Timofte, 2017). The training resolution is at $1024\times 1024$, and the degradation pipeline is derived from Real-ESRGAN (Wang et al., 2021c). Testing: For portrait ISR, we test our model on our PortraitSR-4K-Te, with the input resolution at $256\times 256$. For general ISR, we follow (Wu et al., 2024a) to evluate the models, consisting of images from DIV2K-VAL (Agustsson & Timofte, 2017), RealSR (Cai et al., 2019), and DRealSR (Wei et al., 2020), whose LQ and HQ image is at $128\times 128$ and $512\times 512$ resolution, respectively. For FSR, following  (Tsai et al., 2024), we choose celeba-Test (Liu et al., 2015) as the testing set.

#### Baselines.

**Table 1: Comparison on Face ISR. We compare HeadsUp with specialist face restoration models. We achieve the best performance on many metrics, with the rest of the metrics comparable to other methods. The top two performances are labeled in red and blue.**
| Method | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | FID$\downarrow$ | NIQE$\downarrow$ | IDA$\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| PSFRGAN | 20.303 | 0.536 | 0.450 | 66.367 | 3.811 | 1.260 |
| GFP-GAN | 19.574 | 0.522 | 0.453 | 46.130 | 4.061 | 1.268 |
| GPEN | 20.545 | 0.552 | 0.425 | 55.308 | 3.913 | 1.141 |
| RestoreFormer | 20.146 | 0.494 | 0.467 | 54.395 | 4.013 | 1.231 |
| CodeFormer | 21.449 | 0.575 | 0.365 | 62.021 | 4.570 | 1.049 |
| VQFR | 19.484 | 0.472 | 0.456 | 54.010 | 3.328 | 1.237 |
| DR2 | 20.327 | 0.595 | 0.409 | 63.675 | 5.104 | 1.215 |
| DAEFER | 19.919 | 0.553 | 0.388 | 52.056 | 4.477 | 1.071 |
| Ours | 22.924 | 0.684 | 0.345 | 29.226 | 4.264 | 1.103 |

PortraitISR: To the best of our knowledge, there is no PortraitISR specialist model, thus we employed two kinds of baselines: (1) The General ISR (GISR) approaches, including OSEDiff (Wu et al., 2024a),
and PiSA-SR (Sun et al., 2024a). (2) The practical blending approaches, which are the combination of an ISR and an FSR model. Specifically, we choose Real-ESRGAN (Wang et al., 2021c) + CodeFormer (Wu et al., 2024b) and OSEDiff (Wu et al., 2024a) + OSEDiff_Face (Wu et al., 2024a) as the baselines. GISR: Following PiSA-SR (Sun et al., 2024a), we choose state-of-the-art diffusion-based methods (Yue et al., 2023; Wang et al., 2024b; Lin et al., 2024; Yang et al., 2024; Wang et al., 2024d; Wu et al., 2024b; a; Sun et al., 2024a) as baselines. FSR: We align our baseline models with recent DEAFR (Tsai et al., 2024). The baselines are set to PSFRGAN (Chen et al., 2021a), GFP-GAN (Wang et al., 2021b), GPEN (Yang et al., 2021), RestoreFormer (Wang et al., 2022), CodeFormer (Zhou et al., 2022), VQFR (Gu et al., 2022), DR2 (Wang et al., 2023b), DAEFR (Tsai et al., 2024).

#### Metrics.

Following (Wu et al., 2024a), we employ PSNR, SSIM (Wang et al., 2004), LPIPS (Zhang et al., 2018a), FID (Heusel et al., 2017), and NIQE (Zhang et al., 2015) as evaluation metrics for all tasks. For the FISR task, we add the IDA (Deng et al., 2019) metric to align with in (Tsai et al., 2024). For the GISR task, we further use MUSIQ (Ke et al., 2021), CLIPIQA (Wang et al., 2023a), DISTS (Ding et al., 2020) and MANIQA (Yang et al., 2022) as the evaluating metrics, which is consistent with (Wu et al., 2024a). For PortraitISR, we employed MUSIQ (Ke et al., 2021), DISTS (Ding et al., 2020) and MANIQA (Yang et al., 2022).
We employ the similarity score as defined in Eq. [4](https://arxiv.org/html/2510.09924v1#S2.E4) to evaluate the identity similarity. However, the cosine distance-based metric may be noisy; we further conduct user studies to measure the identity similarity.
We conducted extra user studies to evaluate the identity similarity and the naturalness of the faces produced by different models. In the user study, the participants are given several candidate face images, and they are asked to select one image that best meets a certain criterion. We report the “win rate (WR)” as the user study metric. The win rate of one method is the frequency it is selected as the best image, out of all the selections. We denote the face naturalness metric as $\rm{WR}_{N}$, where users are asked to select the most natural face. The identity similarity is measured by $\rm{WR}_{\rm{id}}$, where users are given the ground-truth face and asked to select the most similar candidate face image.

**Table 2: Comparison on PortraitISR. We compare HeadsUp with state-of-the-art general ISR models and blending-based models. OSEDiff_Face indicates OSEDiff trained on face images. We achieve the best performance on most of the metrics.**
| Type | Method | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | DISTS$\downarrow$ | FID$\downarrow$ | NIQE$\downarrow$ | MUSIQ$\uparrow$ | ID-Score$\uparrow$ | $\rm{WR}_{\rm{id}}\uparrow$ | $\rm{WR}_{\rm{N}}\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| General ISR | OSEDiff | 25.19 | 0.7802 | 0.3287 | 0.1755 | 116.80 | 4.8639 | 66.0657 | 0.2821 | 0.21 | 0.24 |
| PiSA-SR | 24.93 | 0.7519 | 0.3506 | 0.1725 | 119.79 | 4.3204 | 66.4237 | 0.3118 | 0.17 | 0.13 |  |
| Blending | RealESRGAN+Codeformer | 24.86 | 0.7400 | 0.4700 | 0.2600 | 187.61 | 5.2831 | 48.8817 | 0.1664 | 0.06 | 0.02 |
| OSEDiff+OSEDiff_Face | 25.26 | 0.7810 | 0.3311 | 0.1728 | 116.86 | 4.8791 | 64.9308 | 0.3128 | 0.16 | 0.21 |  |
| Portrait ISR | Ours | 25.64 | 0.8060 | 0.2573 | 0.1398 | 101.13 | 4.8813 | 67.7528 | 0.3715 | 0.40 | 0.40 |

#### Implementation Details.

We initialize the base diffusion model $\Psi$ using SD 2.1 model (Rombach et al., 2021). The degradation function $\Phi$ is derived from Real-ESRGAN (Wang et al., 2021c). The face extractor is set as CVLFace (Kim et al., 2024). For the text prompt extractor, we adopt the DAPE module in (Wu et al., 2024b). The LoRA rank for all modules is set to 4. We train our model with AdamW (Loshchilov & Hutter, 2017) optimizer with the learning rate of $5\times 10^{-5}$. The loss weighting is set as $\lambda^{\rm{fid}}=1,\lambda_{\rm{LPIPS}}^{\rm{id}}=0.8,\lambda^{\rm{id}}=4$, while the rest weights are derived from OSEDiff (Wu et al., 2024a). We train our model on eight A100 GPUs.

**Table 3: Comparison on General ISR. We compare HeadsUp with state-of-the-art GISR models on the DIV2K, RealSR, and DRealSR dataset. We achieve state-of-the-art performance on some metrics while maintaining comparable on the rest metrics. We highlight the top two performances on each metric using red and blue. ‘S’ indicates the number of diffusion steps.**
| DataSet | Method | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | DISTS$\downarrow$ | FID$\downarrow$ | NIQE$\downarrow$ | MUSIQ$\uparrow$ | CLIPIQA$\uparrow$ | MANIQA$\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| DIV2K | ResShift-S15 | 24.69 | 0.6175 | 0.3374 | 0.2215 | 36.01 | 6.82 | 60.92 | 0.6089 | 0.5450 |
| StableSR-S200 | 23.31 | 0.5728 | 0.3129 | 0.2138 | 24.67 | 4.76 | 65.63 | 0.6682 | 0.6188 |  |
| DiffBIR-S50 | 23.67 | 0.5653 | 0.3541 | 0.2129 | 30.93 | 4.71 | 65.66 | 0.6652 | 0.6204 |  |
| PASD-S20 | 23.14 | 0.5489 | 0.3607 | 0.2219 | 29.32 | 4.40 | 68.83 | 0.6711 | 0.6484 |  |
| SinSR-S1 | 24.43 | 0.6012 | 0.3262 | 0.2066 | 35.45 | 6.02 | 62.80 | 0.6499 | 0.5395 |  |
| SeeSR-S50 | 23.71 | 0.6045 | 0.3207 | 0.1967 | 25.83 | 4.82 | 68.49 | 0.6857 | 0.6239 |  |
| OSEDiff-S1 | 23.72 | 0.6108 | 0.2941 | 0.1976 | 26.32 | 4.71 | 67.97 | 0.6683 | 0.6148 |  |
| PiSA-SR-S1 | 23.87 | 0.6058 | 0.2823 | 0.1934 | 25.07 | 4.55 | 69.68 | 0.6927 | 0.6400 |  |
| Ours-S1 | 23.83 | 0.6170 | 0.3265 | 0.2102 | 28.91 | 4.45 | 64.15 | 0.6445 | 0.6162 |  |
| RealSR | ResShift-S15 | 26.31 | 0.741 | 0.3489 | 0.2498 | 142.81 | 7.27 | 58.10 | 0.5450 | 0.5305 |
| StableSR-S200 | 24.69 | 0.7052 | 0.3091 | 0.2167 | 127.20 | 5.76 | 65.42 | 0.6195 | 0.6211 |  |
| DiffBIR-S50 | 24.88 | 0.6673 | 0.3567 | 0.2290 | 124.56 | 5.63 | 64.66 | 0.6412 | 0.6231 |  |
| PASD-S20 | 25.22 | 0.6809 | 0.3392 | 0.2259 | 123.08 | 5.18 | 68.74 | 0.6502 | 0.6461 |  |
| SinSR-S1 | 26.30 | 0.7354 | 0.3212 | 0.2346 | 137.05 | 6.31 | 60.41 | 0.6204 | 0.5389 |  |
| SeeSR-S50 | 25.33 | 0.7273 | 0.2985 | 0.2213 | 125.66 | 5.38 | 69.37 | 0.6594 | 0.6439 |  |
| OSEDiff-S1 | 25.15 | 0.7341 | 0.2921 | 0.2128 | 123.50 | 5.65 | 69.09 | 0.6693 | 0.6339 |  |
| PiSA-SR-S1 | 25.50 | 0.7417 | 0.2672 | 0.2044 | 124.09 | 5.50 | 70.15 | 0.6702 | 0.6560 |  |
| Ours-S1 | 25.22 | 0.7238 | 0.2671 | 0.1943 | 131.07 | 4.86 | 65.05 | 0.6296 | 0.6332 |  |
| DRealSR | ResShift-S15 | 28.45 | 0.7632 | 0.4073 | 0.2700 | 175.92 | 8.28 | 49.86 | 0.5259 | 0.4573 |
| StableSR-S200 | 28.04 | 0.7460 | 0.3354 | 0.2287 | 147.03 | 6.51 | 58.50 | 0.6171 | 0.5602 |  |
| DiffBIR-S50 | 26.84 | 0.6660 | 0.4446 | 0.2706 | 167.38 | 6.02 | 60.68 | 0.6292 | 0.5902 |  |
| PASD-S20 | 27.48 | 0.7051 | 0.3854 | 0.2535 | 157.36 | 5.57 | 64.55 | 0.6714 | 0.6130 |  |
| SinSR-S1 | 28.41 | 0.7495 | 0.3741 | 0.2488 | 177.05 | 7.02 | 55.34 | 0.6367 | 0.4898 |  |
| SeeSR-S50 | 28.26 | 0.7698 | 0.3197 | 0.2306 | 149.86 | 6.52 | 64.84 | 0.6672 | 0.6026 |  |
| OSEDiff-S1 | 27.92 | 0.7835 | 0.2968 | 0.2165 | 135.29 | 6.49 | 64.65 | 0.6963 | 0.5899 |  |
| PiSA-SR-S1 | 28.31 | 0.7804 | 0.2960 | 0.2169 | 130.61 | 6.20 | 66.11 | 0.6970 | 0.6156 |  |
| Ours-S1 | 27.70 | 0.7826 | 0.2885 | 0.2038 | 137.48 | 5.81 | 61.59 | 0.6609 | 0.5970 |  |

### 4.2 Comparison on PortraitISR

We compare our model with the baselines on PortraitSR-4K-Te dataset. The quantitative results are shown in Tab. [2](https://arxiv.org/html/2510.09924v1#S4.T2). We observe that HeadsUp continuously achieves state-of-the-art performance on most of the metrics. The qualitative results are illustrated in Fig [3](https://arxiv.org/html/2510.09924v1#S3.F3), from which we observe that (1) the general ISR approaches can achieve good enhancement in non-facial regions, but they usually suffer from hallucinated faces. (2) The hybrid blending approaches, in contrast, can preserve more detail and identity of the face, but can easily cause inconsistency around the boundary regions.

### 4.3 Comparison on GISR and FISR Tasks

In this section, we evaluate HeadsUp on the general image super-resolution (GISR) and face image super-resolution (FISR) datasets. The experimental results show that our method can generalize to other datasets and achieves competitive performance on all the tasks.

**Table 4: Ablation Studies. We ablate the key component of HeadsUp. Compare the first and fourth row, adding the face-aware loss significantly improves both the photometric criteria of the entire image and the identity similarity of the face region. Adding $L_{F}^{\rm fid}$ alone (second row) slightly improves the identity similarity. Using $L_{F}^{\rm id}$ alone (third row) can significantly improve the identity similarity, at the cost of introducing significant blur effects, shown by the low non-reference metrics and [Fig. 4](https://arxiv.org/html/2510.09924v1#S4.F4). Comparing the fourth and fifth row, adding a reference image can slightly improve identity similarity.**
| ${\bm{L}}_{\bm{F}}^{\rm{fid}}$ | ${\bm{L}}_{\bm{F}}^{\rm{id}}$ | w/ $\bm{x}_{r}$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | DISTS$\downarrow$ | FID$\downarrow$ | NIQE$\downarrow$ | MUSIQ $\uparrow$ | ID-Score$\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  |  | 25.07 | 0.8059 | 0.2755 | 0.1533 | 111.16 | 4.9286 | 69.2658 | 0.2600 |
| ✓ |  |  | 25.65 | 0.8113 | 0.2770 | 0.1631 | 110.44 | 5.1377 | 67.4218 | 0.3010 |
|  | ✓ |  | 25.85 | 0.8122 | 0.2680 | 0.1527 | 104.82 | 5.7209 | 64.5272 | 0.4348 |
| ✓ | ✓ |  | 25.74 | 0.8040 | 0.2517 | 0.1360 | 99.55 | 4.5785 | 69.0616 | 0.3634 |
| ✓ | ✓ | ✓ | 25.64 | 0.8060 | 0.2573 | 0.1398 | 101.13 | 4.8813 | 67.7528 | 0.3715 |

#### Comparison on GISR task.

We show that our model can generalize to the GISR task. We compare our model with state-of-the-art GISR methods on real-world image super-resolution datasets. The results are shown in [Tab. 3](https://arxiv.org/html/2510.09924v1#S4.T3). We achieve state-of-the-art in multiple metrics (NIQE on RealSR, LPIPS on DRealSR, etc.). Regarding the rest of the metric, we reach comparable performance to the state-of-the-art general image super-resolution approaches.

#### Comparison on FISR task.

Figure: Figure 4: Ablation Studies. Using the identity loss only, though, improves the identity-preservation ability, but will lead to over-smooth and blurred faces.
Refer to caption: figs/abl.png

We also show the HeadsUp’s ability to generalize to the face image super-resolution task.
We employed the widely used synthetic face dataset CelebA-Test (Liu et al., 2015) as the testing set. CelebA-Test consists of 3000 cropped and aligned faces.
Following the settings in (Tsai et al., 2024), we evaluated HeadsUpon the aligned face dataset, and compared it with the state-of-the-art specialist FISR models. The results can be found in  [Tab. 1](https://arxiv.org/html/2510.09924v1#S4.T1). We achieve the best performance on most metrics and comparable performance on the rest of the metrics.

### 4.4 Ablation Studies

#### Ablation on Face Aware Losses.

We ablate each component of the face-aware losses. From Tab. [4](https://arxiv.org/html/2510.09924v1#S4.T4) we observe that without face-aware supervision (first row ) will lead to better no-reference metrics, while the identity of the face is not preserved. Adding ${\mathcal{L}}_{\mathcal{P}}^{\rm{fid}}$ (second row )can improve the ID-Score. Using the identity loss alone (third row) can encourage the model to keep the identity, but the image quality will significantly drop. The reason is that the identity loss employed a feature similarity score, which may encourage the model to produce blurred and smoothed faces, as shown in Fig. [4](https://arxiv.org/html/2510.09924v1#S4.F4).

#### Ablation on Reference Images.

Our model can restore portrait images with or without a reference image. We compare the performance of the models with reference (last row) and without reference (fourth row). We observe that the reference image slightly improves the identity score but reduces image quality by a small margin.

## 5 Related Works

### 5.1 General Image Super-Resolution

General image super-resolution (GISR) aims to restore HQ images from LQ inputs. SRCNN (Dong et al., 2015) first employed convolutional neural networks to solve the GISR problem. Following approaches further developed more powerful deep learning approaches for GISR by improving CNN structure (Lim et al., 2017; Zhang et al., 2018c; b), introducing attentions (Dai et al., 2019; Zhang et al., 2022) or transformers (Chen et al., 2023a; b; Liang et al., 2021; Chen et al., 2021b), or leveraging multi-scale features (Li et al., 2018a; Gao & Zhuang, 2019). With the development of generative models, SR-by-generating became an important branch of GISR. SRGan (Ledig et al., 2017) first introduced the generative prior of GAN (Goodfellow et al., 2020) into image super-resolution. The generative prior enables SR models to produce better texture details. The following works (Zhang et al., 2021; Wang et al., 2021c; Xie et al., 2023; Liang et al., 2022b; a; Zhang et al., 2021) push super-resolution to real-world images, whose degradation is unknown to the algorithm, by developing complex degradation pipelines. Recently, many works have explored leveraging generative priors in diffusion models for GISR (Wu et al., 2024b; Yue et al., 2023; Yang et al., 2024; Wu et al., 2024a; Wang et al., 2024d; Sun et al., 2024a; Lin et al., 2024). Although achieving great success regarding the general image quality, just like other GISR methods, they often produce unnatural and low-fidelity faces for portrait images.

### 5.2 Face Image Super-Resolution

Face image super-resolution (FISR) is a task that specifically focuses on aligned facial images. Compared to general natural images, humans are more sensitive to the details of faces. Thus, FISR requires more fine-grained restoration of details. Many designs have been made to achieve high-fidelity face image super-resolution. For example, some approaches (Chen et al., 2018; Shen et al., 2018; Kim et al., 2019; Yu et al., 2018; Zhang & Wu, 2022) employed face structure or landmarks to provide a geometric prior in FISR. Reference-based methods (Zhang et al., 2024; Li et al., 2020a; b; 2018b; Chong et al., 2025) leverage one or more reference images to alleviate ambiguity and preserve more identity while conducting super-resolution. Quantized codebook (Wang et al., 2024c; Zhou et al., 2022; Tsai et al., 2024) is an effective way to model general face features. By querying the codebook, these kinds of methods can generate high-quality face images from the low-quality ones. Similar to GISR, there are also a large number of FISR models that leverage the generative prior from GAN (Yang et al., 2020; Wan et al., 2020; Menon et al., 2020; Wang et al., 2021b) or diffusion models (Wang et al., 2024c; Yang et al., 2023). Most FISR methods rely on the prior where the face images are roughly aligned with a template face. When applying to a portrait image, another GISR model is required to enhance the non-facial regions.

### 5.3 Diffusion Models in Image Super-Resolution

Recent success on pre-trained large diffusion model (Rombach et al., 2021; Zhang et al., 2023; Esser et al., 2024; Peebles & Xie, 2023) has significantly facilitated a large number of visual tasks (Li et al., 2024b; Lugmayr et al., 2022; Saxena et al., 2023; Baranchuk et al., 2021). StableSR (Wang et al., 2024b) employs a trainable adapter to leverage the generative prior of pre-trained diffusion models. SeD (Li et al., 2024a) combined GANs and diffusion models to produce more photo-realistic images. PASD (Yang et al., 2024) leverages both high- and low-level features to enable diffusion models to perceive image
local structures at a pixel-wise level. Further, several efforts have been put into reducing the diffusion steps (Wu et al., 2024a; Wang et al., 2024d; Sun et al., 2024a). Further, PiSA-SR (Sun et al., 2024a) provides a flexible trade-off between the pixel-wise fidelity and semantic-level details by
introducing two adjustable guidance scales on two
LoRA modules.

## 6 Conclusion

We introduce the task of PortraitISR, which aims to enhance portrait images that consist of both a human face and other components, such as the human body and natural background. Existing ISR approaches either general images or aligned face images, which usually suffer from low-fidelity face restoration or inconsistency around boundaries when applied to portrait images. We propose HeadsUp, the first end-to-end PortraitISR framework. We designed a face-aware region loss and a reference-guidance structure to improve facial restoration quality. We further build PortraitSR-4K, a high-resolution portrait data, facilitating future research and benchmarking in PortraitISR tasks. Experimental results on multiple datasets show that we achieve state-of-the-art performance on the PortraitISR task and competitive performance on the general ISR and face ISR tasks. Our proposed PortraitSR-4K provides high-quality portrait data, which can potentially facilitate various future research and benchmarking in fields like portrait super-resolution, generation, matting, among others.

## References

- Agustsson & Timofte (2017)
Eirikur Agustsson and Radu Timofte.
Ntire 2017 challenge on single image super-resolution: Dataset and study.
In *Proceedings of the IEEE conference on computer vision and pattern recognition workshops*, pp. 126–135, 2017.
- Baranchuk et al. (2021)
Dmitry Baranchuk, Ivan Rubachev, Andrey Voynov, Valentin Khrulkov, and Artem Babenko.
Label-efficient semantic segmentation with diffusion models.
*arXiv preprint arXiv:2112.03126*, 2021.
- Cai et al. (2019)
Jianrui Cai, Hui Zeng, Hongwei Yong, Zisheng Cao, and Lei Zhang.
Toward real-world single image super-resolution: A new benchmark and a new model.
In *Proceedings of the IEEE/CVF international conference on computer vision*, pp. 3086–3095, 2019.
- Chen & Mo (2022)
Chaofeng Chen and Jiadi Mo.
IQA-PyTorch: Pytorch toolbox for image quality assessment.
[Online]. Available: [https://github.com/chaofengc/IQA-PyTorch](https://github.com/chaofengc/IQA-PyTorch), 2022.
- Chen et al. (2021a)
Chaofeng Chen, Xiaoming Li, Lingbo Yang, Xianhui Lin, Lei Zhang, and Kwan-Yee K Wong.
Progressive semantic-aware style transformation for blind face restoration.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 11896–11905, 2021a.
- Chen et al. (2021b)
Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao.
Pre-trained image processing transformer.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 12299–12310, 2021b.
- Chen et al. (2023a)
Xiangyu Chen, Xintao Wang, Jiantao Zhou, Yu Qiao, and Chao Dong.
Activating more pixels in image super-resolution transformer.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 22367–22377, 2023a.
- Chen et al. (2018)
Yu Chen, Ying Tai, Xiaoming Liu, Chunhua Shen, and Jian Yang.
Fsrnet: End-to-end learning face super-resolution with facial priors.
In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pp. 2492–2501, 2018.
- Chen et al. (2023b)
Zheng Chen, Yulun Zhang, Jinjin Gu, Linghe Kong, Xiaokang Yang, and Fisher Yu.
Dual aggregation transformer for image super-resolution.
In *Proceedings of the IEEE/CVF international conference on computer vision*, pp. 12312–12321, 2023b.
- Chong et al. (2025)
Min Jin Chong, Dejia Xu, Yi Zhang, Zhangyang Wang, David Forsyth, Gurunandan Krishnan, Yicheng Wu, and Jian Wang.
Copy or not? reference-based face image restoration with fine details.
In *2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)*, pp. 9660–9669. IEEE, 2025.
- Dai et al. (2019)
Tao Dai, Jianrui Cai, Yongbing Zhang, Shu-Tao Xia, and Lei Zhang.
Second-order attention network for single image super-resolution.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 11065–11074, 2019.
- Deng et al. (2019)
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou.
Arcface: Additive angular margin loss for deep face recognition.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 4690–4699, 2019.
- Ding et al. (2020)
Keyan Ding, Kede Ma, Shiqi Wang, and Eero P Simoncelli.
Image quality assessment: Unifying structure and texture similarity.
*IEEE transactions on pattern analysis and machine intelligence*, 44(5):2567–2581, 2020.
- Dong et al. (2015)
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang.
Image super-resolution using deep convolutional networks.
*IEEE transactions on pattern analysis and machine intelligence*, 38(2):295–307, 2015.
- Esser et al. (2024)
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al.
Scaling rectified flow transformers for high-resolution image synthesis.
In *Forty-first international conference on machine learning*, 2024.
- Fritsche et al. (2019)
Manuel Fritsche, Shuhang Gu, and Radu Timofte.
Frequency separation for real-world super-resolution.
In *2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)*, pp. 3599–3608. IEEE, 2019.
- Gao & Zhuang (2019)
Shangqi Gao and Xiahai Zhuang.
Multi-scale deep neural networks for real image super-resolution.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops*, pp. 0–0, 2019.
- Goodfellow et al. (2020)
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.
Generative adversarial networks.
*Communications of the ACM*, 63(11):139–144, 2020.
- Gu et al. (2022)
Yuchao Gu, Xintao Wang, Liangbin Xie, Chao Dong, Gen Li, Ying Shan, and Ming-Ming Cheng.
Vqfr: Blind face restoration with vector-quantized dictionary and parallel decoder.
In *European Conference on Computer Vision*, pp. 126–143. Springer, 2022.
- Heusel et al. (2017)
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.
Gans trained by a two time-scale update rule converge to a local nash equilibrium.
*Advances in neural information processing systems*, 30, 2017.
- Hu et al. (2022)
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al.
Lora: Low-rank adaptation of large language models.
*ICLR*, 1(2):3, 2022.
- Karras et al. (2019)
Tero Karras, Samuli Laine, and Timo Aila.
A style-based generator architecture for generative adversarial networks.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 4401–4410, 2019.
- Ke et al. (2021)
Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar, and Feng Yang.
Musiq: Multi-scale image quality transformer.
In *Proceedings of the IEEE/CVF international conference on computer vision*, pp. 5148–5157, 2021.
- Kim et al. (2019)
Deokyun Kim, Minseon Kim, Gihyun Kwon, and Dae-Shik Kim.
Progressive face super-resolution via attention to facial landmark.
*arXiv preprint arXiv:1908.08239*, 2019.
- Kim et al. (2024)
Minchul Kim, Yiyang Su, Feng Liu, Anil Jain, and Xiaoming Liu.
Keypoint relative position encoding for face recognition.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 244–255, 2024.
- Ledig et al. (2017)
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al.
Photo-realistic single image super-resolution using a generative adversarial network.
In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pp. 4681–4690, 2017.
- Li et al. (2024a)
Bingchen Li, Xin Li, Hanxin Zhu, Yeying Jin, Ruoyu Feng, Zhizheng Zhang, and Zhibo Chen.
Sed: Semantic-aware discriminator for image super-resolution.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 25784–25795, 2024a.
- Li et al. (2018a)
Juncheng Li, Faming Fang, Kangfu Mei, and Guixu Zhang.
Multi-scale residual network for image super-resolution.
In *Proceedings of the European conference on computer vision (ECCV)*, pp. 517–532, 2018a.
- Li et al. (2024b)
Renjie Li, Panwang Pan, Bangbang Yang, Dejia Xu, Shijie Zhou, Xuanyang Zhang, Zeming Li, Achuta Kadambi, Zhangyang Wang, Zhengzhong Tu, et al.
4k4dgen: Panoramic 4d generation at 4k resolution.
*arXiv preprint arXiv:2406.13527*, 2024b.
- Li et al. (2018b)
Xiaoming Li, Ming Liu, Yuting Ye, Wangmeng Zuo, Liang Lin, and Ruigang Yang.
Learning warped guidance for blind face restoration.
In *Proceedings of the European conference on computer vision (ECCV)*, pp. 272–289, 2018b.
- Li et al. (2020a)
Xiaoming Li, Chaofeng Chen, Shangchen Zhou, Xianhui Lin, Wangmeng Zuo, and Lei Zhang.
Blind face restoration via deep multi-scale component dictionaries.
In *European conference on computer vision*, pp. 399–415. Springer, 2020a.
- Li et al. (2020b)
Xiaoming Li, Wenyu Li, Dongwei Ren, Hongzhi Zhang, Meng Wang, and Wangmeng Zuo.
Enhanced blind face restoration with multi-exemplar images and adaptive spatial feature fusion.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 2706–2715, 2020b.
- Li et al. (2023)
Yawei Li, Kai Zhang, Jingyun Liang, Jiezhang Cao, Ce Liu, Rui Gong, Yulun Zhang, Hao Tang, Yun Liu, Denis Demandolx, et al.
Lsdir: A large scale dataset for image restoration.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 1775–1787, 2023.
- Liang et al. (2022a)
Jie Liang, Hui Zeng, and Lei Zhang.
Details or artifacts: A locally discriminative learning approach to realistic image super-resolution.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 5657–5666, 2022a.
- Liang et al. (2022b)
Jie Liang, Hui Zeng, and Lei Zhang.
Efficient and degradation-adaptive network for real-world image super-resolution.
In *European Conference on Computer Vision*, pp. 574–591. Springer, 2022b.
- Liang et al. (2021)
Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte.
Swinir: Image restoration using swin transformer.
In *Proceedings of the IEEE/CVF international conference on computer vision*, pp. 1833–1844, 2021.
- Lim et al. (2017)
Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee.
Enhanced deep residual networks for single image super-resolution.
In *Proceedings of the IEEE conference on computer vision and pattern recognition workshops*, pp. 136–144, 2017.
- Lin et al. (2024)
Xinqi Lin, Jingwen He, Ziyan Chen, Zhaoyang Lyu, Bo Dai, Fanghua Yu, Yu Qiao, Wanli Ouyang, and Chao Dong.
Diffbir: Toward blind image restoration with generative diffusion prior.
In *European Conference on Computer Vision*, pp. 430–448. Springer, 2024.
- Liu et al. (2015)
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.
Deep learning face attributes in the wild.
In *Proceedings of International Conference on Computer Vision (ICCV)*, December 2015.
- Loshchilov & Hutter (2017)
Ilya Loshchilov and Frank Hutter.
Decoupled weight decay regularization.
*arXiv preprint arXiv:1711.05101*, 2017.
- Lugmayr et al. (2022)
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool.
Repaint: Inpainting using denoising diffusion probabilistic models.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 11461–11471, 2022.
- Menon et al. (2020)
Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, and Cynthia Rudin.
Pulse: Self-supervised photo upsampling via latent space exploration of generative models.
In *Proceedings of the ieee/cvf conference on computer vision and pattern recognition*, pp. 2437–2445, 2020.
- Meyer et al. (2024)
Jordan Meyer, Nick Padgett, Cullen Miller, and Laura Exline.
Public domain 12m: A highly aesthetic image-text dataset with novel governance mechanisms.
*arXiv preprint arXiv:2410.23144*, 2024.
- Peebles & Xie (2023)
William Peebles and Saining Xie.
Scalable diffusion models with transformers.
In *Proceedings of the IEEE/CVF international conference on computer vision*, pp. 4195–4205, 2023.
- Rombach et al. (2021)
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.
High-resolution image synthesis with latent diffusion models, 2021.
- Saxena et al. (2023)
Saurabh Saxena, Charles Herrmann, Junhwa Hur, Abhishek Kar, Mohammad Norouzi, Deqing Sun, and David J Fleet.
The surprising effectiveness of diffusion models for optical flow and monocular depth estimation.
*Advances in Neural Information Processing Systems*, 36:39443–39469, 2023.
- Schuhmann et al. (2022)
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.
Laion-5b: An open large-scale dataset for training next generation image-text models.
*Advances in neural information processing systems*, 35:25278–25294, 2022.
- Shen et al. (2018)
Ziyi Shen, Wei-Sheng Lai, Tingfa Xu, Jan Kautz, and Ming-Hsuan Yang.
Deep semantic face deblurring.
In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pp. 8260–8269, 2018.
- Sun et al. (2024a)
Lingchen Sun, Rongyuan Wu, Zhiyuan Ma, Shuaizheng Liu, Qiaosi Yi, and Lei Zhang.
Pixel-level and semantic-level adjustable super-resolution: A dual-lora approach.
*arXiv preprint arXiv:2412.03017*, 2024a.
- Sun et al. (2024b)
Lingchen Sun, Rongyuan Wu, Zhengqiang Zhang, Hongwei Yong, and Lei Zhang.
Improving the stability of diffusion models for content consistent super-resolution.
*CoRR*, 2024b.
- Tsai et al. (2024)
Yu-Ju Tsai, Yu-Lun Liu, Lu Qi, Kelvin CK Chan, and Ming-Hsuan Yang.
Dual associated encoder for face restoration.
In *The Twelfth International Conference on Learning Representations*, 2024.
- Wan et al. (2020)
Ziyu Wan, Bo Zhang, Dongdong Chen, Pan Zhang, Dong Chen, Jing Liao, and Fang Wen.
Bringing old photos back to life.
In *proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 2747–2757, 2020.
- Wang et al. (2024a)
Hanhui Wang, Yihua Zhang, Ruizheng Bai, Yue Zhao, Sijia Liu, and Zhengzhong Tu.
Edit away and my face will not stay: Personal biometric defense against malicious generative editing.
*arXiv preprint arXiv:2411.16832*, 2024a.
- Wang et al. (2023a)
Jianyi Wang, Kelvin CK Chan, and Chen Change Loy.
Exploring clip for assessing the look and feel of images.
In *Proceedings of the AAAI conference on artificial intelligence*, volume 37, pp. 2555–2563, 2023a.
- Wang et al. (2024b)
Jianyi Wang, Zongsheng Yue, Shangchen Zhou, Kelvin CK Chan, and Chen Change Loy.
Exploiting diffusion prior for real-world image super-resolution.
*International Journal of Computer Vision*, 132(12):5929–5949, 2024b.
- Wang et al. (2024c)
Jingkai Wang, Jue Gong, Lin Zhang, Zheng Chen, Xing Liu, Hong Gu, Yutong Liu, Yulun Zhang, and Xiaokang Yang.
Osdface: One-step diffusion model for face restoration.
*arXiv preprint arXiv:2411.17163*, 2024c.
- Wang et al. (2021a)
Longguang Wang, Yingqian Wang, Xiaoyu Dong, Qingyu Xu, Jungang Yang, Wei An, and Yulan Guo.
Unsupervised degradation representation learning for blind super-resolution.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 10581–10590, 2021a.
- Wang et al. (2018)
Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy.
Recovering realistic texture in image super-resolution by deep spatial feature transform.
In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pp. 606–615, 2018.
- Wang et al. (2021b)
Xintao Wang, Yu Li, Honglun Zhang, and Ying Shan.
Towards real-world blind face restoration with generative facial prior.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 9168–9178, 2021b.
- Wang et al. (2021c)
Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan.
Real-esrgan: Training real-world blind super-resolution with pure synthetic data.
In *Proceedings of the IEEE/CVF international conference on computer vision*, pp. 1905–1914, 2021c.
- Wang et al. (2024d)
Yufei Wang, Wenhan Yang, Xinyuan Chen, Yaohui Wang, Lanqing Guo, Lap-Pui Chau, Ziwei Liu, Yu Qiao, Alex C Kot, and Bihan Wen.
Sinsr: diffusion-based image super-resolution in a single step.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 25796–25805, 2024d.
- Wang et al. (2023b)
Zhixin Wang, Ziying Zhang, Xiaoyun Zhang, Huangjie Zheng, Mingyuan Zhou, Ya Zhang, and Yanfeng Wang.
Dr2: Diffusion-based robust degradation remover for blind face restoration.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 1704–1713, 2023b.
- Wang et al. (2004)
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.
Image quality assessment: from error visibility to structural similarity.
*IEEE transactions on image processing*, 13(4):600–612, 2004.
- Wang et al. (2022)
Zhouxia Wang, Jiawei Zhang, Runjian Chen, Wenping Wang, and Ping Luo.
Restoreformer: High-quality blind face restoration from undegraded key-value pairs.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 17512–17521, 2022.
- Wei et al. (2020)
Pengxu Wei, Ziwei Xie, Hannan Lu, Zongyuan Zhan, Qixiang Ye, Wangmeng Zuo, and Liang Lin.
Component divide-and-conquer for real-world image super-resolution.
In *Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VIII 16*, pp. 101–117. Springer, 2020.
- Wu et al. (2023)
Haoning Wu, Zicheng Zhang, Weixia Zhang, Chaofeng Chen, Liang Liao, Chunyi Li, Yixuan Gao, Annan Wang, Erli Zhang, Wenxiu Sun, et al.
Q-align: Teaching lmms for visual scoring via discrete text-defined levels.
*arXiv preprint arXiv:2312.17090*, 2023.
- Wu et al. (2024a)
Rongyuan Wu, Lingchen Sun, Zhiyuan Ma, and Lei Zhang.
One-step effective diffusion network for real-world image super-resolution.
*Advances in Neural Information Processing Systems*, 37:92529–92553, 2024a.
- Wu et al. (2024b)
Rongyuan Wu, Tao Yang, Lingchen Sun, Zhengqiang Zhang, Shuai Li, and Lei Zhang.
Seesr: Towards semantics-aware real-world image super-resolution.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 25456–25467, 2024b.
- Xie et al. (2023)
Liangbin Xie, Xintao Wang, Xiangyu Chen, Gen Li, Ying Shan, Jiantao Zhou, and Chao Dong.
Desra: detect and delete the artifacts of gan-based real-world super-resolution models.
*arXiv preprint arXiv:2307.02457*, 2023.
- Yang et al. (2025)
Jiarui Yang, Tao Dai, Yufei Zhu, Naiqi Li, Jinmin Li, and Shu-Tao Xia.
Diffusion prior interpolation for flexibility real-world face super-resolution.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 39, pp. 9211–9219, 2025.
- Yang et al. (2020)
Lingbo Yang, Shanshe Wang, Siwei Ma, Wen Gao, Chang Liu, Pan Wang, and Peiran Ren.
Hifacegan: Face renovation via collaborative suppression and replenishment.
In *Proceedings of the 28th ACM international conference on multimedia*, pp. 1551–1560, 2020.
- Yang et al. (2023)
Peiqing Yang, Shangchen Zhou, Qingyi Tao, and Chen Change Loy.
Pgdiff: Guiding diffusion models for versatile face restoration via partial guidance.
*Advances in Neural Information Processing Systems*, 36:32194–32214, 2023.
- Yang et al. (2022)
Sidi Yang, Tianhe Wu, Shuwei Shi, Shanshan Lao, Yuan Gong, Mingdeng Cao, Jiahao Wang, and Yujiu Yang.
Maniqa: Multi-dimension attention network for no-reference image quality assessment.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 1191–1200, 2022.
- Yang et al. (2021)
Tao Yang, Peiran Ren, Xuansong Xie, and Lei Zhang.
Gan prior embedded network for blind face restoration in the wild.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 672–681, 2021.
- Yang et al. (2024)
Tao Yang, Rongyuan Wu, Peiran Ren, Xuansong Xie, and Lei Zhang.
Pixel-aware stable diffusion for realistic image super-resolution and personalized stylization.
In *European Conference on Computer Vision*, pp. 74–91. Springer, 2024.
- Yu et al. (2018)
Xin Yu, Basura Fernando, Richard Hartley, and Fatih Porikli.
Super-resolving very low-resolution face images with supplementary attributes.
In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pp. 908–917, 2018.
- Yuan et al. (2018)
Yuan Yuan, Siyuan Liu, Jiawei Zhang, Yongbing Zhang, Chao Dong, and Liang Lin.
Unsupervised image super-resolution using cycle-in-cycle generative adversarial networks.
In *Proceedings of the IEEE conference on computer vision and pattern recognition workshops*, pp. 701–710, 2018.
- Yue et al. (2023)
Zongsheng Yue, Jianyi Wang, and Chen Change Loy.
Resshift: Efficient diffusion model for image super-resolution by residual shifting.
*Advances in Neural Information Processing Systems*, 36:13294–13307, 2023.
- Zhang et al. (2024)
Howard Zhang, Yuval Alaluf, Sizhuo Ma, Achuta Kadambi, Jian Wang, and Kfir Aberman.
Instantrestore: Single-step personalized face restoration with shared-image attention.
*arXiv preprint arXiv:2412.06753*, 2024.
- Zhang et al. (2021)
Kai Zhang, Jingyun Liang, Luc Van Gool, and Radu Timofte.
Designing a practical degradation model for deep blind image super-resolution.
In *Proceedings of the IEEE/CVF international conference on computer vision*, pp. 4791–4800, 2021.
- Zhang et al. (2015)
Lin Zhang, Lei Zhang, and Alan C Bovik.
A feature-enriched completely blind image quality evaluator.
*IEEE Transactions on Image Processing*, 24(8):2579–2591, 2015.
- Zhang et al. (2023)
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.
Adding conditional control to text-to-image diffusion models.
In *Proceedings of the IEEE/CVF international conference on computer vision*, pp. 3836–3847, 2023.
- Zhang et al. (2018a)
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.
The unreasonable effectiveness of deep features as a perceptual metric.
In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pp. 586–595, 2018a.
- Zhang & Wu (2022)
Xi Zhang and Xiaolin Wu.
Multi-modality deep restoration of extremely compressed face videos.
*IEEE Transactions on Pattern Analysis and Machine Intelligence*, 45(2):2024–2037, 2022.
- Zhang et al. (2022)
Xindong Zhang, Hui Zeng, Shi Guo, and Lei Zhang.
Efficient long-range attention network for image super-resolution.
In *European conference on computer vision*, pp. 649–667. Springer, 2022.
- Zhang et al. (2018b)
Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu.
Image super-resolution using very deep residual channel attention networks.
In *Proceedings of the European conference on computer vision (ECCV)*, pp. 286–301, 2018b.
- Zhang et al. (2018c)
Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu.
Residual dense network for image super-resolution.
In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pp. 2472–2481, 2018c.
- Zhou et al. (2022)
Shangchen Zhou, Kelvin Chan, Chongyi Li, and Chen Change Loy.
Towards robust blind face restoration with codebook lookup transformer.
*Advances in Neural Information Processing Systems*, 35:30599–30611, 2022.

## Appendix A Appendix Overview

Due to space constraints in the main draft, we include implementation details, the data curation, and experimental results in the appendix. Specifically, in Sec. [B](https://arxiv.org/html/2510.09924v1#A2), we offer further explanation of the implementation of our framework and the experiments. In Sec. [C](https://arxiv.org/html/2510.09924v1#A3), we present the details of PortraitSR-4K. Finally, in Sec. [D](https://arxiv.org/html/2510.09924v1#A4), we present additional visual results of the main experiments.

## Appendix B Implementation Details

#### Backbone Settings.

We use the Stable Diffusion 2.1 model (Rombach et al., 2021) as the base model for LoRA finetuning. We employ the CVLFace (Kim et al., 2024) as the face feature extractor. The DAPE model as in (Wu et al., 2024b) is utilized to extract the language description of the input LQ images, while the negative prompt is fixed as “painting, oil painting, illustration, drawing, art, sketch, cartoon, CG Style, 3D render, unreal engine, blurring, dirty, messy, worst quality, low quality, frames, watermark, signature, jpeg artifacts, deformed, lowres, over-smooth” to avoid synthetic and low-quality styles.

Figure: Figure 5: Visualization of some portrait images sampled from PortraitSR-4K.
Refer to caption: figs/supp-vis-data.jpg

#### Model details.

We concatenated the LQ latent (4 channels), the reference latent (4 channels), and the resize binary mask (one channel) before sending them to the denoise UNet. We modify the ‘conv-in’ layer to enable the denoising UNet with a nine-channel input. The first four channels for the new convolution filter are initialized from the base model, while the five channels are initialized to zero. The bias of the convolution layer is initialized from the base model.

#### Face Alignment.

We leverage FaceLib to detect face landmarks following (Zhou et al., 2022). We then estimate the affine transformation between the detected landmark and a template landmark using the ‘estimateAffinePartial2D’ in OpenCV. The wrapping transformation is implemented using ‘affine_grid’ and ‘grid_sample’ in PyTorch to ensure the procedure is differentiable.

#### Training and Evaluation.

The training and inference resolution is at $1024\times 1024$. We train the model for 150k steps in two stages. The first consists of 40k steps, the ratio of training data is $\rm{PortraitSR-4K}:\rm{ffhq}:\rm{lsdir}:\rm{div2k}=0.15:0.05:1.7:0.3$. The second stage consists of the remaining 110k steps, and the ratio of mixed training data is $\rm{PortraitSR-4K}:\rm{ffhq}:\rm{lsdir}:\rm{div2k}=1.5:0.5:1.7:0.3$. The probability of dropping the reference image in both stages is 0.2. For evaluation metrics, we use pyiqa (Chen & Mo, 2022) to calculate the PSNR, SSIM, LPIPS, DISTS, NIQE, MUSIQ, CLIPIQA, and MANIQA-pipal scores. The rest of the metrics are calculated using their original codes.

#### User Study Details.

We collected 36 questionnaires for the win rate on similarity ($\rm{WR}_{\rm{id}}$), each questionnaire contains 58 questions randomly sampled from the testset. For each question, participants are given five face images produced by the five approaches in Tab. [2](https://arxiv.org/html/2510.09924v1#S4.T2), and one ground-truth face image. Participants are asked to select the image that is most similar to the ground-truth face image. We collected 28 questionnaires for the win rate on naturalness ($\rm{WR}_{\rm{N}}$), each questionnaire contains 58 questions randomly sampled from the testset. For each question, participants are provided with five face images produced by the five models. The ground-truth image is not provided. Participants are asked to select the most natural face out of the five given images. In both questionnaires, the order of options is randomly shuffled.

## Appendix C PortraitSR-4K Details

We selected the images that are at least $3840\times 2160$, with the longest side exceeding 3840 pixels, as candidates. The aspect ratio of the images ranges from 0.6 to 1.6. We further filter the images using the Q-align (Wu et al., 2023) aesthetic and quality scores.
We detect the face in each image using FaceLib. We drop faces whose distance between the two eyes is less than 64 pixels. We leverage  CVLFace(Kim et al., 2024) to estimate the similarity of face pairs. We exclude face pairs whose similarity is below 0.65. The reference face is cropped and aligned using the face alignment techniques described in Sec. [B](https://arxiv.org/html/2510.09924v1#A2). We visualize several portrait images from PortraitSR-4K in Fig. [5](https://arxiv.org/html/2510.09924v1#A2.F5).

## Appendix D Experimental Results

We show more visualization results on the PortraitISR task in Fig. [6](https://arxiv.org/html/2510.09924v1#A4.F6). As shown in the figure, our model achieves better face fidelity compared to general ISR models, while avoiding inconsistent borders compared to blending-based approaches. The visual results for the GISR task are shown in Fig. [7](https://arxiv.org/html/2510.09924v1#A4.F7), which demonstrate our competitive performance compared to the state-of-the-art models. The visual results for the FISR task are shown in Fig. [8](https://arxiv.org/html/2510.09924v1#A4.F8). Our model achieves performance comparable to that of the specialists for face image super-resolution.

Figure: Figure 6: Visual Comparison on PortraitISR. OSEDiff-Face is a blending-based approach, the background is handled by OSEDiff (Wu et al., 2024a) trained for general image super-resolution, and the face region is processed by a specialist DSEDiff model that is trained on the face dataset. Similarly, RESR+CF is the blending-based approach which combines Real-ESRGAN (Wang et al., 2021c) and CodeFormer (Zhou et al., 2022).
Refer to caption: x1.png

Figure: Figure 7: Visual Comparison on GISR. Our model achieves competitive results.
Refer to caption: x2.png

Figure: Figure 8: Visual Comparison on FISR. Our model achieves performance comparable to that of the FISR specialists.
Refer to caption: x3.png

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 4
presentation: 3
questions: See Weakness.
soundness: 2
strengths: 1, It jointly deals with general image SR and face image SR, producing good results without boundary artifacts.

2, It adds extra losses on face regions and introduces the face identiy loss.

3, It proposes a 4K portrait image SR dataset with reference faces.

4, It is well-written and easy-to-read.
summary: This paper proposes to deal with the boundary artifacts when applying face SR and general SR methods to the face and the rest background separately. In detail, based on OSEDiff, it adds extra face losses (including MSE loss, LPIPS loss, GAN loss and face identiy loss) on the cropped face regions. By jointly training general image SR and face image SR, it removes the blending inconsistency and generates natural images.
weaknesses: Overall, this paper deals with a new sub-problem in image SR with existing tools (one-step diffusion, MSE loss, LPIPS loss, GAN loss and face identity loss). This kind of exploration is useful and it does solve a specific real-world problem (the boundary artifacts after blending). However, the novelty on methodology is arguable. 

1, The OSEDiff proposes the training pipeline and training losses. It has also done experiments on general image SR and face image SR, separately. Compared with OSEDiff, the main contribution of this paper seems to be combining the general and face image SR losses.

2, In Table 4, why adding the face-aware loss can significantly improve the PSNR of the entire image? What is the average proportion of the face region in the image? Generally, adding an extra loss (face loss) will lead to increased loss on the origina loss (image loss). Can you show the origina loss value with and without face losses?

3, In Table 4, the ID-Score is similar to the face identity loss. Why does adding the identity loss make the ID-Score drop from 0.4348 to 0.3634?

4, In Table 4, adding the identity loss increases the PSNR from 25.07 to 25.85. Is it reasonable?

5, Lack of detailed experiments on how to choose the weighting parameters.

---

confidence: 2
contribution: 2
initial_rating: 4
presentation: 2
questions: Please refer to the weaknesses part.
soundness: 2
strengths: 1. This paper proposes an end-to-end framework for high-fidelity, face-aware PortraitISR using a single model.
2. This paper constructs a large-scale, high-resolution benchmark, PortraitSR- 4K, that contains 30k high-quality 4K portrait images, curated and filtered from web-scale data.
3. The proposed approach achieves state-of-the-art results in portrait ISR.
summary: This paper introduces HeadsUp, a novel framework designed to address the specific challenges of portrait image super-resolution (PortraitISR). Current super-resolution (ISR) methods typically focus on either general real-world images, which often produce unnatural faces , or strictly aligned facial images, which are incapable of restoring the background and other parts of a full portrait. A common practical solution involves blending a specialist face model with a general ISR model ; however, this "blending approach" inevitably introduces visible boundary artifacts and inconsistencies between the face and the background. HeadsUp overcomes these limitations as a single-step, end-to-end diffusion model that seamlessly restores and upscales the entire portrait image. It utilizes a "face supervision mechanism" to guide the model to focus on the facial region and integrates a reference-based mechanism to help with identity restoration and reduce ambiguity in low-quality inputs. To support this work, the authors also built PortraitSR-4K, a new large-scale, high-quality 4K portrait dataset. Experiments show that HeadsUp achieves state-of-the-art performance on the PortraitISR task.
weaknesses: 1. The paper mentions that inconsistency is a key challenge in PortraitSR, which arises from failing to handle the face and background at the same time. However, the paper doesn't clearly explain how its proposed end-to-end model actually solves this problem and manages to handle both parts simultaneously without creating new inconsistencies.
2. While the proposed method is straightforward and easy to implement, it lacks sufficient novelty.
3. The visual results in the paper do not effectively demonstrate that its performance is superior to other methods.

---

confidence: 4
contribution: 4
initial_rating: 6
presentation: 2
questions: Does PortraitSR-4K include demographic diversity (age, ethnicity) and non-frontal faces? If not, how do you ensure that HeadsUp does not exhibit performance bias (e.g., poor restoration quality for elderly faces)?
soundness: 2
strengths: HeadsUp’s end-to-end design eliminates blending, a long-standing limitation of portrait SR. Unlike blending methods (e.g., OSEDiff+OSEDiff Face), it processes the entire portrait in one pass, ensuring boundary consistency.
Experiments are comprehensive: covering PortraitISR (5 metrics + user study), general ISR (9 metrics), and face ISR (6 metrics), with ablations verifying each component’s necessity.
PortraitSR-4K’s curation is rigorous: filtering by resolution (≥4K), face size (eye distance ≥64px), and aesthetic score (Q-align), ensuring high data quality for training.
summary: The HeadsUp framework proposed in this paper addresses the core issues of "face-background boundary inconsistency" and "low face fidelity" in the Portrait Image Super-Resolution (PortraitISR) task. It innovatively constructs an end-to-end single-step diffusion model, achieving seamless super-resolution without post-processing fusion through face-aware supervision and a reference-guided mechanism.  Additionally, the authors built the PortraitSR-4K dataset—containing 30k high-quality 4K portrait images. Experiments show that HeadsUp achieves a PSNR of 25.64 dB on PortraitSR-4K (0.45 dB higher than OSEDiff), a PSNR of 22.924 dB on the Face Image Super-Resolution (FISR) task (1.475 dB better than CodeFormer), and maintains competitiveness in the General Image Super-Resolution (GISR) task. The overall design balances innovation and practicality. While the proposed approach is valuable and shows promising results, there are still several aspects that require further clarification and improvement.
weaknesses: The key structural details of the HeadsUp framework diagram are unclear. In the HeadsUp framework diagram, the authors start with a pre-trained latent diffusion model and incorporate LoRA adapters into the VAE encoder and denoising network. However, the diagram fails to indicate the specific location of the latent diffusion model or where the LoRA adapters are integrated. 
Moreover, the source of the latent diffusion model is not provided, nor is there any reference to its origin.
The paper introduces "reference image" as an optional input, but only injects it into the network in the form of latent concatenation, lacking a clear description of the feature interaction mechanism.
Experiments show that using reference images can improve the accuracy of identity recognition, but slightly sacrifices image quality. However, the authors have not conducted a theoretical analysis of this counterintuitive phenomenon. why does it slightly reduce image quality?

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 3
questions: Please refer to the weaknesses.
soundness: 2
strengths: 1.	Addressing the unified super-resolution of portraits with backgrounds is a challenging and practical problem, and the idea of a single, integrated model is solid.
2.	The creation of the PortraitSR-4K dataset adds favorable value to the community, offering a large-scale, high-quality benchmark for future research.
3.	The paper provides extensive ablation studies and comparative experiments.
summary: The paper tackles the challenge of face super-resolution within portrait images, proposing a unified diffusion-based framework called HeadsUp that aims to produce seamless, high-fidelity results without boundary artifacts. It introduces a face-aware region loss, an adaptive identity guidance mechanism, and a new high-resolution dataset (PortraitSR-4K) to facilitate training and benchmarking. The authors claim state-of-the-art performance on the PortraitISR task and competitive results on general and face-specific super-resolution benchmarks.
weaknesses: 1.	The core innovations are limited, i.e., face-aware loss and reference-guided identity guidance, which are incremental and lack thorough theoretical justification or innovative design that advances the state of the art significantly, especially in the area of face SR.
2.	Despite claiming superiority, the comparisons with recent leading face super-resolution models are limited, including the transformer, diffusion, and LLM-based models. Besides, the reported improvements are marginal, as shown in Table 3, only the LPIPS and DISTS are favorable, and the evaluation mainly relies on metrics that may not fully capture perceptual quality or identity preservation.
3.	The approach’s robustness to pose variations, lighting differences, occlusions, and severe degradations is not sufficiently addressed. This omission weakens the claimed applicability to real-world scenarios. More in-depth ablations should be added, especially the visualizations.
4.	Missing model efficiency analysis and comparison.