SSL: A Self-similarity Loss for Improving Generative Image Super-resolution

Authors: du  chen
zhengqiang  zhang
jie  liang
lei  zhang
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-20 11:17:31

Abstract: Generative adversarial networks (GAN) and generative diffusion models (DM) have been widely used in real-world image super-resolution (Real-ISR) to enhance the image perceptual quality. However, these generative models are prone to generating visual artifacts and false image structures, resulting in unnatural Real-ISR results. Based on the fact that natural images exhibit high self-similarities, i.e., a local patch can have many similar patches to it in the whole image, in this work we propose a simple yet effective self-similarity loss (SSL) to improve the performance of generative Real-ISR models, enhancing the hallucination of structural and textural details while reducing the unpleasant visual artifacts. Specifically, we compute a self-similarity graph (SSG) of the ground-truth image, and enforce the SSG of Real-ISR output to be close to it. To reduce the training cost and focus on edge areas, we generate an edge mask from the ground-truth image, and compute the SSG only on the masked pixels. The proposed SSL serves as a general plug-and-play penalty, which could be easily applied to the off-the-shelf Real-ISR models. Our experiments demonstrate that, by coupling with SSL, the performance of many state-of-the-art Real-ISR models, including those GAN and DM based ones, can be largely improved, reproducing more perceptually realistic image details and eliminating many false reconstructions and visual artifacts. Codes and supplementary material are available at https://github.com/ChrisDud0257/SSL

Main paper: 
 introduction: 
 Image super-resolution (ISR) is a fundamental problem in lowlevel vision. Given a low-resolution (LR) input, ISR aims to recover its high-resolution (HR) counterpart with high fidelity in contents, which has a wide range of applications in digital photography [28] , high definition display [83] , medical image analysis [27] , remote sensing [34] , etc. Starting from SRCNN [13] , various convolutional neural network (CNN) based methods have been proposed to improve the ISR performance, such as residual connections [22, 31, 39, 42] , dense connections [91] and channelattention [23, 90] . Recently, some transformer-based ISR methods [7-9, 17, 40, 89] have also emerged and demonstrated more powerful performance.In the early stage, researchers usually employed simple degradations, such as bicubic downsampling and downsampling after Gaussian smoothing, to synthesize the LR-HR training pairs, while focusing on the study of ISR network design. However, the image degradations in real-world are much more complex, and the ISR [40] , StableSR [63] , our SSL guided StableSR and the ground-truth (GT) image. SwinIR-GAN produces over-smoothed and wrong results, while Sta-bleSR produces more details but with false structures and artifacts. Our SSL guided StableSR generates more faithful details while suppressing much the artifacts. models trained on those simple synthetic data can hardly be generalized to real-world applications. Therefore, in recent years many works have been done on real-world ISR (Real-ISR), aiming to obtain perceptually realistic ISR results on degraded images in real-world scenarios [10, 19, 36-38, 45, 53, 64, 65, 71, 73, 77, 80, 82, 85, 93] . Some researchers proposed to collect real-world LR-HR image pairs by using long-short camera focal lengths [4, 5, 69, 70, 74, 75, 88] ; however, this is very costly and the trained models may only work well when similar photographing devices are used. Therefore, researchers propose to synthesize more realistic training data by designing more complex degradation models. The notable works include BSRGAN [84] and Real-ESRGAN [65] . In BSRGAN [84] , Zhang et al. randomly shuffled and combined blur, downsampling and noise degradations to form a complex degradation, while in Real-ESRGAN [65] , Wang et al. developed a high-order degradation model with several repeated degradation operations. Recently, researchers have also proposed to introduce human guidance into the training data generation process [6] .Given the training data with more realistic degradations, another issue is how to train the network to achieve the goal of Real-ISR. It is well-known that the 𝐿 1 or 𝐿 2 loss, which aims to minimize the fidelity error, often results in over-smoothed image details. To tackle this issue, in the past a few years, the generative adversarial networks (GANs) [18] have been widely adopted to train Real-ISR models [33, 35, 41, 46, 52, 66, 67, 87] . With the help of adversarial loss, GAN models can learn to find an image reconstruction path to generate more sharp details. Though great progress has been made, one critical limitation of GAN based Real-ISR models remain, i.e., they incline to hallucinate visually unpleasant artifacts. Very recently, with the rapid development of diffusion models (DMs) [25, 59] , it becomes popular to leverage the pre-trained large scale text-to-image models, such as stable diffusion (SD) [55] , to achieve Real-ISR. Benefiting from the strong generative priors in DMs, some recent works [56, 63, 79] have demonstrated encouraging Real-ISR results with fine-scale and realistic details. However, DMs have high randomness, which lead to unstable Real-ISR outputs and false image details.In this paper, we aim to improve the GAN and DM based Real-ISR methods, reducing the artifacts and producing more realistic details, by proposing a new training loss function. It is well-known that natural images exhibit repetitive patterns across the whole image. Such a property of self-similarity has been extensively used in many image restoration algorithms, such as BM3D [11] , NCSR [15] , WNNM [21] , and NLSN [49] , where the image self-similarity is used as a prior to regularize the restored image. In this work, we employ the image self-similarity property as a powerful penalty to supervise the Real-ISR training progress. The proposed image self-similarity loss (SSL) could act as a plug-and-play penalty in most of the existing generative Real-ISR models, guiding them to exploit more effectively the inherent image self-similarity information for detail reconstruction. Specifically, we compute a self-similarity graph (SSG) to describe the image structural dependency, and minimize the distance between the SSGs of the ground-truth (GT) and Real-ISR output to optimize the model. To make the training process more efficient and focus more image edge/texture areas, we generate an edge mask from the GT image in an offline manner, and only build the SSG upon the edge pixels.Our proposed SSL can be easily adopted into the off-the-shelf GAN-based and DM-based Real-ISR models as an extra penalty to enhance image details and reduce the unpleasant artifacts. An example is shown in Fig. 1 . One can see that SwinIRGAN [40] oversmooths the image textures and generates wrong details, while the recent DM-based StableSR [63] restores much clearer details but still fails to generate some fine scale structures or correct textures. In comparison, the StableSR model trained with our SSL could reconstruct both clear content and more realistic textures with better perception quality. Our extensive experiments on state-ofthe-art Real-ISR models validate the effectiveness of our proposed SSL, either in GAN-based or DM-based ISR tasks. 
related work: 
 Traditional Fidelity-Oriented ISR. Since SRCNN [13] , many CNN backbones have been developed to promote the ISR performance in terms of PSNR and SSIM [68] measures. EDSR [42] and RDN [91] incorporate residual and densely connections, respectively. RCAN [90] , OA-DNN [16] , CRAN [92] and HAN [51] make use of channel/spatial attention modules. Recently, transformerbased models have shown stronger ability towards long-range dependency modeling. SwinIR [40] utilizes shifted partition windows to compute the image self-attention. ELAN [89] introduces multiscale self-attention blocks to extract long-distance dependency. ACT [78] utilizes CNN to extract local interaction and transformer to obtain long-range dependency.GAN-based Generative Real-ISR. The fidelity-oriented ISR models often generate over-smoothed details, sacrificing the perceptual quality of natural images. Inspired by GAN [18] , many generative ISR methods have been proposed to obtain more photo-realistic results. SRGAN [33] , and ESRGAN [67] utilize VGG-style [58] discriminator to perform the adversarial training. BSRGAN [84] and Figure 3 : Illustration of the self-similarity graph (SSG) computing process. We first generate a mask to indicate the image edge areas by applying the Laplacian Operator on the GT image. During the training period, for each edge pixel in the mask, we find the corresponding pixels in the GT image and ISR image, and set a search area centred at them. A local sliding window is utilized to calculate the similarity between each pixel in the search area and the central pixel so that an SSG can be respectively computed for the GT image and the ISR image, with which the SSL can be computed. The red pixel means the edge pixel, while the blue block means the sliding window.Real-ESRGAN [65] introduce complex degradation processes to synthesize the real-world degradation. HGGT [6] annotates positive and negative training pairs to enhance the perceptual quality in GAN training progress. In order to make the GAN training process more stable, SROBB [54] presents an enhanced perceptual loss to restrain the model with different semantic labels. SPSR [46] embeds the well-extracted structure prior into the RRDB network [67] . RankSRGAN [87] firstly trains a ranker model to indicate the relative perception quality of an image, and then utilizes a well-trained ranker to guide the generator to reconstruct better details. BebyGAN [35] searches the best candidate GT patch in the neighborhood to perform LR-HR supervision. LDL [41] computes an artifact map to indicate the local artifacts in ISR outputs, then imposes appropriate penalty on the artifact areas to improve the perceptual quality.DM-based Generative Real-ISR. The powerful generative priors embedded in DMs can be exploited for Real-ISR. SR3 [56] utilizes a conditional pixel-level DM to iterativelly denoise and generate super-resolved results. StableSR [62] performs Real-ISR in the latent diffusion space by using a controllable feature wrapping module to balance between the reconstruction fidelity and the perceptual quality. PASD [76] introduces a pixel-aware cross attention block as a controllable module to guide the high-quality details generation. DiffBIR [43] utilizes a two-stage model, which first reduces complicated degradation factors and then uses the well-trained generative prior in SD to reconstruct delicate contents. DiffIR [72] pre-trains a DM with high-quality GT images to obtain abundant priors and then finetunes the DM with their low-quality counterparts to complete the Real-ISR task. ResShift [79] adopts an efficient sampling strategy through shifting the residual between high-quality and low-quality images to largely accelerate the diffusion steps. 
image self-similarity loss: 
 The proposed training framework is illustrated in Fig. 2 . In addition to the commonly used 𝐿 1 , perceptual loss, adversarial loss in GAN-based methods, or the Gaussian noise prediction MSE loss in Table 1 : Quantitative results of seven representative GAN-based Real-ISR models and their counterparts coupled with the proposed SSL. The bicubic degradation model is used here. For each of the seven groups of comparisons, the better results are highlighted in boldface. The PSNR and SSIM indices are computed in the Y channel of Ycbcr space. DM-based methods, we calculate the self-similarity graphs (SSG) of both ISR output and ground-truth (GT), and consequently introduce a self-similarity loss (SSL) between them to supervise the reconstruction of image details and structures.For a natural image, one can observe many repetitive patterns across it, known as the image self-similarity. Such a property has been used to improve the image restoration performance for a long time [3, 11] . Actually, the self-attention mechanism [44, 61] in transformer models exploits the image self-similarity in deep feature space. In this paper, we adopt the Exponential Euclidean distance [3] to calculate the self-similarity. For any two patches𝐼 𝑝 , 𝐼 𝑞 ∈ R (2𝑓 +1) × (2𝑓 +1) ×𝐶 centered at pixels 𝜇 𝑝 and 𝜇 𝑞 in image 𝐼 ∈ R 𝐻 ×𝑊 ×𝐶 , respectively, where 𝑓 denotes the patch radius, 𝐻 , 𝑊 and 𝐶 are the image height, width and channel number (𝐶=3 for RGB images), we firstly compute the squared Euclidean distance between 𝐼 𝑝 and 𝐼 𝑞 :EQUATIONwhere 𝜇 𝑖 𝑝+𝑗 and 𝜇 𝑖 𝑞+𝑗 denote the neighborhood pixels around 𝜇 𝑖 𝑝 and 𝜇 𝑖 𝑞 in patch 𝐼 𝑝 and 𝐼 𝑞 , respectively. The similarity 𝑆 (𝐼 𝑝 , 𝐼 𝑞 ) between 𝐼 𝑝 and 𝐼 𝑞 is calculated as:EQUATIONwhere ℎ > 0 is a scaling factor. One can see that 0 ≤ 𝑆 (𝐼 𝑝 , 𝐼 𝑞 ) ≤ 1.When the Euclidean distance 𝑑 2 (𝐼 𝑝 , 𝐼 𝑞 ) approaches to 0, the similarity 𝑆 (𝐼 𝑝 , 𝐼 𝑞 ) approaches to 1, indicating that the two patches are highly similar.By using the self-similarity measure defined in Eq. 2, we could compute the similarity of a patch with all the other patches in the whole image, and construct a self-similarity graph (SSG). However, this is computationally expensive because the size of such an SSG will be 𝐻 where 𝑡 is a threshold. We empirically set it to 20 to retain most of the true edge pixels while filtering out smooth and trivial image features. 𝑀 is computed in an off-line manner to avoid the repetitive computation in each iteration.In the training progress, for pixels at (𝑖, 𝑗) where 𝑀 𝑖,𝑗 = 1, we find the corresponding RGB pixels 𝜇 𝑝 in the GT image and the ISR output, and calculate their SSG for comparison. On the DF2K_OST training dataset, the edge pixels occupy only 13% of the total amount of image pixels. By using 𝑀 to guide the construction of SSG, we can not only reduce significantly the training cost, but also concentrate on the image edges and textures.For an edge pixel 𝑝 in the original RGB image 𝐼 (the corresponding pixel in the Mask 𝑀 is 𝑀 𝑝 = 1), we define a search area 𝐼 𝐾 𝑠 ∈ R 𝐾 𝑠 ×𝐾 𝑠 ×𝐶 as well as a local window 𝐼 𝑝 ∈ R 𝐾 𝑤 ×𝐾 𝑤 ×𝐶 centered at it, where 𝐾 𝑤 = 2𝑓 + 1 and 𝑓 is the radius of the window. Then for each pixel 𝑞 in the search area, we extract a sliding window 𝐼 𝑞 ∈ R 𝐾 𝑤 ×𝐾 𝑤 ×𝐶 to calculate its similarity with 𝐼 𝑝 , i.e., 𝑆 (𝐼 𝑝 , 𝐼 𝑞 ), by Eq. 1 and Eq. 2. Then we normalize 𝑆 (𝐼 𝑝 , 𝐼 𝑞 ) as:EQUATIONwhere 𝜖 = 𝑞 ∈𝐼 𝐾𝑠 𝑆 (𝐼 𝑝 , 𝐼 𝑞 ) is the normalization factor.The overall calculation procedure of SSG is illustrated in Fig. 3 . To be more specific, for each edge pixel in the mask, we find the corresponding pixels in the GT image and ISR image, and set a search area centred at them. Then we set a local sliding window to calculate the similarity between the patch centered at the central pixel and another patch centered at the pixels in the search area. All the values of S (𝐼 𝑝 , 𝐼 𝑞 ) builds the SSG of image 𝐼 , which describes the inherent structural similarity distribution of the image. In practice, we could sample 𝐼 𝑞 with a stride 𝑠 to further reduce the computational cost (we set 𝑠 = 3 in our implementation).Denote by S𝐻𝑅 and S𝐼𝑆𝑅 the SSG of the GT image and the ISR output, respectively. We can use their distance as the loss to supervise the network training. Here we employ the KL-divergence and 𝐿 1 distance to build the SSL:EQUATIONwhere 𝛼 is a balance parameter and we simply set it as 1 in all our experiments. SSL in GAN-based Models. To apply SSL into an off-the-shelf GAN-based Real-ISR method, we just need to add the above 𝐿 𝑆𝑆𝐿 loss to its original loss function 𝐿 𝑜𝑟𝑖𝑔𝑖𝑛𝑎𝑙 (e.g. pixel-wise 𝐿1 loss, perceptual loss and GAN loss), and then re-train the model:EQUATIONwhere 𝛽 is a balance parameter. SSL in DM-based Models. As for those latent DM-based Real-ISR methods, StableSR [63] and ResShift [79] , the 𝐿 𝑜𝑟𝑖𝑔𝑖𝑛𝑎𝑙 is applied to predict the desired noise in latent space. Since SSL is computed in image space, we need to pass the predicted noise through the VAE decoder to output the ISR image, as shown in Fig. 2(b) , and then apply the SSL to the reconstructed image. We also employ a pixel-wise 𝐿 1 loss for more stable training. The total loss is:EQUATIONwhere 𝛽, 𝛾 are balance parameters. The 𝐿 𝑆𝑆𝐿 and 𝐿 1 will backpropagate their gradients to update the parameters of the denoising UNet and the controlling parts in DMs. 
experiments on gan-based models: 
 Comparison Methods. Our proposed SSL can be directly applied to the existing GAN-based Real-ISR models with either simple bicubic degradation or complex mixture degradations [65, 84] as a plug-and-play module to improve their performance. For bicubic degradation, we embed SSL into ESRGAN [67] , RankSRGAN [87] , SPSR [46] , BebyGAN [35] and LDL [41] . For complex mixture degradations, we embed SSL into Real-ESRGAN [65] and BSRGAN [84] . Most of the above models employ the CNN backbone (e.g., RRDB [67] or SRResNet [33] ) as the generator. In this paper, we also employ the transformer backbones, i.e., SwinIR [40] and ELAN [89] , as the generator, resulting in the SwinIRGAN and ELANGAN models. For each of the above Real-ISR models (e.g., ESRGAN), we denote by "*-SSL" (e.g., ESRGAN-SSL).Training Details. For each of the evaluated Real-ISR methods, we train its SSL guided counterpart with the same patch size and training dataset (i.e., DIV2K [60] , DF2K [1, 60] and DF2K-OST [1, 60, 66] ) as the original method. In the experiments with complex degradations, since the original degradation setting in Real-ESRGAN and BSRGAN is too heavy, we follow the Real-ESRGAN and BSRGAN settings in HGGT [6] (which has weaker degradation level) for training data generation. The Adam [32] optimizer is adopted. The initial learning rate is set to 1e-4, which is halved after 200K iterations for CNN backbones, and 200K, 250K, 275K, 287.5K iterations for transformer backbones. When calculating SSG, the search area 𝐼 𝐾 𝑠 is set to 25, the sliding window 𝐼 𝐾 𝑤 is set to 9, and the scaling factor ℎ is set to 0.004. 𝛽 is set to 1000. All experiments are conducted on NVIDIA RTX 3090 GPUs. All of the SSL guided models are fine-tuned from a well-trained fidelity-oriented version (e.g. RRDB [67] , SwinIR [40] or ELAN [89] which is trained only with 𝐿1 loss without a discriminator) for better initialization.Evaluation Datasets and Metrics. We employ the widely-used testing benchmarks, including Set5 [2] , Set14 [81] , DIV2K100 [60] , Urban100 [26] , BSDS100 [47] , Manga109 [48] , General100 [14] , to evaluate the competing methods. Considering the fact that there is certain randomness in the synthesis of LR images when using the complex mixture degradation models in [65, 84] , for each test image, we synthesize a group of 30 LR images using randomly sampled degradation factors, and report the averaged metrics for fair and solid evaluation. We compute PSNR and SSIM [68] in the Y channel for fidelity measurement. For perceptual quality, LPIPS [86] and DISTS [12] are used for quantitative assessment.Results for Bicubic Degradation. Table 1 shows the quantitative results of different Real-ISR models when bicubic degradation is used. It can be seen that on all the 7 testing datasets, our SSL guided models surpass their original counterparts in most of the fidelity (PSNR, SSIM) and perceptual (LPIPS, DISTS) measures, no matter the CNN or transformer backbones are used. This demonstrates that the image SSG could characterize the image inherent structures, and our SSL could provide effective supervision in the Real-ISR model training process, enforcing the models to hallucinate more correct contents with better fidelity and suppressing the visual artifacts to achieve better perceptual quality. It is worth mentioning that our SSL will not introduce any extra cost in the inference process.Qualitative Result. Fig . 4 provides visual comparisons between major Real-ISR models and their SSL-guided version in the case of bicubic degradation. One can clearly see that the SSL-guided models could generate clearer textures (the 1st column), or richer details (the 2nd column) and correct the twisted textures (the 3rd/4th/5th/6th/7th columns) generated by the original models. Such observations echo with the results in Table 1 , proving again that SSL could hallucinate correct details and suppress artifacts.Results for Complex Degradation. Due to the page limit, we provide the quantitative results of GAN-based Real-ISR models and their SSL guided versions under complex image degradation, as well as the visualization comparisons between them, in the supplementary material.Comparison Methods. We embed SSL into three representative DM-based models, including StableSR [62] , ResShift [79] and DiffIR [72] . For each of the above Real-ISR models (e.g., StableSR), we denote by "*-SSL" (StableSR-SSL).Training Details. For each of the evaluated DM-based Real-ISR method, we employ the same training datasets (including DF2K-OST [1, 60, 66] , DIV8K [20] , FFHQ [29] ), and apply the same degradation pipeline as that used in StableSR [62] . The training patch size and iterations in SSL-guided versions are set to the same as the original method. The Adam [32] optimizer is used. The learning rate is fixed to 5e-5. When calculating SSG, the search area 𝐼 𝐾 𝑠 is set to 25, the sliding window 𝐼 𝐾 𝑤 is set to 9, and the scaling factor ℎ is set to 0.004. For SSL guided StableSR and DiffIR, the weight 𝛽 and 𝛾 in Eq. 7 are set to 1 and 0.1, respectively. For SSL guided DiffIR, since the original model [72] already utilizes pixel-wise 𝐿1 loss, then we implement the loss function type as Eq. 6, 𝛽 is set to 1000. All experiments are conducted on NVIDIA V100 GPUs. We update all of the parameters of UNet in the pre-trained DM as well as the controlling parts towards the SSL-guided counterparts.Evaluation Datasets and Metrics. We utilize the testing images from StableSR [62] , including the 3000 synthesized DIV2K100 low-quality testing images (each GT image has a group of 30 LR images generated from DIV2K100 [60] dataset with complicated degradation factors), RealSR [4] (100 real-world low-quality images with their corresponding GTs obtained by cameras), DRealSR [70] (93 real-world low-quality images with their corresponding GTs captured by cameras), DPED-iphone [28] (113 real-world low-quality images taken by iPhone without GT). We compute full-reference image quality metrics, including PSNR, SSIM [68] , LPIPS [86] and DISTS [12] , and no-reference image quality metrics, including NIQE [50] , CLIP-IQA [62] and MUSIQ [30] . The statistical distance metric FID [24] is also calculated.Quantitative Result. Tab. 2 shows the numerical results of the original DM-based Real-ISR methods and their SSL guided versions. One can see that StableSR-SSL obtains better no-reference metrics (NIQE/CLIP-IQA/MUSIQ) while gets worse full-reference metrics (PSNR/SSIM/LPIPS/DISTS). ResShift-SSL gets better full-reference metrics (PSNR/SSIM/LPIPS/DISTS) but worse no-reference metrics (NIQE/CLIP-IQA/MUSIQ). DiffIR-SSL gets better perceptualrelevant metrics (LPIPS/DISTS/NIQE/CLIP-IQA/MUSIQ). While different SSL guided models obtain different performance, it is still reasonable for the following reasons: (1). StableSR-SSL leverages a pretrained Stable-Diffusion model [55] , which is trained on LAION-5B [57] , a multi-modal dataset that contains a vast number of textto-image pairs. This results in a distinct data distribution divergence when compared to the general training datasets used for SR tasks, such as DF2K [1, 60] and DIV8K [20] . Consequently, during inference stage, the results generated by StableSR-SSL, exhibit a notable difference from the GT in the test set (such as DIV2K100). Thus all full-reference metrics get failure, while get much better no-reference metrics, which also indicate better perception quality.(2) ResShift-SSL gets better FR-IQA (PSNR/SSIM/LPIPS/DISTS/FID) results, this indicated that SSL could help ResShift reconstruct textures with higher fidelity. As for the worse NR-IQA metrics, this is mainly because the existing NR-IQA metrics, including NIQE, CLIP-IQA and MUSIQ, favor the images with more high-frequency details, even these details are wrong. As can be seen from Fig. 6 in the supplementary, ResShift hallucinates many wrong details (e.g., on the windows in column 1), while ResShift-SSL successfully removes those artifacts. Our user study in Fig. ?? also showed that 73.07% of the observers pick the results of ResShift-SSL. However, the NR-IQA metrics prefer the results of ResShift because they are not accurate enough to represent the human perception yet. (3). DiffIR-SSL not only trains a latent-diffusion model from scratch, but also utilizes a discriminator. Due to the influence introduced by the discriminator, the PSNR/SSIM just get worse, but obtains better perceptionrelevant metrics (LPIPS/DISTS/NIQE/CLIP-IQA/MUSIQ).Qualitative Result. Fig. 5 shows the visualization results. One can see that compared with the original DM-based Real-ISR methods, their SSL guided versions perform significantly better in restoring the image structures and details, demonstrating the strong structure regularization capability of SSL. For example, StableSR generates false patterns in the T-shirt (column 1) and incomplete details on the peristyle (column 2), while the SSL guided StableSR restores correct T-shirt pattern and hallucinates more complete structures on peristyle. For ResShift, it either over-smooths the details (column 3) or generate wrong textures (column 4), while SSL can help to solve this issue. Similar observations go to DiffIR. All these results validate the effectiveness of SSL in encouraging the Real-ISR model to generate more delicate details. More visual comparisons that validate the perception imporvement of SSL can be found in the supplementary material.Due to the limited pages, we also provide the following contents in our supplementary material: (1) . A user study to validate the performance of SSL. (2) . The training cost analysis towards SSL. (3) . Ablation studies about the selection of hyper-parameters, such as 𝐾 𝑠 , 𝐾 𝑤 , 𝛽 in Eq. 6 and Eq. 7. (4). Limitation of SSL. 
conclusion: 
 Generative image super-resolution methods, including GAN-based and DM-based ones, are prone to generating visual artifacts. In this work, we proposed a novel use of the image self-similarity prior for improving the generative real-world image super-resolution results. Specifically, we explicitly computed the self-similarity graph (SSG) of the image, and took the difference between the SSG maps of ground-truth image and Real-ISR output as a self-similarity loss (SSL) to supervise the network training. SSL could be easily embedded in off-the-shelf Real-ISR models, including GAN-based and DM-based ones, as a plug-and-play penalty, guiding the model to more stably generate realistic details and suppress false generations and visual artifacts. Our extensive experiments on benchmark datasets validated the generality and effectiveness of the proposed SSL in generative Real-ISR tasks. 
 
 Appendix:  


Decision: The final ratings are 3 weak accept. The reviewers acknowledged sufficient novelty and extensive evaluation. Initial concerns seemed to be resolved mostly through rebuttal.

Reviews: review: paper_summary: paper_summary: The authors propose a novel loss function which can improve the performance of image super resolution models. It take advantage of the self similarity within images to produce more accurate textures (typically repeated patterns and around edges). They compute this self-similarity loss by computing a scene graph on image patches and comparing the distribution. This loss produces more realistic textures in the reconstructions and improves quantitative image comparison metrics. Performance is also validated by a user study.
strength_weakness: strength_weakness: * Creative and novel solution
* Extensive evaluation
* Good results
limitations: limitations: * The explanations for worse quantitative performance on the diffusion-based methods is weak. I find these claims are made without any evidence to back it up and the explanations are not sufficient, for instance why performance on ResShift-SSL has consistently worse performance in NQIE, CLIP-IQA, and MUSIQ
* No training time comparison with/without their metric. As computing the proposed metric requires many calculations the affect on training time should be examined.
* I would appreciate to have more comparisons on smoother textures, to ensure training with the proposed does not reduce performance on those areas.
paper_summary: paper_summary: This paper aims at improving image super-resolution with the self-similarity loss. The self-similarity is one characteristic of images. The proposed self-similarity loss can be plugged into existing GAN-based or Diffusion-based Image Resolution solutions. Extensive experiments are performed to demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. Identify the artifacts of current image super-resolution techniques. Current methods generate either false structures or over-smoothed results. The proposed method generates more faithful details while suppressing the artifacts.
2. Explore the choice of several parameters of self-similarity calculation including the K_w and K_s which denotes the radius of the window and the search area. Also perform experiments to explore the new loss weight in GAN and Diffusion driven techniques.
3. In order to avoid naively applying the self-similarity on all pixels for one image, authors propose to exploit the mask at the edge area to selectively calculate important self-similarity of pixels.
limitations: limitations: 1. No further analysis of mask choice. Authors use edge masks to reduce the computation cost. Are all pixels at the edges the best to calculate self-similarity? In other words, how do we define the good pixels of self-similarity loss to improve the image super-resolution.
2. Lack the experiments to verify whether the proposed self-similarity loss can benefit unsupervised image super-resolution solutions.
3. The proposed method analyzes several parameters like radius of the window, search area, loss weight setting. Are these parameters related to the image resolution or other loss levels? In the multi-loss items, what is the characteristic of the best parameter setting? With this kind of analysis, we can further understand how to set the tunable parameters.
paper_summary: paper_summary: This paper proposes a self-similarity loss (SSL) to improve the performance of generative models for real-world image super-resolution (Real-ISR), such as GANs and DMs. Natural images exhibit self-similarity property - local patches can have similar patches in the whole image. The paper computes self-similarity graphs (SSG) to describe this property for ground-truth and Real-ISR images. An edge mask is used to calculate SSG on edge areas only. SSL minimizes the distance between SSGs to supervise detail/structure generation and reduce artifacts. Experiments show the effectiveness of SSL in enhancing state-of-the-art generative Real-ISR models.
strength_weakness: strength_weakness: -	The self-similarity property is well-motivated and has been successfully used in image restoration before. Leveraging it for Real-ISR training is reasonable.
-	The edge mask and SSG computation make the method efficient and focus on important areas.
-	The SSL can be easily added to existing Real-ISR models as a plug-and-play module.
-	Comprehensive experiments are conducted on benchmark datasets to validate the method.
limitations: limitations: - Lack of comparison with some SOTA methods, like FeMaSR and DeSRA.
- How about adopting other edge detection operations rather than Laplacian operator.
- Lack of potential limitations of SSL