One-to-many Approach for Improving Super-Resolution
Authors: Sieun Park
Caleb Eunho Lee
Venue: NeurIPS 2021 Submitted (2021)
Keywords: Super resolution
Deep learning

Abstract: Super-resolution (SR) is a one-to-many task with multiple possible solutions. However, previous works were not concerned about this characteristic. For a one-to-many pipeline, the generator should be able to generate multiple estimates of the reconstruction, and not be penalized for generating similar and equally realistic images. To achieve this, we propose adding weighted pixel-wise noise after every Residual-in-Residual Dense Block (RRDB) to enable the generator to generate various images. We modify the strict content loss to not penalize the stochastic variation in reconstructed images as long as it has consistent content. Additionally, we observe that there are out-of-focus regions in the DIV2K, DIV8K datasets that provide unhelpful guidelines. We filter blurry regions in the training data using the method of [10]. Finally, we modify the discriminator to receive the low-resolution image as a reference image along with the target image to provide better feedback to the generator. Using our proposed methods, we were able to improve the performance of ESRGAN in x4 perceptual SR and achieve the state-of-the-art LPIPS score in x16 perceptual extreme SR.

One-to-many Approach for Improving Super-Resolution
Anonymous Author(s)
Affiliation Address
email
Abstract
Super-resolution (SR) is a one-to-many task with multiple possible solutions. However, previous works were not concerned about this characteristic. For a one- to-many pipeline, the generator should be able to generate multiple estimates of the reconstruction, and not be penalized for generating similar and equally realistic images. To achieve this, we propose adding weighted pixel-wise noise after every Residual-in- -n-Residual Dense Block (RRDB) to enable the generator generate various images. We modify the strict content loss to not penalize the stochastic variation in reconstructed images as long as it has consistent content. Additionally, we observe that there are out-of-focus regions in the DIV2K, DIV8K datasets that provide unhelpful guidelines. We filter blurry regions the training data using the method of [10]. Finally, we modify the discriminator to receive the low- resolution image as a reference image along with the target image to provide better feedback to the generator. Using our proposed methods, we were able to improve the performance of ESRGAN x4 perceptual SR and achieve the state-of-the-art LPIPS score in x 16 perceptual extreme SR.
10
11
12
14
Introduction
16
Super-resolution is the task of recovering high-resolution (HR) image from low-resolution (LR) 18 image. Recent works have achieved significant performance in SR using deep convolutional neural network (CNN) based approaches. Some of them exploit strict content loss as the training objective
20 for super-resolution and propose various network architectures to improve the PSNR score. However,
21 these methods often result in overly smooth images and have poor perceptual quality [6]. Another
22 branch works focuses on improving perceptual quality with perceptual training methods [1,6,7]. These methods employ generative adversarial networks (GAN) and perceptual loss functions to drive
24 the network's output towards the natural image manifold of possible HR images. We assess an ld further improve the perceptual quality of these works.
25
26 Because super-resolution is a one-to-many problem with multiple possible reconstructions for one image, methods based strict content loss often lead to predicting the average of possible reconstruc- tions[6]. Perceptual-driver solutions utilize perceptual and adversarial loss, which both don' penalize the generator for generating equally realistic images with stochastic variance. However,
28
discover
30 two incomplete aspects in the current perceptual SR pipeline. First, although the above-mentioned
31 losses don't penalize stochastic variation, the final loss is mixed with the strict content loss which
32 strictly penalizes these variations. Second, the generator doesn't have the ability to generate multiple estimates of the image despite one-to-man ma-y problem. To implement such one-to-many pipeline, we provide the generator with pixel-wise noise and improve the content loss so it doesn't 't restrict the variation in the image while ensuring the consistency of the content.
34
35
36
The key contributions of our work can be described as follows:
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

37
We propose a weaker content loss that does not penalize generating and stochastic variation in the image.
detail
38
39
We enable the generator to generate diverse outputs by adding scaled pixel-wise noise after each RRDB block.
40
41
We filter blurry regions in the training data using Laplacian activation[10] 100.
42
We additionally provide the LR image to the discriminator to give better gradient feedback to the generator.
43
Related work
44
45
Since the pioncering work of SRCNN[9], many works have exploited the pixel-wise loss and PSNR- oriented training objectives to learn the end-to-end mapping from LR to HR images. We denote such pixel-wise losses as the strict content loss. Many network architectures and techniques were experimented with to improve the complexity of such networks. Deeper network architectures(11 [17], residual networks[6]. channel attention [8], and techniques to remove batch normalization [19] were introduced. Although these works achieved state-of of- SR performance in the peak signal-to- noise ratio (PSNR) metric, they often produce overly smooth images.
48
47
48
49
50
51
52
To improve the perceptual image quality of SR, SRGAN [6] proposes perceptual loss and GAN-based training. The perceptual loss is measured using intermediate activations of the VGG-19 network and a discriminator is used for the adversarial training process. Enhanced SRGAN (ESRGAN) further improves SRGAN by modifying the generator architecture with Residual in Residual Dense Block (RRDB), the Relativistic GAN [16] loss, and improving the perceptual loss. Such methods were superior to PSNR oriented methods at generating ahoto-realistic SR images with sharp details, achiev- ing high perceptual scores. However, we could still often find unpleasant artifacts and problematic textures the reconstructions of ESRGAN. Such cases are exemplified in Figure 4.
53
54
55
56
57
58
59
60
Traditional metrics for assessing image quality such as PSNR and SSIM (Structural Similarity Index Measure) fail coincide with human perception[4]. The PSNR score is calculated based the pixel-wise MSE, so methods that minimize pixel-wise differences tend to achieve high PSNR scores [9]. However, the PSNR- oorienedd solutions fail at generating geeuufrry details and often drive the reconstruction towards the average of possible solutions, producing overly smooth images[6]. The learned perceptual image patch similarity (LPIPS) score[4] was proposed to measure the perceptual quality on various computer vision tasks. According to [2], the LPIPS score reliably coincides with human perception for assessing super-resolved images. We use the LPIPS score as an indicator of perceptual image quality in our experiments.
61
62
63
64
65
66
67
68
CycleGAN(8] is pipeline for image-to-image translation with unpaired images using generative adversarial nets and cycle loss. CycleGAN consists of 2 generators G1, G2, and 2 discriminators D1, D2. where Gj and G2 each translate the input image in cycling manner. The generators trained to minimize the adversarial loss and cycle loss (G)(G)(z)) xlli between the input image
70
71
72
73 and cycled image. We were able to design loss based on the cycle loss to reliably measure the
74
content consistency without such complicated design.
3 Method
78
We design a one-to-many approach for perceptual super-resolution by modifying the generator and the training objective. We also describe additional modifications to the training process and discriminator to improve the perceptual quality of SR.
77
78
79
3.1 Cycle consistency loss
Most works on perceptual super-resolutionfL. 6, 7] combine the content loss, adversarial loss (GAN 81 loss), and perceptual loss for the training objective as in Equation 1. Although the strict content
80
82 loss and adversarial loss are fundamentally disagreeing objectives, relying exclusively on either loss
83 each has significant issues. The strict content loss guides the network output to be exactly consistent
84 with the HR image, guiding the network to learn the mean of possible reconstructions and thus tends to give smorly-d results. Although the GAN framework is powerful method for

---

Qule pcde-onaittnc loo
Down-sungie
LR
SR
IIK
Upsasppl
Refarence
Image
An overview of our method. The cycle consistency loss is measured by comparing the LR image with the downsampled SR image. The discriminator is provided with the target image and reference image generated by bicubie-upsampling the LR image.
Figure I:
86 reallssc image generation, adversarial learning is highly unstable, and while the adversarial
87 loss and perceptual loss guide the network be perceptually convincing, they don't enforce the content of the super-resolved image to be consistent with the low-resolution image.
88
LTotal Lieroce + ALGAN + +TL1
(I)
We regard simply trading off these disagreeing losses incomplete objective for super-resolution since the mixing of such losses will obstruct the optimization of either loss. An improved training 91 objective must be GAN-oriented while ensuring consistent content of the image. That is, there needs
90
92
a content loss that doesn't hamper the generation images with high fetb-ffrqunney details.
93 We propose soft content loss inspired by the cycle loss of CycleGAN[8] to ensure the output of the generator be consistent with the low-resolution image while disturbing the generation of high-frequency information.
94
95
We view the super-resolution problem image space and apply the CycleGAN framework. To simplify the problem, we exploit our prior knowledge
Image-to-image translation task between the LR and HR
87
98
G2 HR- > LR. We denote the downsampling operation as f and G2 to be f instead of learning it. Consequently, our pipeline doesn't 't require learning D2 which is tool for learning G2. This leaves only Gı and D₃ be learned. We can write the cycle consistency loss as Equation 2. This loss won't It penalize generating high-free high-frequency details in any way while the SR image remains consistent with the LR image. Finally, can conclude our generator loss as Equation
89
100
102
103
3.
Lege (IR() LR[]]
(2)
Lo (G1) Loye + ALGAN(GI, + n percep
(3)
104
3.2
Providing scaled Gaussian noise to the generator
105
For the generator to be capable of generating more than one solution given single image, it must receive and apply random information. The variation between super-resolved images will mostly be stochastic variation in high- gigh-frequency textures. StyleGAN[3] achieves stochastic variation in images by adding pixel-wise Gaussian noise the output of each layer in the generator. We adopt this method and add the noise after every RRDB layer in the generator.
107
109
110
However, the sensitivity and the desired magnitude of noise would differ for each channel. Adding same noise directly after every layer could rather harm the ability the generator. For example, channel that detects edges would be seriously harmed by the noise. The sensitivity will also depend on
111
113
the depth of the network. To mitigate such possible issues, we allow each channel learn the desired
magnitude of the noise. Specifically, before adding the noise to the output of each layer, we multiply

---

0.0030
0.0025
0.0025
0.0020
0.0020
0.0015
0.0010
8
0.0010
0.0005
0.0005
0.0000
0.0000
layer
layer
(a) Noisc magnitude of method (c)
(b) Noise magnitude of method (I)
Figure 2: Boxplot of the scaling factors against the position of the layer in the network. The desired magnitude of noise increases in deeper layers, while the final layers have smaller scaling factors. The sensitivity to random noise varies for each layer and channel.
the noise with a channel-wise scaling factor. The scaling factor is learned concurrently with the network parameters. We observe that the desired magnitude differs along the network depending on the position of the layer. This shows that our method effectively implements one-to-many generator for super-resolution. The early layers seem to be focusing more on extracting the feature of the image, while the final layers preferred the noise be scaled before being applied to the reconstruction. Details are illustrated in Figure The noise is not applied at evaluation.
116
118
119
120
121
3.3
Reference image for the discriminator
Traditionally, the discriminator network receives single image and is trained to classify whether the given image is real or a generated image. This setting will provide the generator with gradients to any natural image" instead of towards the corresponding HR image. In extreme example, the traditional discriminator won't penalize the generator for generating completely different but equally realistic images from LR image. Although this is unlikely due to the existence of other content and perceptual losses, the gradient feedback given by the discriminator is sub-optimal for the task of super-resolution.
123
125
127
128
129
As solution, we provide the low-resolution image as a reference along with the target image the discriminator. This enables the discriminator to learn more important features for discriminating the generated image and provide better gradient feedback according to the LR image. For details, refer to Figure 1. We upsample the LR image the same size the HR image and concatenate them, feeding a tensor of shape (H, W, 6) to the discriminator. Despite its simplicity, conditioning the discriminator on the input is erucial modification for training such supervised problem with GAN -orieneet losses.
131
133
135
136
3.4
Blur detection
We recognized that there are often severely blurry regions in the images from the DIV2K[14) and DIV datasets. Although the authors of [15] argue that the data was collected by "paying special attention to image quality". there were many scenes with out-of-focus backgrounds. These blurry regions might plague the generator to learn to generate such blurry patches. Blurry backgrounds are might argue
138
140
often indistinguishable from finer objects based only the LR image. Though that the blurry backgrounds must also be learned, we were able to achieve finer detail and higher LPIPS score by detecting and removing blurry patches from both datasets.
142
144
We propose detect and remove blurry patches before the network trained on those patches. There are various methods for blur detection e.g. algorithmic methods and deep-learningbert based approaches[ 1, 12]. However, most deep-learning-ber maps of the image, which wouldn't be suited for our needs. Mostly, the algorithmic method of [10] was successful reliably detecting blurry patches as can be observed in Figure We measure the variance of the Laplacian activation of the patch and consider patches with variance of under 100 as
145
146
works focus on predicting pixel-wise blur
148

---

Figure 3: Randomly selected samples of the blur detection algorithm tested on image 0031 from the DIV8K dataset. The top Iwo rows are the patches classified as clear and the bottom rows are blurry patches. Regions that are clear in the image (person, pole) are correctly considered as clear patches by the detection algorithm.
150 blurry patches. The algorithm detects 28.8% blurry patches in a sample of 16,000 randomly cropped patches of size 96 from the DIV2K dataset and 48.9% of patches in a sample of 140,000 patches from the DIV8K dataset.
151
152
4 Experiments
154
We conduct experiments to evaluate the effectiveness of our proposed techniques in x4 and x 16 resolution and compare them with the baseline ESRGAN. We first experiment the effects of blur detection, then we perform an ablation study of our proposed training methods to evaluate their effectiveness. Implementation detail and training logs can be found on GitHut All our experiments were performed on a single Tesla T4 or Tesla K80 GPU on Google Colaboratory.
155
156
157
158
We observed that large portion of the training was used for loading nigh-resolutton images, despite most of the images not being used. As an implementation detail to improve training speed significantly, extract multiple patches and save them in a buffer while training instead of extracting only single patch after loading the image. We randomly pick images from the buffer for training and discard the selected patches from the buffer. In all of our experiments, extract 128 patches from each image and create a buffer of 1024 patches.
160
162
184
185
4.1
x4 4 super-resolution
186
4.1.1
Training details
167
We employ the ESRGAN network architecture with 23 RRDB blocks and most of its training configurations for the baseline of our experiments on x4 super-resolution. The training process is divided into two stages. We first pretrain the Roieeoriented models then train the ASRGAN-basee models.
168
189
170
The PSNR-oriented models are trained with the L1 loss with batch size of 16 for 500K iterations. We apply learning rate decay with an initial learning rate of 2 x 10-1 decayed by a factor of every 200k iterations. We initialize the GAN-based model with the PSNR-oriented model. We initialize the learning rate with x 10-4 for both G and D,, decaying the learning rate by factor use the Adam optimizer for both pretrained networks and GAN-based models, with 3, = 0.9 and 82 =2=00 0.99. The learning rate decay schedule corresponds the one proposed by ESRGAN. We implement our models and methods with the Tensorflow framework. The loss function is scaled with 7) 10 and À = 5 x 10-3, which is equivalent the training configuration of ESRGAN used in the PIRM-SR challenge. This is slightly different from the configuration used in the released model trained with 77 = 10
172
174
175
of 2 at [50k, 100k, 200k, 300k] iterations. For optimization,
176
177
178
179
180
All of our networks trained exclusively on the DIV2K dataset[ 14], while the original ESRGAN was trained with DIV2K, Flickr2K, and OST datasets combined. We obtained the LR images by
182
5

---

Table 1: LPIPS, PSNR, SSIM scores of various configurations for x4.
Methods
Set5
Set14
BSD100
100
(LPIPS SPPIS/PSNNR SSIM) 11,3173033003//
Pretrained (a)
0.22237 22.2377667688777 70076688877755
0.2705/27.22647
21176/24.877700777
+Blur detection (b)
0.1327 30.4582 0.7525
0.2229 / 26684488 007544
0.2684/27.255 /.77473
0.174//250881/ 0.7822
ESRGAN (Official)
0.0597 28.56/22.4362////
0.1129/23.4722 /0.6276
0.1285/23.3653
0.1025/22.712//00
ESRGAN (c)
0.05387 27.9387 7,79968
0.11177 24.52777452477777 0.6602
0.1256/24.6544//
0.1026/23.2835
+aeGGAN (d)
0.0536 / 27,9871 0,8014
0.1157 / 24.4505 / /0.6611
0.1275 / 24.5896 0.6470
0.1027 / 23.0449 0.7100
+Add noise (e)
0,04998/ / 28.23 0.8081
0.1104/ 24.48 / 0.66626
0,1209 24 8439 0.6577
0.1007 / 23,2204 0,72203
+Cycle loss (f)
0.0524/ 1322 0.8033
0.1082
24.5802 /066634
0.1264 24.6180 0.6468
0.1015 23,1333 /0.70000 0.7100
Perceprual loss (g)
0.2690 23.4608 0.6311
0.2727 22:27033 /0.5685
0.29K5 / 24.1644/ /0.589
0.2411 20../12.00.9..00 0066244
183
downsampling the HR images with MATLAB bicubic interpolation. We compare the effects of our
184
methods on LPIPS, PSNR, and SSIM scores on the Set5, Set14, BSD100, and Urban datasets. Scores evaluated on the Set5 and 4 datasets are obtained by averaging the final checkpoints, each recorded at [480k, 485k, 490K, 495k, 500k] iterations.
185
186
187
4.1.2
Ablation study
188
To study the effects of our proposed methods, we perform an ablation study of our proposed method. We enable our proposed methods one by one and list the resulting scores in Table 1. Each training configuration was fully trained with the original training configurations. We provide the saved model and configuration files to reproduce our results in our project repository. We also list the results of the official ESRGAN for fair comparison. The improvements from the official results and the result from configuration(e) is because the 7) value is different from the official model. First, blur detection is experimented with configuration(h) and improves the L.IPS score for all benchmarks. We train our baseline ESRGAN in configuration(e) and get reasonable results. By applying the technique of Section 3.3 in configuration d), we slightly harm the network in terms of the LPIPS score. However, providing conditional information to the discriminator is crucial for learning such supervised problem with adversarial learning. Our method of directly concatenating the reference image in the input is not optimal. The low-resolution image could be applied through SPADE[20] or alternative spatial transformation methods for improvements. Applying scaled noise shows large improvements as experimented in configuration(e).
189
190
192
194
196
198
200
201
202
The cycle consistency loss applied in configuration(f) shows neutral and slightly negative effects on the LPIPS score. The reason for this is mostly because of the incompetent GAN framework lacking the training techniques of modern GAN literature. Our statement is stated by the failure of configuration(g) where the GAN framework alone is responsible for learning the super-resolution process. The GAN framework of ESRGAN is incapable of lead the training process and thus the image quality wasn't 't improved when gave more responsibility to the adversarial loss in configuration(f). However, coupled with improved GAN techniques in further research, the cycle consistency content loss will further enhance the image quality.
204
206
208
4.2
x 16 super-resolution
211
4.2.1
Training details
212
We employ the RFB- -TB-ESRRAN of [21] as the baseline for our experiments on x x16 super-resolution. The RFB-ESRGAN proposes architecture using Receptive Field Blocks(RFB) and Residual of Receptive Field Dense Block(RREDB), each as an alternative for convolution and RRDB blocks. The RFB-ESRGAN uses less memory compared to methods that manipulate the image in the intermediate x4 resolution[22] and this allowed larger batch size in our environment. We employ the RFB- ESRGAN network architecture with 16 RRDB blocks and 8 RRFDB blocks for the baseline of our experiments on x16 super-resolution.
214
216
218
219
The model is first trained with the L1 loss for 100K iterations with an initial learning rate 2 x 10-4, decayed by a factor of two every 2.5 x 10S iteration. The GAN-based model is initialized with the pretrained model and is trained for 200K iterations, which is shorter than the original 400K iterations. Additionally, the batch size is decreased from 16 to and we therefore approximately scale the initial learning rate of 1 to 2 x 1 by factor of 5, The learning rate is decayed at [50k, 100k]
220
222

---

Table 2: LPIPS, PSNR scores for various configurations for x16 super-resolution.
Methods
DIV8K validation
Pretrained (a)
0.4664 30.3603
+Blur detection (b)
0.4603 / 25.53
RFB-ESRGAN(officaa)
0.345724.00
Baseline REB-ESRGAN (c)
0.356 / 24.78
Ours w/o cycle-loss (d)
0.321 / 23.95
Ours w/ cycle-loss (c)
0.323 / 23.49
224
iterations. We don't use model ensemble to further stabilize the network. All other models and equal. We train the network the DIVSK dataset[15], while the original network was trained with additional datasets including DIV2K, Flicker2K, OST dataset. The first 1,400 images of DIVSK used training data and the rest 100 validation images used for evaluation.
hyperparameter configurations
226
228
4.2.2
Ablation study
The -SNR-oriented method is improved using blur detection in configuration(b). Our GAN-baed model of configuration(c) achieves worse performance compared the the results reported in [21] because of the lighter training configurations. We were able make significant improvements in
231
233
the LPIPS score from the baseline RFB-ESRGAN using our proposed methods in configuration(d))
234
We apply all of our proposed methods except the cycle consistency loss in configuration(d), We also train the model with cycle consistency loss and get similar results in configuration(e). We were able
235
236
make such improvements using much lighter training configurations with only half iteration steps,
237
x4 smaller batch size, and without model ensemble. The results are described in Table 2.
Conclusion
239
proposed a one-to-many approach for super-resolution and achieve improved perceptual quality and better LPIPS score from the baseline ESRGAN configuration and achieve the state-of-art LPIPS score x16 perceptual super-resolution. We provide scaled pixel-wise to the generator to allow stochastic variation in the reconstructed image and implement generator capable of one-to-many pipeline. We also address the limitations mixing the strict content loss with perceptual losses and propose an alternative based the cycle loss. Our newly modified loss will ensure the consistency of the content while not penalizing high-frequency detail. Additionally, we further propose more techniques such as blur detection using Laplacian activation and redesign the discriminator input by providing a reference image to further improve the perceptual quality of x and x 16 super-resolution. However, the GAN framework from ESRGAN was incompetent to guide the training on its Modern GAN training techniques could be applied to further improve the GAN framework used in super-resolution. Our proposed loss function will become more effective content loss when coupled with robust GAN framework since it will reduce constraints generating high-frequency detail. Such improvements are left for future work.
241
243
245
247
249
250
251
252

---

11
ISSGAN
C
G.T
ESROAN
Chuntd
GI
ESROAN CHos
GT
ESRGAN
GT
ESRGAN
Chest8
G1
ESRGAN
Our method(f)
Ground Truth
Figure 4: Qualitative comparison of our methods with the official ESRGAN. We compare the poorly reconstructed outputs ESRGAN from BSD 100 and Urban 100 datasets with our proposed model trained with configuration(f). Our method produces sharp textures and more realistic structures compared to the baseline ESRGAN, although it also fails to accurately reconstruct human faces.

---

253 References
254
[1] Wang, Xintao, et al. "Esrgan: Enhanced super-resolution generative adversarial networks." Proceedings of the European Conference on Computer Vision (ECCV) Workshops. 2018.
256
[2] Zhang, Kai, Shuhang Gu, and Radu Timofte. "Ntire 2020 challenge perceptual extreme super-resolution: Methods and results." " Proceedings of the TEFE/CVF Conference Computer Vision and Pattern Recognition Workshops. 2020.
258
259
[3] Karras, Tero, Samuli Laine, and Timo Aila. "A style-based generator architecture for generative adversarial networks." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.
280
261
[4] Zhang, Richard, et al. "The unreasonable effietivvnness of deep features as perceptual metric." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018
262
263
[S] Jo. Younghyun, Sejong Yang, and Seon Joo Kim. "Investigating loss functions for extreme super-resolution. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops. 2020.
[6] Ledig, Christian, et al. "Photo-realistic single image super-resolution using generative adversarial network." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.
266
267
[7] Sajjadi, Mchdi SM, Bernhard Scholkopf, and Michael Hirsch. "Enhancenet: Single image super-resolution through automated texture synthesis." " Proceedings of the IEEE International Conference on Computer Vision. 2017.
268
269
270
[8] Zhu, Yan, et al. "Unpaired image-to-image translation using cycle-consistent adversarial networks." Proceedings of the IECE international conference on computer vision. 2017.
271
272
[9] Dong, Chao, al. "Image super-resolution using deep convolutional networks." IEBE transactions pattern analysis and machine intelligence 38.2 (2015): 295-307.
[10] Bansal, Raghav, Gaurav Raj. and Tanupriya Choudhury. "Blur image detection using Laplacian operator and Open-CV." 2016 International Conference System Modeling & Advancement in Research Trends (SMART). IEEE, 2016.
275
276
277
[1]] Tang, Chang, et al. "Defusionnet: Defocus blur detection via recurrently fusing and refining multi-scale deep features." Proceedings of the IERE/CVF Conference on Computer Vision and Pattern Recognition. 2019.
278
279 [12] Wang, Xuewei, et al. "Accurate and fast blur detection using a pyramid M-Shaped deep neural network." IEEE Access 7 (2019): 86611-8666 1-6624..
[13] Yuan, Yuan, al. "UUnuppervised image super-resolution using cycle-in-cycle generative adversarial networks." " Proceedings of the IEEE Conference Computer Vision and Pattern Recognition Workshops. 2018.
[14] Agustsson, Eirikur, and Radu Timofte. "Ntire 2017 challenge single image super-resolution: Dataset and study." Proceedings the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2017.
284
285
[15] Gu, Shuhang. et al. "Div8k: Diverse 8k resolution image dataset." 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW). IEEE, 2019.
286
287
[16] Jolicocur-Martincau, Alcxia. "The relativistic discriminator: a key element missing from standard GAN." Xiv preprint arXiv: 000088070000000 (2018).
288
289
[17] Kim, Jiwon, Jung Kwon Lec, and Kyoung Mu Lec. "Accurate image super-resolution using very deep convolutional networks." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.
[18] Zhang, Yulun, et al. "Image super-resolution using very deep residual channel attention networks." Proceedings of the European conference computer vision (ECCV). 2018.
292
[19] Lim, Bee, al. "Enhanced deep residual networks for single image super-resolution." " Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 2017.
294
295
[20] Park, Taesung, et al. "Semantic image synthesis with spatially-adaptive normalization." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.
296
297
[21] Shang. Taizhang, et al. "Perceptual extreme super-resolution network with receptive field block." Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 2020.
298
299
[22] Jo, Younghyun, Sejong Yang. and Seon Joo Kim. "Investigating loss functions for extreme super-resolution." Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops. 2020.

---

Checklist
302
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No] . [N/A] You strongly encouraged to include justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
304
305
306
Did you include the license to the code and datasets? [Yes] See Section
2
307
Did you include the license to the code and datasets? [No] The code and data are proprietary.
308
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
310
311
312
1. For all authors...
313
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contribu- tions and scope? [Yes] The focus of our paper is on the proposal of a one-to-mane pipeline for super-resolution.
314
316
(b) Did you describe the limitations of your work? [Yes] The GAN framework used in our work weak despite the successes of modern GANs.
(c) Did you discuss any potential negative societal impacts of your work? [No]
319
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
320
2. If you are including theoretical results...
321
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
322
(b) Did you include complete proofs of all theoretical results? [N/A]
323
3. If you ran experiments...
324
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as URLY? [Yes] As mentioned in the paper, all
the code and training configurations are available at htpps//gttbb.comkkreeettliii
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)?
328
[Yes]
(c) Did you report error bars (e.g.. with respect to the random seed after running experiments multiple times)? [No] There was hardware limitations for repeating our work multiple times, however the ablation study sufficiently described improvements of our methods.
330
331
332
(d) Did you include the total amount of compute and the type resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] All our experiments were executed on Google COLAB with a single Tesla T4 or Tesla K80 GPU.
333
334
335
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
336
(a) If your work uses existing assets, did you cite the creators? [Yes] The DIV2K and DIVSK datasct used and cited.
(b) Did you mention the license of the assets? [N/A]
339
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
340
(d) Did you discuss whether and how consent was obtained from people whose data you're 'r us-
ing/curating? [Yes] The DIV2K dataset was used for the baseline ESRGAN, which was compared with methods and the DIVSK dataset was used for the -EB-ESSRAN.
(c) Did you discuss whether the data you using/curating contains personally identifiable informa- tion or offensive content? [No]
345
5. If you used crowdsourcing or conducted research with human subjects...
346
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
347
[N/A]
348
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
349
350
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
351
10


Reviews:
[{"review_id": "wnWOcVgFVj_", "reviewer": "Reviewer_PHmW", "summary": "This paper proposes to address the one-to-many problem of SR by adding random noise to each layer of generator and considering a new perceptual loss to allow high-frequency mismatch. The new network performs better than ESRGAN by further selecting non-blur area for training. Ablation studies verify the effectivness of new network design.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- Noise generator to address one-to-many problem.\n- New content loss not to penalize high-frequency gradients.\n- Using laplacian to select non-blur area for training.\n- Better subjective and objective quality than ESRGAN\n\nCons:\n- Limited comparisons to state-of-the-arts, such as SRFlow and its improvments.\n- Blured images compared with ESRGAN.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper may consider adding the proposed techniques to other GAN-based methods to verify the effectiveness of various tricks. The new content cycle loss is interesting but up-sampling and down-sampling may induce some information loss in generation and discriminator. More comparisons to state-of-the-arts GAN-based methods is needed for drawing a persuasive conlusion for solving the one-to-may SR problem, since automatically selecting the most human-favarable estimation (without artifacts) from many possible estimations is still a hot research area and a non-trivial problem. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "nv9n_Odqtm-", "reviewer": "Reviewer_rx3L", "summary": "This paper develops new loss functions, network architectures, and training methods for solving the single image super resolution problem using convolutional neural networks. The aim of this effort is to develop a network that is capable of producing the many plausible HR images that could have produced a LR image. With regards to loss functions, the authors propose penalizing the difference between a downsampled HR reconstruction and the input LR image (rather than a GT HR image) and also conditioning a discriminator network on the low resolution image. With regards to network architectures, the authors pump noise into the intermediate layers of the network to generate diverse outputs. With regards to training methods, the authors remove smooth patches from the DIV2K dataset for training. ", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n# Originality\nOther recent works have noted that there should be multiple solutions to the SISR problem and developed their own solutions [A]. These methods are not mentioned nor compared against.\n\nThe others innovations in this work don't seem particularly novel.\n\n[A] Lugmayr, Andreas, et al. \"Srflow: Learning the super-resolution space with normalizing flow.\" European Conference on Computer Vision. Springer, Cham, 2020.\n\n\n# Quality\nThe paper provides few high quality images and those that are included aren't presented clearly: Is Figure 4 demonstrating 4x or 16x superresoltuion results? I think 16x because both methods have major artifacts.\n\nThe comparisons are each limited to a single method that is no longer SOTA. See [A].\n\nMuch of the paper is motivated by the idea the one LR image should produce multiple plausible HR images. However, the main text provides no examples of this occuring.\n\nUser studies are preferable to relying on LPIPS alone.\n\n# Clarity\nThe authors motivate the L1 penalty in the LR domain as part of a cycle consistency loss. Howevever, cycle consistency seems fundamentally at odds with the goals of the paper; generating a diverse set of HR outputs form a LR input. A true cycle consistent loss would ensure that the LR image formed by a single HR image would always produce that same HR image.\n\n# Significance\nThe paper set out to develop a one-to-many approach to super-resolution, which is arguably an important problem. Unfortunately, there's little evidence this was accomplished.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "At 16x, super-resolution is essentially hallucinating plausible scenes. It's worth adding a sentence that even if a super-res algorithm could produce pleasing reconstructions, they don't necessarily represent the truth.\n", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "MzWxTN8c7F", "reviewer": "Reviewer_44pW", "summary": "This paper focuses on addressing the ill-posed problem in super-resolution task. To this end, the authors propose a new data augmentation with Laplacian activation, soft content loss and adding noise into the extracted features. The experimental results show the proposed method can superior the baseline ESRGAN on Set5, Set14, BSD100 and Urban100. ", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\t\n1. The idea of the paper is clearly written and empirical results demonstrate its effectiveness.\n2. The paper presented many interesting ideas, e.g., soft content loss, new data augmentation with blur detection, one-to-many generator, and conduct extensive ablation study for each of the terms.\n3. There are additional results in Table 2 of the paper that validate the key idea for x16 SR.\n\nWeaknesses\n1. The authors cannot claim this paper as the first to consider SR as a one-to-many problem. \nActually many previous papers have already model SR in a one-to-many way. For example, [1] gave a solid theoretical explaination of SR problems and give an effective solution. And GAN based SR methods, use discriminator loss for the same reason.\n\n[1] Amortised MAP Inference for Image Super-resolution\n\n2. The motivation of adding noise to intermediate layers seems lacking theoretical support. \nIndeed, adding noise can generate various outputs. However, candidate multiple HR images are not just random. They lie in some `null` space  of the downsample operator. And adding random noise cannot ensure to generate proper HR results.\n\n3. The major weakness is the novelty of the proposed method, although the ablation studies validate the effectives of each technique on several datasets. The novelty is limited, and there are few new techniques for me. In my view, the proposed method seems to stack many tricks for improving the performances, such as a new data augmentation and adding the pixel-wise noise into the extracted features.\n\n4. Table 1, The comparison to STDA methods do not include the recent state of the art methods like \" Enhancing Perceptual Loss with Adversarial Feature\n\n5. Matching for Super-Resolution\", IJCNN’20; “Journey Towards Tiny Perceptual\n\n6. Super-Resolution”, ECCV’20; “Generative adversarial network-based image super-resolution using perceptual content losses”, ECCV’18 (and many others).\n\n7. The manuscript is filled with informal words and needs a rewrite.\n28: don’t -> do not\n98: “->” -> “\\rightarrow”\n125: won’t -> will not\n224: don’t -> do not\n241: “x16 “ -> “\\times 16”\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. Important references are mssing, such as:\n[1] Amortised MAP Inference for Image Super-resolution\nAnd the author cannot claim to be the first to consider SR as a one-to-many problem.  \n2. The author should compare more recent state-of-the-art methods in Table 1.\n3. Although the adding scaled Gaussian noise to the generator appears to be novel and brings some marginal improvements on performances, its motivation is weird to me. The submission says “the sensitivity and the desired magnitude of noise would differ for each channel.” However, it is not intuitive. Do you have any further justification of this assumption?\n4. Overall, I feel the three contributions are not well justified/evaluated", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "1"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a one-to-many approach to super-resolution. All reviewers and the AC agree that this is an important problem and a very interesting research direction. 

The original manuscript missed several important prior work (as listed by Reviewers rx3L and 44pW), which also treat SR as a one-to-many problem. This significantly reduced the novelty aspect of the work. In their rebuttal the authors acknowledged these similarities and proposed to change the objective of work to "adapting the concepts of SRFlow to improve GAN-based SR by properly implementing the one-to-many property". The AC agrees with the authors that the paper does improve performance and make interesting contributions (see below). However, incorporating these changes, adding new baselines and reformulating the work would certainly require another round of revisions.

The paper makes several practical contributions (soft content loss, pixel-wise noise after each RRDB block, filter blurry regions using Laplacian activation) that seem to work well in practice. This was highlighted by Reviewer 44pW.

Reviewer rx3L points out that the paper does not clearly show the ability of the model of generating multiple HR images from a given LR image. The AC agrees that this would strengthen the work significantly.

The AC agrees with the concerns raised by the reviewers and finds no basis for overruling their recommendation. The AC appreciates the constructive rebuttal and encourages the authors to incorporate the reviewers feedback and resubmit to a future conference.

Decision: Reject