One-to-many Approach for Improving Super-Resolution

Super-resolution ( SR ) is a one-to-many task with multiple possible solutions.1 However , previous works were not concerned about this characteristic . For a one-2 to-many pipeline , the generator should be able to generate multiple estimates of3 the reconstruction , and not be penalized for generating similar and equally realistic4 images . To achieve this , we propose adding weighted pixel-wise noise after every5 Residual-in-Residual Dense Block ( RRDB ) to enable the generator to generate6 various images . We modify the strict content loss to not penalize the stochastic7 variation in reconstructed images as long as it has consistent content . Additionally,8 we observe that there are out-of-focus regions in the DIV2K , DIV8K datasets9 that provide unhelpful guidelines . We filter blurry regions in the training data10 using the method of [ 10 ] . Finally , we modify the discriminator to receive the low-11 resolution image as a reference image along with the target image to provide better12 feedback to the generator . Using our proposed methods , we were able to improve13 the performance of ESRGAN in ×4 perceptual SR and achieve the state-of-the-art14 LPIPS score in ×16 perceptual extreme SR.15 1 Introduction16 Super-resolution is the task of recovering a high-resolution ( HR ) image from a low-resolution ( LR ) 17 image . Recent works have achieved significant performance in SR using deep convolutional neural18 network ( CNN ) based approaches . Some of them exploit strict content loss as the training objective19 for super-resolution and propose various network architectures to improve the PSNR score . However,20 these methods often result in overly smooth images and have poor perceptual quality [ 6 ] . Another21 branch of works focuses on improving perceptual quality with perceptual training methods [ 1,6,7 ] .22 These methods employ generative adversarial networks ( GAN ) and perceptual loss functions to drive23 the network ’ s output towards the natural image manifold of possible HR images . We assess an1d24 further improve the perceptual quality of these works.25 Because super-resolution is a one-to-many problem with multiple possible reconstructions for one26 image , methods based on strict content loss often lead to predicting the average of possible reconstruc-27 tions [ 6 ] . Perceptual-driven solutions utilize perceptual and adversarial loss , which both don ’ t penalize28 the generator for generating equally realistic images with stochastic variance . However , we discover29 two incomplete aspects in the current perceptual SR pipeline . First , although the above-mentioned30 losses don ’ t penalize stochastic variation , the final loss is mixed with the strict content loss which31 strictly penalizes these variations . Second , the generator doesn ’ t have the ability to generate multiple32 estimates of the image despite a one-to-many problem . To implement such a one-to-many pipeline,33 we provide the generator with pixel-wise noise and improve the content loss so it doesn ’ t restrict the34 variation in the image while ensuring the consistency of the content.35 The key contributions of our work can be described as follows:36 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . • We propose a weaker content loss that does not penalize generating high-frequency detail37 and stochastic variation in the image.38 • We enable the generator to generate diverse outputs by adding scaled pixel-wise noise after39 each RRDB block.40 • We filter blurry regions in the training data using Laplacian activation [ 10 ] .41 • We additionally provide the LR image to the discriminator to give better gradient feedback42 to the generator.43 2 Related work44 Since the pioneering work of SRCNN [ 9 ] , many works have exploited the pixel-wise loss and PSNR-45 oriented training objectives to learn the end-to-end mapping from LR to HR images . We denote46 such pixel-wise losses as the strict content loss . Many network architectures and techniques were47 experimented with to improve the complexity of such networks . Deeper network architectures [ 17 ] ,48 residual networks [ 6 ] , channel attention [ 18 ] , and techniques to remove batch normalization [ 19 ] were49 introduced . Although these works achieved state-of-the-art SR performance in the peak signal-to-50 noise ratio ( PSNR ) metric , they often produce overly smooth images.51 To improve the perceptual image quality of SR , SRGAN [ 6 ] proposes perceptual loss and GAN-based52 training . The perceptual loss is measured using intermediate activations of the VGG-19 network and53 a discriminator is used for the adversarial training process . Enhanced SRGAN ( ESRGAN ) further54 improves SRGAN by modifying the generator architecture with Residual in Residual Dense Block55 ( RRDB ) , the Relativistic GAN [ 16 ] loss , and improving the perceptual loss . Such methods were56 superior to PSNR-oriented methods at generating photo-realistic SR images with sharp details , achiev-57 ing high perceptual scores . However , we could still often find unpleasant artifacts and problematic58 textures in the reconstructions of ESRGAN . Such cases are exemplified in Figure 4.59 Traditional metrics for assessing image quality such as PSNR and SSIM ( Structural Similarity Index60 Measure ) fail to coincide with human perception [ 4 ] . The PSNR score is calculated based on the61 pixel-wise MSE , so methods that minimize pixel-wise differences tend to achieve high PSNR scores62 [ 9 ] . However , the PSNR-oriented solutions fail at generating high-frequency details and often drive63 the reconstruction towards the average of possible solutions , producing overly smooth images [ 6 ] . The64 learned perceptual image patch similarity ( LPIPS ) score [ 4 ] was proposed to measure the perceptual65 quality on various computer vision tasks . According to [ 2 ] , the LPIPS score reliably coincides with66 human perception for assessing super-resolved images . We use the LPIPS score as an indicator of67 perceptual image quality in our experiments.68 CycleGAN [ 8 ] is a pipeline for image-to-image translation with unpaired images using generative69 adversarial nets and cycle loss . CycleGAN consists of 2 generators G1 , G2 , and 2 discriminators70 D1 , D2 , where G1 and G2 each translate the input image in a cycling manner . The generators are71 trained to minimize the adversarial loss and cycle loss ||G2 ( G1 ( x ) ) − x||1 between the input image72 and cycled image . We were able to design a loss based on the cycle loss to reliably measure the73 content consistency without such a complicated design.74 3 Method75 We design a one-to-many approach for perceptual super-resolution by modifying the generator and the76 training objective . We also describe additional modifications to the training process and discriminator77 to improve the perceptual quality of SR.78 3.1 Cycle consistency loss79 Most works on perceptual super-resolution [ 1 , 6 , 7 ] combine the content loss , adversarial loss ( GAN80 loss ) , and perceptual loss for the training objective as in Equation 1 . Although the strict content81 loss and adversarial loss are fundamentally disagreeing objectives , relying exclusively on either loss82 each has significant issues . The strict content loss guides the network output to be exactly consistent83 with the HR image , guiding the network to learn the mean of possible reconstructions and thus84 tends to give overly-smoothed results . Although the GAN framework is a powerful method for85 photo-realistic image generation , adversarial learning is highly unstable , and while the adversarial86 loss and perceptual loss guide the network to be perceptually convincing , they don ’ t enforce the87 content of the super-resolved image to be consistent with the low-resolution image.88 LTotal = Lpercep + λLGAN + ηL1 ( 1 ) We regard simply trading off these disagreeing losses as an incomplete objective for super-resolution89 since the mixing of such losses will obstruct the optimization of either loss . An improved training90 objective must be GAN-oriented while ensuring consistent content of the image . That is , there needs91 a content loss that doesn ’ t hamper the generation of images with high-frequency details.92 We propose a soft content loss inspired by the cycle loss of CycleGAN [ 8 ] to ensure the output of93 the generator to be consistent with the low-resolution image while not disturbing the generation of94 high-frequency information.95 We view the super-resolution problem as an image-to-image translation task between the LR and HR96 image space and apply the CycleGAN framework . To simplify the problem , we exploit our prior97 knowledge on G2 : HR− > LR . We can denote the downsampling operation as f and set G2 to98 be f instead of learning it . Consequently , our pipeline doesn ’ t require learning D2 which is a tool99 for learning G2 . This leaves only G1 and D1 to be learned . We can write the cycle consistency loss100 as Equation 2 . This loss won ’ t penalize generating high-frequency details in any way while the SR101 image remains consistent with the LR image . Finally , we can conclude our generator loss as Equation102 3.103 Lcyc ( G1 ) = ||f ( G1 ( LR ) ) − LR||1 ( 2 ) LTotal ( G1 ) = Lcyc ( G1 ) + λLGAN ( G1 , D1 ) + ηLpercep ( 3 ) 3.2 Providing scaled Gaussian noise to the generator104 For the generator to be capable of generating more than one solution given a single image , it must105 receive and apply random information . The variation between super-resolved images will mostly106 be stochastic variation in high-frequency textures . StyleGAN [ 3 ] achieves stochastic variation in107 images by adding pixel-wise Gaussian noise to the output of each layer in the generator . We adopt108 this method and add the noise after every RRDB layer in the generator.109 However , the sensitivity and the desired magnitude of noise would differ for each channel . Adding110 the same noise directly after every layer could rather harm the ability of the generator . For example , a111 channel that detects edges would be seriously harmed by the noise . The sensitivity will also depend on112 the depth of the network . To mitigate such possible issues , we allow each channel to learn the desired113 magnitude of the noise . Specifically , before adding the noise to the output of each layer , we multiply114 the noise with a channel-wise scaling factor . The scaling factor is learned concurrently with the115 network parameters . We observe that the desired magnitude differs along the network depending on116 the position of the layer . This shows that our method effectively implements a one-to-many generator117 for super-resolution . The early layers seem to be focusing more on extracting the feature of the image,118 while the final layers preferred the noise to be scaled before being applied to the reconstruction.119 Details are illustrated in Figure 2 . The noise is not applied at evaluation.120 3.3 Reference image for the discriminator121 Traditionally , the discriminator network receives a single image and is trained to classify whether the122 given image is real or a generated image . This setting will provide the generator with gradients to123 `` any natural image '' instead of towards the corresponding HR image . In an extreme example , the124 traditional discriminator won ’ t penalize the generator for generating completely different but equally125 realistic images from an LR image . Although this is unlikely due to the existence of other content126 and perceptual losses , the gradient feedback given by the discriminator is sub-optimal for the task of127 super-resolution.128 As a solution , we provide the low-resolution image as a reference along with the target image to the129 discriminator . This enables the discriminator to learn more important features for discriminating130 the generated image and provide better gradient feedback according to the LR image . For details,131 refer to Figure 1 . We upsample the LR image to the same size as the HR image and concatenate132 them , feeding a tensor of shape ( H , W , 6 ) to the discriminator . Despite its simplicity , conditioning133 the discriminator on the input is a crucial modification for training such a supervised problem with134 GAN-oriented losses.135 3.4 Blur detection136 We recognized that there are often severely blurry regions in the images from the DIV2K [ 14 ] and137 DIV8K [ 15 ] datasets . Although the authors of [ 15 ] argue that the data was collected by `` paying special138 attention to image quality '' , there were many scenes with out-of-focus backgrounds . These blurry139 regions might plague the generator to learn to generate such blurry patches . Blurry backgrounds are140 often indistinguishable from finer objects based only on the LR image . Though some might argue141 that the blurry backgrounds must also be learned , we were able to achieve finer detail and higher142 LPIPS score by detecting and removing blurry patches from both datasets.143 We propose to detect and remove blurry patches before the network is trained on those patches.144 There are various methods for blur detection e.g . algorithmic methods and deep-learning-based145 approaches [ 11 , 12 ] . However , most deep-learning-based works focus on predicting pixel-wise blur146 maps of the image , which wouldn ’ t be suited for our needs . Mostly , the algorithmic method of [ 10 ] 147 was successful at reliably detecting blurry patches as can be observed in Figure 3 . We measure the148 variance of the Laplacian activation of the patch and consider patches with variance of under 100 as149 blurry patches . The algorithm detects 28.8 % blurry patches in a sample of 16,000 randomly cropped150 patches of size 96×96 from the DIV2K dataset and 48.9 % of patches in a sample of 140,000 patches151 from the DIV8K dataset.152 4 Experiments153 We conduct experiments to evaluate the effectiveness of our proposed techniques in ×4 and ×16154 resolution and compare them with the baseline ESRGAN . We first experiment the effects of blur155 detection , then we perform an ablation study of our proposed training methods to evaluate their156 effectiveness . Implementation detail and training logs can be found on GitHub1 . All our experiments157 were performed on a single Tesla T4 or Tesla K80 GPU on Google Colaboratory.158 We observed that a large portion of the training was used for loading high-resolution images , despite159 most of the images not being used . As an implementation detail to improve training speed significantly,160 we extract multiple patches and save them in a buffer while training instead of extracting only a single161 patch after loading the image . We randomly pick images from the buffer for training and discard the162 selected patches from the buffer . In all of our experiments , we extract 128 patches from each image163 and create a buffer of 1024 patches.164 4.1 ×4 super-resolution165 4.1.1 Training details166 We employ the ESRGAN network architecture with 23 RRDB blocks and most of its training167 configurations for the baseline of our experiments on ×4 super-resolution . The training process is168 divided into two stages . We first pretrain the PSNR-oriented models then train the ESRGAN-based169 models.170 The PSNR-oriented models are trained with the L1 loss with a batch size of 16 for 500K iterations.171 We apply learning rate decay with an initial learning rate of 2 × 10−4 , decayed by a factor of 2172 every 200k iterations . We initialize the GAN-based model with the PSNR-oriented model . We173 initialize the learning rate with 1× 10−4 for both G1 and D1 , decaying the learning rate by a factor174 of 2 at [ 50k , 100k , 200k , 300k ] iterations . For optimization , we use the Adam optimizer for both175 pretrained networks and GAN-based models , with β1 = 0.9 and β2 = 0.99 . The learning rate decay176 schedule corresponds to the one proposed by ESRGAN . We implement our models and methods with177 the Tensorflow framework . The loss function is scaled with η = 10 and λ = 5 × 10−3 , which is178 equivalent to the training configuration of ESRGAN used in the PIRM-SR challenge . This is slightly179 different from the configuration used in the released model trained with η = 10−2.180 All of our networks are trained exclusively on the DIV2K dataset [ 14 ] , while the original ESRGAN181 was trained with DIV2K , Flickr2K , and OST datasets combined . We obtained the LR images by182 1https : //github.com/krenerd/ultimate-sr downsampling the HR images with MATLAB bicubic interpolation . We compare the effects of our183 methods on LPIPS , PSNR , and SSIM scores on the Set5 , Set14 , BSD100 , and Urban100 datasets.184 Scores evaluated on the Set5 and Set14 datasets are obtained by averaging the final 5 checkpoints,185 each recorded at [ 480k , 485k , 490k , 495k , 500k ] iterations.186 4.1.2 Ablation study187 To study the effects of our proposed methods , we perform an ablation study of our proposed method.188 We enable our proposed methods one by one and list the resulting scores in Table 1 . Each training189 configuration was fully trained with the original training configurations . We provide the saved model190 and configuration files to reproduce our results in our project repository . We also list the results191 of the official ESRGAN for fair comparison . The improvements from the official results and the192 result from configuration ( c ) is because the η value is different from the official model . First , blur193 detection is experimented with in configuration ( b ) and improves the LPIPS score for all benchmarks.194 We train our baseline ESRGAN in configuration ( c ) and get reasonable results . By applying the195 technique of Section 3.3 in configuration ( d ) , we slightly harm the network in terms of the LPIPS196 score . However , providing conditional information to the discriminator is crucial for learning such a197 supervised problem with adversarial learning . Our method of directly concatenating the reference198 image in the input is not optimal . The low-resolution image could be applied through SPADE [ 20 ] 199 or alternative spatial transformation methods for improvements . Applying scaled noise shows large200 improvements as experimented in configuration ( e ) .201 The cycle consistency loss applied in configuration ( f ) shows neutral and slightly negative effects202 on the LPIPS score . The reason for this is mostly because of the incompetent GAN framework203 lacking the training techniques of modern GAN literature . Our statement is stated by the failure of204 configuration ( g ) where the GAN framework alone is responsible for learning the super-resolution205 process . The GAN framework of ESRGAN is incapable of lead the training process and thus206 the image quality wasn ’ t improved when we gave more responsibility to the adversarial loss in207 configuration ( f ) . However , coupled with improved GAN techniques in further research , the cycle208 consistency content loss will further enhance the image quality.209 4.2 ×16 super-resolution210 4.2.1 Training details211 We employ the RFB-ESRGAN of [ 21 ] as the baseline for our experiments on ×16 super-resolution.212 The RFB-ESRGAN proposes an architecture using Receptive Field Blocks ( RFB ) and Residual of213 Receptive Field Dense Block ( RRFDB ) , each as an alternative for convolution and RRDB blocks . The214 RFB-ESRGAN uses less memory compared to methods that manipulate the image in the intermediate215 ×4 resolution [ 22 ] and this allowed larger batch size in our environment . We employ the RFB-216 ESRGAN network architecture with 16 RRDB blocks and 8 RRFDB blocks for the baseline of our217 experiments on ×16 super-resolution.218 The model is first trained with the L1 loss for 100K iterations with an initial learning rate 2× 10−4,219 decayed by a factor of two every 2.5× 105 iteration . The GAN-based model is initialized with the220 pretrained model and is trained for 200K iterations , which is shorter than the original 400K iterations.221 Additionally , the batch size is decreased from 16 to 4 and we therefore approximately scale the222 initial learning rate of 10−4 to 2× 10−5 by a factor of 5 . The learning rate is decayed at [ 50k , 100k ] 223 iterations . We don ’ t use model ensemble to further stabilize the network . All other models and224 hyperparameter configurations are equal . We train the network on the DIV8K dataset [ 15 ] , while the225 original network was trained with additional datasets including DIV2K , Flicker2K , OST dataset . The226 first 1,400 images of DIV8K are used as training data and the rest 100 validation images are used for227 evaluation.228 4.2.2 Ablation study229 The PSNR-oriented method is improved using blur detection in configuration ( b ) . Our GAN-baed230 model of configuration ( c ) achieves worse performance compared the the results reported in [ 21 ] 231 because of the lighter training configurations . We were able to make significant improvements in232 the LPIPS score from the baseline RFB-ESRGAN using our proposed methods in configuration ( d ) .233 We apply all of our proposed methods except the cycle consistency loss in configuration ( d ) . We also234 train the model with cycle consistency loss and get similar results in configuration ( e ) . We were able235 to make such improvements using much lighter training configurations with only half iteration steps,236 ×4 smaller batch size , and without model ensemble . The results are described in Table 2.237 5 Conclusion238 We proposed a one-to-many approach for super-resolution and achieve improved perceptual quality239 and better LPIPS score from the baseline ESRGAN configuration and achieve the state-of-art LPIPS240 score in x16 perceptual super-resolution . We provide scaled pixel-wise to the generator to allow241 stochastic variation in the reconstructed image and implement a generator capable of a one-to-many242 pipeline . We also address the limitations of mixing the strict content loss with perceptual losses and243 propose an alternative based on the cycle loss . Our newly modified loss will ensure the consistency244 of the content while not penalizing high-frequency detail . Additionally , we further propose more245 techniques such as blur detection using Laplacian activation and redesign the discriminator input by246 providing a reference image to further improve the perceptual quality of×4 and×16 super-resolution.247 However , the GAN framework from ESRGAN was incompetent to guide the training on its own.248 Modern GAN training techniques could be applied to further improve the GAN framework used249 in super-resolution . Our proposed loss function will become more effective as a content loss when250 coupled with a robust GAN framework since it will reduce constraints in generating high-frequency251 detail . Such improvements are left for future work.252 References253 [ 1 ] Wang , Xintao , et al . `` Esrgan : Enhanced super-resolution generative adversarial networks . '' Proceedings of254 the European Conference on Computer Vision ( ECCV ) Workshops . 2018.255 [ 2 ] Zhang , Kai , Shuhang Gu , and Radu Timofte . `` Ntire 2020 challenge on perceptual extreme super-resolution:256 Methods and results . '' Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition257 Workshops . 2020.258 [ 3 ] Karras , Tero , Samuli Laine , and Timo Aila . `` A style-based generator architecture for generative adversarial259 networks . '' Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2019.260 [ 4 ] Zhang , Richard , et al . `` The unreasonable effectiveness of deep features as a perceptual metric . '' Proceedings261 of the IEEE conference on computer vision and pattern recognition . 2018.262 [ 5 ] Jo , Younghyun , Sejong Yang , and Seon Joo Kim . `` Investigating loss functions for extreme super-resolution . `` 263 Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops . 2020.264 [ 6 ] Ledig , Christian , et al . `` Photo-realistic single image super-resolution using a generative adversarial network . `` 265 Proceedings of the IEEE conference on computer vision and pattern recognition . 2017.266 [ 7 ] Sajjadi , Mehdi SM , Bernhard Scholkopf , and Michael Hirsch . `` Enhancenet : Single image super-resolution267 through automated texture synthesis . '' Proceedings of the IEEE International Conference on Computer Vision.268 2017.269 [ 8 ] Zhu , Jun-Yan , et al . `` Unpaired image-to-image translation using cycle-consistent adversarial networks . `` 270 Proceedings of the IEEE international conference on computer vision . 2017.271 [ 9 ] Dong , Chao , et al . `` Image super-resolution using deep convolutional networks . '' IEEE transactions on pattern272 analysis and machine intelligence 38.2 ( 2015 ) : 295-307.273 [ 10 ] Bansal , Raghav , Gaurav Raj , and Tanupriya Choudhury . `` Blur image detection using Laplacian operator274 and Open-CV . '' 2016 International Conference System Modeling & Advancement in Research Trends ( SMART ) .275 IEEE , 2016.276 [ 11 ] Tang , Chang , et al . `` Defusionnet : Defocus blur detection via recurrently fusing and refining multi-scale277 deep features . '' Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2019.278 [ 12 ] Wang , Xuewei , et al . `` Accurate and fast blur detection using a pyramid M-Shaped deep neural network . `` 279 IEEE Access 7 ( 2019 ) : 86611-86624.280 [ 13 ] Yuan , Yuan , et al . `` Unsupervised image super-resolution using cycle-in-cycle generative adversarial281 networks . '' Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops . 2018.282 [ 14 ] Agustsson , Eirikur , and Radu Timofte . `` Ntire 2017 challenge on single image super-resolution : Dataset and283 study . '' Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops . 2017.284 [ 15 ] Gu , Shuhang , et al . `` Div8k : Diverse 8k resolution image dataset . '' 2019 IEEE/CVF International Conference285 on Computer Vision Workshop ( ICCVW ) . IEEE , 2019.286 [ 16 ] Jolicoeur-Martineau , Alexia . `` The relativistic discriminator : a key element missing from standard GAN . `` 287 arXiv preprint arXiv:1807.00734 ( 2018 ) .288 [ 17 ] Kim , Jiwon , Jung Kwon Lee , and Kyoung Mu Lee . `` Accurate image super-resolution using very deep289 convolutional networks . '' Proceedings of the IEEE conference on computer vision and pattern recognition . 2016.290 [ 18 ] Zhang , Yulun , et al . `` Image super-resolution using very deep residual channel attention networks . `` 291 Proceedings of the European conference on computer vision ( ECCV ) . 2018.292 [ 19 ] Lim , Bee , et al . `` Enhanced deep residual networks for single image super-resolution . '' Proceedings of the293 IEEE conference on computer vision and pattern recognition workshops . 2017.294 [ 20 ] Park , Taesung , et al . `` Semantic image synthesis with spatially-adaptive normalization . '' Proceedings of the295 IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2019.296 [ 21 ] Shang , Taizhang , et al . `` Perceptual extreme super-resolution network with receptive field block . '' Proceed-297 ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops . 2020.298 [ 22 ] Jo , Younghyun , Sejong Yang , and Seon Joo Kim . `` Investigating loss functions for extreme super-resolution . `` 299 Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops . 2020.300 Checklist301 The checklist follows the references . Please read the checklist guidelines carefully for information on how to302 answer these questions . For each question , change the default [ TODO ] to [ Yes ] , [ No ] , or [ N/A ] . You are303 strongly encouraged to include a justification to your answer , either by referencing the appropriate section of304 your paper or providing a brief inline description . For example:305 • Did you include the license to the code and datasets ? [ Yes ] See Section 2.306 • Did you include the license to the code and datasets ? [ No ] The code and the data are proprietary.307 • Did you include the license to the code and datasets ? [ N/A ] 308 Please do not modify the questions and only use the provided macros for your answers . Note that the Checklist309 section does not count towards the page limit . In your paper , please delete this instructions block and only keep310 the Checklist section heading above along with the questions/answers below.311 1 . For all authors ... 312 ( a ) Do the main claims made in the abstract and introduction accurately reflect the paper ’ s contribu-313 tions and scope ? [ Yes ] The focus of our paper is on the proposal of a one-to-many pipeline for314 super-resolution.315 ( b ) Did you describe the limitations of your work ? [ Yes ] The GAN framework used in our work316 was weak despite the successes of modern GANs.317 ( c ) Did you discuss any potential negative societal impacts of your work ? [ No ] 318 ( d ) Have you read the ethics review guidelines and ensured that your paper conforms to them ? [ Yes ] 319 2 . If you are including theoretical results ... 320 ( a ) Did you state the full set of assumptions of all theoretical results ? [ N/A ] 321 ( b ) Did you include complete proofs of all theoretical results ? [ N/A ] 322 3 . If you ran experiments ... 323 ( a ) Did you include the code , data , and instructions needed to reproduce the main experimental324 results ( either in the supplemental material or as a URL ) ? [ Yes ] As mentioned in the paper , all325 the code and training configurations are available at https : //github.com/krenerd/ultimate-sr.326 ( b ) Did you specify all the training details ( e.g. , data splits , hyperparameters , how they were chosen ) ? 327 [ Yes ] 328 ( c ) Did you report error bars ( e.g. , with respect to the random seed after running experiments329 multiple times ) ? [ No ] There was hardware limitations for repeating our work multiple times,330 however the ablation study sufficiently described improvements of our methods.331 ( d ) Did you include the total amount of compute and the type of resources used ( e.g. , type of GPUs,332 internal cluster , or cloud provider ) ? [ Yes ] All our experiments were executed on Google COLAB333 with a single Tesla T4 or Tesla K80 GPU.334 4 . If you are using existing assets ( e.g. , code , data , models ) or curating/releasing new assets ... 335 ( a ) If your work uses existing assets , did you cite the creators ? [ Yes ] The DIV2K and DIV8K336 dataset was used and cited.337 ( b ) Did you mention the license of the assets ? [ N/A ] 338 ( c ) Did you include any new assets either in the supplemental material or as a URL ? [ N/A ] 339 ( d ) Did you discuss whether and how consent was obtained from people whose data you ’ re us-340 ing/curating ? [ Yes ] The DIV2K dataset was used for the baseline ESRGAN , which was341 compared with our methods and the DIV8K dataset was used for the RFB-ESRGAN.342 ( e ) Did you discuss whether the data you are using/curating contains personally identifiable informa-343 tion or offensive content ? [ No ] 344 5 . If you used crowdsourcing or conducted research with human subjects ... 345 ( a ) Did you include the full text of instructions given to participants and screenshots , if applicable ? 346 [ N/A ] 347 ( b ) Did you describe any potential participant risks , with links to Institutional Review Board ( IRB ) 348 approvals , if applicable ? [ N/A ] 349 ( c ) Did you include the estimated hourly wage paid to participants and the total amount spent on350 participant compensation ? [ N/A ] 351

Summary: This paper develops new loss functions, network architectures, and training methods for solving the single image super resolution problem using convolutional neural networks. The aim of this effort is to develop a network that is capable of producing the many plausible HR images that could have produced a LR image. With regards to loss functions, the authors propose penalizing the difference between a downsampled HR reconstruction and the input LR image (rather than a GT HR image) and also conditioning a discriminator network on the low resolution image. With regards to network architectures, the authors pump noise into the intermediate layers of the network to generate diverse outputs. With regards to training methods, the authors remove smooth patches from the DIV2K dataset for training.