SRDGAN: learning the noise prior for Super Resolution with Dual Generative Adversarial Networks

1 INTRODUCTION . This paper targets on single image super resolution ( SISR ) . Various works have been done to recover the details from low-resolution image and generate realistic high-resolution image . Recently , learning based methods have shown great potential in solving image processing problems . Zhang et al . ( 2016 ; 2015 ) ; Guan et al . ( 2017 ) ; Guan & Cham ( 2017 ) . Especially , development of deep learning provides various learning-based solutions for super resolution Chao Dong ( 2016 ) ; Dong et al . ( 2016 ) ; Ledig et al . ( 2017 ) ; Wang et al . ( 2018b ) . Among the learning-based methods , generative adversarial network ( GAN ) , Goodfellow et al . ( 2014 ) , achieved great progress . Ledig et al . ( 2017 ) ; Wang et al . ( 2018b ) proved that GAN-based methods can generate plausible-looking natural details which are consistent with the human visual system ( HVS ) . However , mismatches exist between the data sets utilized by most existing learning-based methods and the real images captured by a mobile phone . First , in most super resolution methods , lowresolution ( LR ) images are simply downsampled from the high-resolution image with a fixed known method , for example bicubic interpolation . As a result , the informative level of the low-resolution image is actually ‘ richer ’ than the image size . However , In most practical case , Bayer pattern decides that one pixel only collect intensity for one color channel , and the left channels are usually upscaled by demosaicing . Thus , the information of mobile real-taken images is under-informative than its actual image size . Therefore , the informative level mismatch exists between the training LR data and the mobile captured data . Second , the LR image generated by downsampling is clear and with no artifacts , while the LR image captured by mobile suffers from many distortions , such as noise and blur , as shown in Fig . 1 ( a ) . The distortion mismatch also exists . These mismatches between the downsampled LR training data and real LR data ( mobile-captured data ) heavily restrict the performance of these learning-based algorithms . As shown in Fig . 1 ( b ) , the HR image generated by the state-of-the-art method ESRGAN Wang et al . ( 2018b ) can not handle real mobile images well . The other way is using optical methods to generate HR and LR image pairs , like Zhang et al . ( 2019 ) . However , it suffers many problems , such as mismatches of angle , lighting , depth-of-field and object position , which is hard to avoid . Besides , the whole process is time-consuming , thus it is hard to generate a very large-scale dataset . Therefore , an end-to-end framework is proposed in this paper , which consists of two parts . First , a high-to-low network ΦH2L is trained to generate realistic HR/LR image pair for training super resolution models , named GMSR . This dataset can promote fair comparison and support further research on the SR problems in general . Second , a low-to-high network ΦL2H is trained to produce the super resolution image . Novel training methods were utilized to reproduce better results . As shown in Fig . 1 ( c ) , the HR image generated by our method can handle the noise better than ESRGAN as illustrated by the blue patch . Moreover , better details can be reproduced as shown in the red patch . To sum up , the proposed SR method can effectively deal with the real distortions and restore fine details . The contributions of our work can be summarized into three categories . First , a general SR problem is explored which straightly face the mismatch between real mobile images and commonly utilized training data . A H2L network is proposed to solve the problems effectively . Second , an end-to-end framework is proposed , which includes two parts , high-to-low network ( H2L ) and super-resolution network ( L2H ) . During training , dual Generative Adversarial Network ( GAN ) is utilized to optimize the parameters.Third , a large-scale General Mobile Super Resolution dataset , GMSR , is generated by high-to-low network . This dataset can support further research on the super resolution domain , and can be utilized either for training or as a fair benchmark . 2 RELATED WORK . Single image super resolution ( SISR ) is an important topic and has been developed for a long time . Early methods Li & Orchard ( 2001 ) ; Allebach & Wong ( 1996 ) ; Zhang & Wu ( 2006 ) ; Duchon ( 1979 ) that are based on the interpolation theory can be very fast , however usually yield over-smooth results . Methods rely on neighbor embedding and sparse representation Chang et al . ( 2004 ) ; Kim & Kwon ( 2010 ) ; Timofte et al . ( 2013 ; 2014 ) ; Yang et al . ( 2010 ) ; Zeyde et al . ( 2010a ) targeted at learning the mapping between LR and HR . Some example-based approaches used image self-similarity property to reduce the amount of training data needed Glasner et al . ( 2009 ) ; Freedman & Fattal ( 2011 ) ; Gao et al . ( 2012 ) , and increased the size of the limited internal dictionary Huang et al . ( 2015 ) . With the development of deep learning technology , methods based on convolution neural network have shown great potential in solving super resolution problems . Chao Dong ( 2016 ) first proposed an end-to-end convolutional neural network to learn the mapping between HR and LR . Kim et al . ( 2016a ) improved the reconstruction accuracy by using very deep convolutional network and residual learning , and they further utilized recursive structure and skip-connection to improve perfor- mance without introducing new parameters Kim et al . ( 2016b ) . To increase inference efficiency , methods with low-resolution images as input Lai et al . ( 2017 ) ; Dong et al . ( 2016 ) ; Shi et al . ( 2016 ) ; Lim et al . ( 2017 ) were proposed . In Lai et al . ( 2017 ) , the authors proposed the Laplacian Pyramid network structure to reduce the computational complexity and improve the performance . Dong et al . ( 2016 ) and Shi et al . ( 2016 ) introduced different deconvolution methods to upscale the LR to HR . Most recently , GAN based super resolution methods have been proposed and achieved great progress . Ledig et al . ( 2017 ) first applied GAN to super resolution task and got highly photorealistic results . Wang et al . ( 2018b ) improved the performance by modifying the network structure and loss based on SRGAN . Network conditioning was used in Wang et al . ( 2018a ) to combine the category prior with the generative network to get more realistic textures . Yuan et al . ( 2018 ) developed unsupervised learning with GAN , where unpaired HR-LR data were utilized . Until now , most super resolution methods use the bicubic downsampled LR image as training data . However , since bicubic downsampling is quiet different from the degradation in real-world , it makes the trained model not effective to the real-world cases . Inspired by ESRGAN Wang et al . ( 2018b ) , We followed its main structure and proposed a deeper network named the low-to-high ( L2H ) network to solve the more complicated real-world cases , and build a large-scale General Mobile Super Resolution Dataset GMSR , which is potentially applicable to other image reconstruction methods . Bulat et al . ( 2018 ) adopted a degradation network to generate LR image first , then use it to train the model and get good result . However , they focus on face , which has much less complexity and diversity than general images . Hence , we extended their work to the general image super resolution . We proposed a high-to-low ( H2L ) network to learn the degradation from HR image to LR image , so that the ‘ realistic ’ LR image can be generated once we have the HR image . Zhao et al . ( 2018 ) proposed a degradation network to generate ‘ realistic ’ LR image and use it to train the degradation and SR construction network . However , they only use one discriminative network to train the degradation network . Different from them , two discriminative networks were utilized to train H2L and L2H network respectively in this work . Moreover , the two networks were jointly optimized as well . 3 PROPOSED FRAMEWORK . The proposed framework aims at solving real-world super resolution problems . In the framework , a high-to-low network is first proposed to generate realistic image pairs , which is trained with GAN using real-word images captured by mobiles . ( Section 3.2 ) . Secondly , a super resolution method with a novel structure and training strategy is explored ( Section 3.3 ) . Besides , new datasets are captured or generated , which are also introduced in this section . 3.1 DATASETS . Three novel datasets were captured and generated to train the proposed framework , including Ipari , In and GMSR . Ipair and In were captured to train the ΦH2L network . Dataset GMSR is simulated by ΦH2L and utilized to train the super resolution model ΦL2H . Details are given as follows . 1 . In order to learn the noise distribution of real world images , we applied smart phone to capture the dataset , named Ipair = { HRc , HRn } . Blackberry Key2 is selected to do this task due to its representative camera module . The resolution of images in Ipair are 4032 × 3024 . It contains 447 pairs of images , which are randomly divided into training , validation and test set with number 400 , 27 and 20 respectively . Each pair contains a noisy image HRn and a corresponding clear image HRc . To generate the clear image HRc , a burst of 20 noisy images were captured with the mobile phone fixed on a tripod . A multi-frame denoising method was used to generate a clear high-resolution image HRc by adaptively fusing the 20 noisy images . Since every noisy images can be base image for fusion , 20 pairs of images can be generated for each scene . During training , LRn is down-sampled from HRn , and constitutes an image pair with HRc . 2 . Dataset In contain 206 captured noisy image by Blackberry key2 with resolution 4032 × 3024 . Contents of the images are different from the ones in Ipair . We set In as the reference image for the discriminative network to help the high-to-low network to learn the actual noise distribution . 3 . General Mobile Super resolution GMSR is generated by ΦH2L and contains 1153 image pairs . Isim represent image pairs in dataset GMSR . The HR images in GMSR are collected from the Internet . 3.2 HIGH-TO-LOW NETWORK . The difficulty of artificially generating HR-LR image pairs is mainly two-fold . One is to model distortions in LR images , the other is to generate image pairs with pixel-wise correspondence . To solve the problems , a high-to-low network ΦH2L is proposed . Notations used in this paper are summarized in Table 1 . LR image is generated based on a clear HRc as follows : LR = ΦH2L ( HRc , N ; θ1 ) ( 1 ) where HRc is a clear high-resolution image , the generated low-resolution image LR is supposed to have similar noise distribution with real-world image . Gaussian random noise N is added to simulate the randomness of the distortions in LR . The mean and standard deviation of the Gaussian noise is 0 and 0.05 , respectively . The two values can be adjusted according to the distortion level of different mobiles . The noise transition between Gaussian distribution and that in real-world images are accomplished by ΦH2L and learned through GAN .

Summary: In the paper, the authors proposed an end-to-end single image super-resolution framework, which is composed of two parts. First, a high-to-low network is trained to generate realistic HR/LR image pairs for training super-resolution models. Using this network, the authors designed a large-scale General Mobile Super Resolution Dataset, GMSR, which can be utilized for training or as a benchmark for super-resolution methods. Second, a low-to-high network is trained to produce a super-resolution image. Beneﬁting from the GMSR dataset and novel training strategies, the proposed low-to-high network could thus deal with the real distortions and restore ﬁne details.