Title: SRDGAN: LEARNING THE NOISE PRIOR FOR SUPER RESOLUTION WITH DUAL GENERATIVE ADVERSAR-IAL NETWORKS

Abstract: Single Image Super Resolution (SISR) is the task of producing a high resolution (HR) image from a given low-resolution (LR) image. It is a well researched problem with extensive commercial applications like digital camera, video compression, medical imaging, etc. Most recent super resolution works focus on the feature learning architecture, like Chao Dong ( 2016); Dong et al. (2016); Wang et al. (2018b); Ledig et al. (2017). However, these works suffer from the following challenges: (1) The low-resolution (LR) training images are artificially synthesized using HR images with bicubic downsampling, which have much more information than real demosaic-upscaled images. The mismatch between training and realistic mobile data heavily blocks the effect on practical SR problem.
(2) These methods cannot effectively handle the blind distortions during super resolution in practical applications. In this work, an end-to-end novel framework, including high-to-low network and low-to-high network, is proposed to solve the above problems with dual Generative Adversarial Networks (GAN). First, the above mismatch problems are well explored with the high-to-low network, where clear high-resolution image and the corresponding realistic low-resolution image pairs can be generated. With high-to-low network, a large-scale General Mobile Super Resolution Dataset, GMSR, is proposed, which can be utilized for training or as a benchmark for super resolution methods. Second, an effective low-to-high network (super resolution network) is proposed in the framework. Benefiting from the GMSR dataset and novel training strategies, the proposed super resolution model can effectively handle detail recovery and denoising at the same time.

1. INTRODUCTION:

This paper targets on single image super resolution (SISR). Various works have been done to recover the details from low-resolution image and generate realistic high-resolution image. Recently, learning based methods have shown great potential in solving image processing problems. Zhang et al. (2016; 2015) ; Guan et al. (2017) ; Guan & Cham (2017) . Especially, development of deep learning provides various learning-based solutions for super resolution Chao Dong (2016) ; Dong et al. (2016) ; Ledig et al. (2017) ; Wang et al. (2018b) . Among the learning-based methods, generative adversarial network (GAN), Goodfellow et al. (2014) , achieved great progress. Ledig et al. (2017) ; Wang et al. (2018b) proved that GAN-based methods can generate plausible-looking natural details which are consistent with the human visual system (HVS).

However, mismatches exist between the data sets utilized by most existing learning-based methods and the real images captured by a mobile phone. First, in most super resolution methods, lowresolution(LR) images are simply downsampled from the high-resolution image with a fixed known method, for example bicubic interpolation. As a result, the informative level of the low-resolution image is actually 'richer' than the image size. However, In most practical case, Bayer pattern decides that one pixel only collect intensity for one color channel, and the left channels are usually upscaled by demosaicing. Thus, the information of mobile real-taken images is under-informative than its actual image size. Therefore, the informative level mismatch exists between the training LR data and the mobile captured data. Second, the LR image generated by downsampling is clear and with no artifacts, while the LR image captured by mobile suffers from many distortions, such as noise and blur, as shown in Fig. 1 (a) . The distortion mismatch also exists. These mismatches between the downsampled LR training data and real LR data (mobile-captured data) heavily restrict the performance of these learning-based algorithms. As shown in Fig. 1 The other way is using optical methods to generate HR and LR image pairs, like Zhang et al. (2019) . However, it suffers many problems, such as mismatches of angle, lighting, depth-of-field and object position, which is hard to avoid. Besides, the whole process is time-consuming, thus it is hard to generate a very large-scale dataset. Therefore, an end-to-end framework is proposed in this paper, which consists of two parts. The contributions of our work can be summarized into three categories. First, a general SR problem is explored which straightly face the mismatch between real mobile images and commonly utilized training data. A H2L network is proposed to solve the problems effectively. Second, an end-to-end framework is proposed, which includes two parts, high-to-low network (H2L) and super-resolution network (L2H). During training, dual Generative Adversarial Network (GAN) is utilized to optimize the parameters.Third, a large-scale General Mobile Super Resolution dataset, GMSR, is generated by high-to-low network. This dataset can support further research on the super resolution domain, and can be utilized either for training or as a fair benchmark.

2. RELATED WORK:

Single image super resolution (SISR) is an important topic and has been developed for a long time.

Early methods Li & Orchard (2001) ; Allebach & Wong (1996) ; Zhang & Wu (2006) ; Duchon (1979) that are based on the interpolation theory can be very fast, however usually yield over-smooth results. Methods rely on neighbor embedding and sparse representation Chang et al. (2004) ; Kim & Kwon (2010) ; Timofte et al. (2013; 2014) ; Yang et al. (2010) ; Zeyde et al. (2010a) targeted at learning the mapping between LR and HR. Some example-based approaches used image self-similarity property to reduce the amount of training data needed Glasner et al. (2009) ; Freedman & Fattal (2011) ; Gao et al. (2012) , and increased the size of the limited internal dictionary Huang et al. (2015) .

With the development of deep learning technology, methods based on convolution neural network have shown great potential in solving super resolution problems. Chao Dong (2016) first proposed an end-to-end convolutional neural network to learn the mapping between HR and LR. Kim et al. (2016a) improved the reconstruction accuracy by using very deep convolutional network and residual learning, and they further utilized recursive structure and skip-connection to improve perfor- Most recently, GAN based super resolution methods have been proposed and achieved great progress. Ledig et al. (2017) first applied GAN to super resolution task and got highly photorealistic results. Wang et al. (2018b) improved the performance by modifying the network structure and loss based on SRGAN. Network conditioning was used in Wang et al. (2018a) to combine the category prior with the generative network to get more realistic textures. Yuan et al. (2018) developed unsupervised learning with GAN, where unpaired HR-LR data were utilized. Until now, most super resolution methods use the bicubic downsampled LR image as training data. However, since bicubic downsampling is quiet different from the degradation in real-world, it makes the trained model not effective to the real-world cases. Inspired by ESRGAN Wang et al. (2018b) , We followed its main structure and proposed a deeper network named the low-to-high(L2H) network to solve the more complicated real-world cases, and build a large-scale General Mobile Super Resolution Dataset GMSR, which is potentially applicable to other image reconstruction methods. Bulat et al.

C O N V C O N V H2L L2H C O N V C O N V C O N V Resblock Resblock Resblock Resblock

(2018) adopted a degradation network to generate LR image first, then use it to train the model and get good result. However, they focus on face, which has much less complexity and diversity than general images. Hence, we extended their work to the general image super resolution. We proposed a high-to-low(H2L) network to learn the degradation from HR image to LR image, so that the 'realistic' LR image can be generated once we have the HR image. Zhao et al. (2018) proposed a degradation network to generate 'realistic' LR image and use it to train the degradation and SR construction network. However, they only use one discriminative network to train the degradation network. Different from them, two discriminative networks were utilized to train H2L and L2H network respectively in this work. Moreover, the two networks were jointly optimized as well.

3. PROPOSED FRAMEWORK:

The proposed framework aims at solving real-world super resolution problems. In the framework, a high-to-low network is first proposed to generate realistic image pairs, which is trained with GAN using real-word images captured by mobiles. (Section 3.2). Secondly, a super resolution method 

EQUATION

where HR c is a clear high-resolution image, the generated low-resolution image LR is supposed to have similar noise distribution with real-world image. Gaussian random noise N is added to simulate the randomness of the distortions in LR. The mean and standard deviation of the Gaussian noise is 0 and 0.05, respectively. The two values can be adjusted according to the distortion level of different mobiles. The noise transition between Gaussian distribution and that in real-world images are accomplished by Φ H2L and learned through GAN. 

3.2.1. MODEL STRUCTURE:

Φ H2L is able to model the image distortion of real mobile images caused in the image capturing and processing pipeline. The structure of Φ H2L network is shown in Fig. 2 , which consists of 4 Resblocks and 1 convolutional layer. Each resblock consists of two convolution layers and two activation functions. In order to further build the global connection Ahn et al. ( 2018), multiple shortcut connections is added as indicated by the red lines. The final convolution layer aims at decreasing the resolution by setting the downscale factor as the stride.

3.2.2. TRAINING PROCEDURE OF Φ H2L:

The training strategy of Φ H2L is summarized in Fig. 3 . During the training process, dataset I pair = {HR n , HR c } and I n are used. In order to get the ground-truth of LR, noisy image HR n is downsampled to generate LR n , which is used to calculate the pixel and feature losses. However, the noise distribution of LR n is not consistent with HR n after the down-sampling. Therefore, Generative Adversarial Network (GAN) is used to further learn the noise distribution of mobile images. The loss function and training strategy are introduced as follows.

Loss function: Pixel loss, feature loss (i.e. perceptual loss) and GAN loss are used to train Φ H2L .

EQUATION

where α 1 , α 2 and α 3 are weighting parameters.

Pixel loss loss p H2L aims at pixel-wisely comparing the image difference between LR n and the network output LR, as

EQUATION

where n p LR corresponds to the number of pixels in LR. Feature loss aims at comparing the difference between high-level features of two images, which has been proved to be effective in many previous works Ledig et al. (2017) ; Wang et al. (2018b) ; He et al. (2016) ; Johnson et al. (2016) . Inspired by these works, VGG Simonyan & Zisserman (2014) is utilized as feature extraction network ϕ:

EQUATION

Where ϕ(LR) is the extracted high-level features of input image LR by ϕ. k represents the index in the extracted features. GAN is then used to learn the distribution by discriminating unpaired images, I nc and LR n , where I nc is the low-resolution image randomly cropped from I n .

loss GAN H2L = 1 n GAN LR n GAN LR n=1 -logD θ2 (Φ H2L (HR n , N )) (5)

where n GAN LR is the number of batch size, D θ2 is the discriminative model with parameters θ 2 . The goal of D θ2 is to well distinguish the generated LR image and real image patch I nc . D θ2 is trained iteratively with the generative model Φ H2L as shown in Equ. 6.

EQUATION

In this work, nearest neighbour is explored as the down-sampling method to better simulate the real-world ill-informative situation of mobile images.

3.3. LOW-TO-HIGH NETWORK:

Network Φ L2H is proposed to implement super resolution based on the low-resolution image LR with parameter θ 3 as in Equation 7.

HR = Φ L2H (LR; θ 3 ) (7) 3.3.1 NETWORK STRUCTURE

The structure of Φ L2H is shown in Fig. 2 . RRDB block from Wang et al. (2018b) is used as the basic network unit. a deeper structure with 25 RRDB blocks is proposed to model the more complex task. The basic unit is not limited to RRDB block, other basic block like residual block and dense block can also work in the proposed network structure.

3.3.2. TRAINING PROCEDURE OF Φ L2H:

The training strategy of Φ L2H is summarized in Fig. 3 . The loss function of Φ L2H is similar to that of Φ H2L , which is described in Section 3.2.2.

EQUATION

where α 4 , α 5 and α 6 are weighting parameters. The main difference of the training strategy between Φ H2L and Φ L2H is training data. For Φ L2H , two categories of training samples are utilized. The most important one is I sim from dataset GMSR that simulates real-world distortions. Furthermore, the popular dataset DIV2K Timofte et al. ( 2017) is utilized to further increase the data diversity.

Finally, global fine-tuning is performed on the whole network structure including Φ H2L , Φ L2H and the discriminative networks, to jointly optimize the performance. [50k, 100k, 200k, 300k] iterations. AdamKingma & Ba (2014) optimizer is utilized with β 1 = 0.9, β 2 = 0.999 without weight decay.

4. EXPERIMENTS:



4.1. SUPER RESOLUTION RESULTS:

Quantitative and Subjective experiments were conducted to evaluate the performance of the proposed method. Table 2 shows the quantitative evaluation results of the popular and state-of-the-art GAN-based super resolution methods with an upscale factor of 4. 2004) are adopted for measuring the effectiveness of the proposed method, where higher value indicates better performance. From Table 2 , it is shown that the proposed method outperforms the other state-of-the-art methods in terms of SSIM in all circumstances, demonstrating its effectiveness on reconstructing structure. Especially, the improvements achieved on Set I pair are significant, showing its good performance on mobile images.The best performing results are shown in bold.

Subjective comparison examples were presented as shown in Fig. 4 . Real-world mobile images are utilized. PSNR and SSIM results are presented at the bottom for reference, which are evaluated on Y channel. In many situations, mobile images suffer from noise and detail missing. Most methods cannot handle these cases well, including bicubic, SRCNN Chao Dong (2016) , SRGAN Ledig et al. (2017) , ESRGAN Wang et al. (2018b) . The noise is still very obvious, and even increased in SR results. Instead, our method can deliver good results by removing all the noise and well-reserve the edges and details.

Moreover, in order to verify the generalization ability of the proposed method, subjective evaluation experiments were conducted based on real phone-captured images. 24 images captured by three different mobile phones (Iphone7p, Xiaomi Mix2s and Iphone Xs Max) were used as inputs. We compared the performance of three SR methods: Bicubic, ESRGAN and the proposed SRDGAN. Totally 21 subjects without expert knowledge participated the experiment, who evaluated each upsampled image subjectively using a score ranging from one to three. Higher score represents better visual quality. The experimental results are shown in Table 3 , which confirms that in the real-world case, the proposed method is applicable, and it largely outperforms Bicubic and the state-of-the-art method ESRGAN. 

4.2. RESULTS OF Φ H2L:



:



Figures and Tables:

Figure 1: (a) Example of an mobile low-resolution image, which is utilized as the input of super resolution (SR) methods. (b) Examples of super-resolution result via state-of-theart method ESRGAN Wang et al. (2018b). (c) Examples of SR result via the proposed method. (Best viewed in color.)

First, a high-to-low network Φ H2L is trained to generate realistic HR/LR image pair for training super resolution models, named GMSR. This dataset can promote fair comparison and support further research on the SR problems in general. Second, a low-to-high network Φ L2H is trained to produce the super resolution image. Novel training methods were utilized to reproduce better results. As shown in Fig. 1 (c), the HR image generated by our method can handle the noise better than ESRGAN as illustrated by the blue patch. Moreover, better details can be reproduced as shown in the red patch. To sum up, the proposed SR method can effectively deal with the real distortions and restore fine details.

Figure2: Pipeline of the proposed SRDGAN framework, including Φ H2L and Φ L2H . The high-tolow network Φ H2L is proposed to generate realistic HR/LR image pairs. The low-to-high network Φ L2H is proposed to produce the super resolution image.

procedure of the proposed framework with dual GAN. (a) introduces the training process of Φ H2L , (b) is the training procedure of Φ L2H .

Figure 4: Subjective comparison between the proposed method and some state-of-the-art methods. Left is the mobile low-resolution image example and a detail patch. Right is the super-resolution result of different methods and high-resolution ground-truth (HR). (Best viewed in color)

Notations used in the paper. Φ H2L High-to-low Network I pair ={HR c , HR n } paired clear image HR c and noisy image HR n LR n Low-resolution image downsampled from HR n Three novel datasets were captured and generated to train the proposed framework, including I pari , I n and GMSR. I pair and I n were captured to train the Φ H2L network. Dataset GMSR is simulated by Φ H2L and utilized to train the super resolution model Φ L2H . Details are given as follows. It contains 447 pairs of images, which are randomly divided into training, validation and test set with number 400, 27 and 20 respectively. Each pair contains a noisy image HR n and a corresponding clear image HR c . To generate the clear image HR c , a burst of 20 noisy images were captured with the mobile phone fixed on a tripod. A multi-frame denoising method was used to generate a clear high-resolution image HR c by adaptively fusing the 20 noisy images. Since every noisy images can be base image for fusion, 20 pairs of images can be generated for each scene. During training, LR n is down-sampled from HR n , and constitutes an image pair with HR c . 2. Dataset I n contain 206 captured noisy image by Blackberry key2 with resolution 4032 × 3024. Contents of the images are different from the ones in I pair . We set I n as the reference image for the discriminative network to help the high-to-low network to learn the actual noise distribution. 3. General Mobile Super resolution GMSR is generated by Φ H2L and contains 1153 image pairs. I sim represent image pairs in dataset GMSR. The HR images in GMSR are collected from the Internet. 3.2 HIGH-TO-LOW NETWORK The difficulty of artificially generating HR-LR image pairs is mainly two-fold. One is to model distortions in LR images, the other is to generate image pairs with pixel-wise correspondence. To solve the problems, a high-to-low network Φ H2L is proposed. Notations used in this paper are summarized in Table 1. LR image is generated based on a clear HR c as follows:

Table content: <table><tr><td>I n</td><td>Mobile captured noisy images. No corresponding clear images</td></tr><tr><td>I nc</td><td>Cropped LR image from I n</td></tr><tr><td>GMSR</td><td>Proposed general mobile super resolution dataset</td></tr><tr><td>I sim</td><td>Image pairs in dataset GMSR</td></tr><tr><td>HR c</td><td>Clear high-resolution images as input of Φ H2L</td></tr><tr><td>N</td><td>Random noise element-wised added to HR c</td></tr><tr><td>Φ L2H</td><td>Low-to-high Network, i.e. super-resolution network</td></tr><tr><td colspan="2">with a novel structure and training strategy is explored (Section 3.3). Besides, new datasets are</td></tr><tr><td colspan="2">captured or generated, which are also introduced in this section.</td></tr><tr><td>3.1 DATASETS</td><td/></tr></table>

Quantitative evaluation of SR algorithms (PSNR/SSIM)

Table content: <table><tr><td>Eexperiments were conducted to</td><td/></tr><tr><td>evaluate the effectiveness of the whole proposed framework and its constituents including the H2L network and the training strategy. The parameters for training the proposed framework are described as follows. Batch-size is 16 with</td><td>Method Bicubic SRCNN SRGAN ESRGAN 30.55/0.8677 26.39/0.7246 38.55/0.9572 Set5 Set14 Set I pair (test) 28.42/0.8104 26.00/0.7027 35.85/0.9674 30.07/0.8627 27.50/0.7513 39.09/0.9716 29.40/0.8472 26.02/0.7397 38.43/0.9596 Ours 30.37/0.8766 27.35/0.7767 39.93/0.9782</td></tr><tr><td>training patch of size 192 × 192.</td><td/></tr><tr><td colspan="2">Following ESRGAN Wang et al. (2018b), the learning rate is initialized with 1×10 -4 , and halved at</td></tr></table>

Subjective evaluation on real mobile images Both general datasets (set5 Bevilacqua et al. (2012), set14 Zeyde et al. (2010b)) and the mobile-specific dataset (I pair ) are utilized. PSNR and SSIM Wang et al. (

Table content: <table><tr><td>Mobile</td><td colspan="3">Iphone7p Xiaomi IphoneXs</td><td>All</td></tr><tr><td>Bicubic</td><td>1.78</td><td>1.35</td><td>1.69</td><td>1.61</td></tr><tr><td>ESRGAN</td><td>1.83</td><td>2.31</td><td>1.98</td><td>2.04</td></tr><tr><td>Ours</td><td>2.39</td><td>2.34</td><td>2.32</td><td>2.35</td></tr></table>

as baseline. Specifically, high-quality HR images of GMSR dataset and DIV2K are downsampled with bicubic interpolation, Gaussian blurred with random intensity to simulate out-of-focus, converted into Bayer data, added with Gaussian noise, and finally debayer-filtered to generate the LR images. We used this baseline dataset to train the proposed SR model and named the new super resolution model SR bl . Quantitative results are shown in Table4. Our model trained on GMSR outperforms SR bl in all cases, demonstrating the effectiveness of the proposed H2L network and the GMSR dataset. 4.3 EXPLORING TRAINING STRATEGIES FOR Φ L2H Experiments were conducted to evaluate the effectiveness of the new dataset, new training strategies and the proposed model structure, as shown in Fig. 5. 'ESRGAN' represents the original ESRGAN method with GAN-based structure and DIV2KTimofte et al. (2017) (bicubic downsampling) as training data. PSNR and SSIM Wang et al. (2004) are utilized, where larger value represents better performance. The test was conducted on both general dataset set14 and the mobile image dataset Set I pair . It can be seen that the 'ESRGAN*' outperforms 'ESRGAN' by changing training data. It proves that the new dataset(GMSR) and new training strategy can be easily generalized to other model structures and improve the performance on both general data and mobile data. Moreover, 'Ours' further improves the performance compared with 'ESRGAN*'. It proves the effectiveness of the model structure in Φ L2H .5 CONCLUSIONIn this work, a novel framework SRDGAN is proposed to solve the noise prior super resolution problem with dual generative adversarial network. A general degradation network H2L is proposed and is able to learn the noise prior of real-world LR image. By using our proposed training strategy, the H2L network is trained to generate the 'realistic' LR image paired with the HR image, and A large-scale benchmark general mobile super resolution dataset, GMSR, is generated from it. The GMSR can help super resolution methods to be well applied in the real-world mobile taken images. Meanwhile, a super resolution network L2H is proposed, which contains new structure and training strategies. Especially, these training strategies are proved to be effective for other SR model, such as ESRGAN, and improve the SR performance.

Table content: <table><tr><td>With Φ H2L , we aim to simulate LR</td><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>images that is a better ensemble of</td><td colspan="9">Quantitative evaluation of SR bl and ours algorithms(PSNR/SSIM)</td></tr><tr><td>real mobile images and to generate a large-scale dataset. To be fair, a train-</td><td/><td>Method SR bl</td><td/><td colspan="6">Set5 29.72/0.8404 27.05/0.7357 Set14</td><td>Set Ipair(test) 39.04/0.9736</td></tr><tr><td>ing dataset that pre-processed with</td><td/><td>Ours</td><td/><td colspan="6">30.37/0.8766 27.35/0.7767</td><td>39.93/0.9782</td></tr><tr><td>artificial degradation methods is gen-</td><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="7">erated and serves PSNR 45 40 'ESRGAN*' represents the new-trained method with the same model structure but new datasets for training, includ-ing DIV2K Timofte et al. (2017) (Near-</td><td>38.55</td><td>39.42</td><td>39.94</td><td>1.1 1</td><td>SSIM</td><td>0.9572</td><td>0.9761</td><td>0.9782</td></tr><tr><td colspan="2">est neighbour downsampling) and GMSR. 'Ours' is the proposed method with the new model structure and new datasets</td><td>35 30</td><td>26.39</td><td>26.94</td><td>27.35</td><td/><td/><td/><td>0.9 0.8</td><td>0.7246</td><td>0.7618</td><td>0.7767</td></tr><tr><td colspan="2">for training. Comparison between 'ES-RGAN' and 'ESRGAN*' can evaluate</td><td>25</td><td/><td/><td/><td/><td/><td/><td>0.7</td></tr><tr><td colspan="2">the effectiveness of new training datasets. While performance comparison between</td><td>20</td><td colspan="2">Set14</td><td/><td/><td>Set</td><td/><td>0.6</td><td>Set14</td><td>Set</td></tr><tr><td colspan="2">'ESRGAN*' and 'Ours' evaluates the im-</td><td colspan="4">ESRGAN</td><td colspan="3">ESRGAN*</td><td>ours</td><td>ESRGAN</td><td>ESRGAN*</td><td>ours</td></tr><tr><td>pacts in the model structure.</td><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td/><td colspan="8">Figure 5: Comparing ESRGAN with the proposed method</td></tr></table>

Peer review (iclr2020):
This paper presents a single image super-resolution method. The motivation is real and practical. In almost all existing methods the low-resolution training images are artificially synthesized using the available high-resolution images with bicubic downsampling, which allows LR images to carry more information than real demosaic-upscaled LR images. This mismatch between training and realistic data hinders the practicability of such solutions (not limited to mobile scenarios). Yet, the proposed remedy in the paper is not an answer either. It simply adds a noise image to the HR image, passes it through a convolutional network to construct an LR version, which is then fed into a deconvolutional network. Both these networks are build by residual blocks. Even worse, the paper simply uses zero-mean Gaussian random noise with a standard deviation of 0.05 to simulate the distortions. This is myopic and questionable. The introduced dataset is very small, it contains only 447 pairs of images, with an additional 206 images as reference images. These pairs consist of a normal image (considered to be noisy) and a computed image by aggregating 20 images (considered to be clean) using a Blackberry Key2. In other words, the dataset only models a single device and a single camera lens. For practical purposes, this is not much different than using a bicubic sampling and adding noise drawn from a fixed distribution. The paper fails to provide comparisons with the top methods in the NTIRE 2019 single image super-resolution challenge leaderboard.