SRDGAN: learning the noise prior for Super Resolution with Dual Generative Adversarial Networks
Authors: Jingwei GUAN
Cheng PAN
Songnan LI and Dahai YU
Venue:  (2020)
Keywords: Super Resolution GAN Denoise

Abstract: Single Image Super Resolution (SISR) is the task of producing a high resolution (HR) image from a given low-resolution (LR) image. It is a well researched prob- lem with extensive commercial applications like digital camera, video compres- sion, medical imaging, etc. Most recent super resolution works focus on the fea- ture learning architecture, like Chao Dong (2016); Dong et al. (2016); Wang et al. (2018b); Ledig et al. (2017). However, these works suffer from the following chal- lenges: (1) The low-resolution (LR) training images are artificially synthesized us- ing HR images with bicubic downsampling, which have much more information than real demosaic-upscaled images. The mismatch between training and realistic mobile data heavily blocks the effect on practical SR problem. (2) These methods cannot effectively handle the blind distortions during super resolution in practical applications. In this work, an end-to-end novel framework, including high-to-low network and low-to-high network, is proposed to solve the above problems with dual Generative Adversarial Networks (GAN). First, the above mismatch prob- lems are well explored with the high-to-low network, where clear high-resolution image and the corresponding realistic low-resolution image pairs can be gener- ated. With high-to-low network, a large-scale General Mobile Super Resolution Dataset, GMSR, is proposed, which can be utilized for training or as a bench- mark for super resolution methods. Second, an effective low-to-high network (super resolution network) is proposed in the framework. Benefiting from the GMSR dataset and novel training strategies, the proposed super resolution model can effectively handle detail recovery and denoising at the same time.

Under review as a conference paper at ICLR 2020
SRDGAN: LEARNING THE NOISE PRIOR FOR SUPER RESOLUTION WITH DUAL GENERATIVE ADVERSAR- IAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Single Image Super Resolution (SISR) is the task of producing high resolution (HR) image from a given low-resolution (LR) image. It is a well researched prob- lem with extensive commercial applications like digital camera, video compres- sion, medical imaging, etc. Most recent super resolution works focus on the fea- ture learning architecture, like Chao Dong 2016); Dong al. (2016); Wang et al.] : Ledig et al. 2017). However, these works suffer from the following chal- lenges: (h)The low-resolution (L.R) training images artificially synthesized us- ing HR images with bicubic downsampling, which have much more information than real demosaic-upscaled images. The mismatch between training and realistic mobile data heavily blocks the effect practical SR problem. (2) These methods cannot effectively handle the blind distortions during super resolution in practical applications. In this work, an end-to-end novel framework, including high-to-low network and low-to-high network, proposed to solve the above problems with dual Generative Adversarial Networks (GAN). First, the above mismatch prob- lems are well explored with the high-to-low network, where clear high-resolution image and the corresponding realistic low-resolution image pairs can be gener- ated. With high-to-low network, a large-scale General Mobile Super Resolution Dataset, GMSR, is proposed, which can be utilized for training as bench- mark for super resolution methods. Second, an effective low-to-high network (super resolution network) is proposed in the framework. Benefiting from the GMSR dataset and novel training strategies, the proposed super resolution model can effectively handle detail recovery and denoising at the same time.
(2018b):
INTRODUCTION
This paper targets on single image super resolution (SISR). Various works have been done to recover the details from low- resolutioti image and generate realistic high-resolution image. Recently, learn- ing based methods have shown great potential in solving image processing problems. Zhang et al.) 2016 2015); Guan al. (2017): ) Guan & Cham (2017). Especially, development of deep learn- ing provides various learning- -aaneng-based based solutions for super resolution Chao Dong (2016); Dong et al.)
(2016): Ledig et al. (2017): Wang al. (2018b). Among the learning-based methods, generative adversarial network (GAN), Goodfellow et al. (2014), achieved great progress. [edig et a (2017);
Wang et al. (20T8b) proved that GAN-based methods can generate plausible-looking natural details which are consistent with the human visual system (HVS).
However, mismatches exist between the data sets utilized by most existing learning-basee based methods and the real images captured by a mobile phone. First, in most super resolution methods, low- resolution(L. images are simply downsampled from the high-resolution image with a fixed known method, for example bicubic interpolation. As result, the informative level of the low-resolution image is actually 'richer' than the image size. However, In most practical case, Bayer pattern decides that one pixel only collect intensity for one color channel, and the left channels are usually upscaled by demosaicing. Thus, the information of mobile real-taken images is under-informatioo than its actual image size. Therefore, the informative level mismatch exists between the training LR data and the mobile captured data. Second, the LR image generated by downsampling is clear and with no artifacts, while the LR image captured by mobile suffers from many distortions, such as

---

Under review as a conference paper at ICLR 2020
noise and blur, shown in Fig. ! (a). The distortion mismatch also exists. These mismatches between the downsampled LR training data and real LR data (mobile-captured data) heavily restrict the performance of these learning- baseed algorithms. As shown in Fig. (b), the HR image generated by the state-of-the-ard method ESRGAN Wang et al. (2088b) cannot handle real mobile images well.
The other way is using optical methods to generate HR and LR image pairs, like Zhang et al. (2019). However, it suffers many problems, such as mismatches of an- gle, lighting, depth-of- of-fill and object po- sition, which is hard to avoid. Besides, the whole process is time-conumming thus is hard to generate very large-scale dataset. Therefore, an end-to-end frame- work is proposed in this paper, which con- sists of two parts. First, a high-to-low net- work H2L is trained to generate realis- tic HR/LR image pair for training super resolution models, named GMSR. This dataset can promote fair comparison and support further research on the SR prob- lems in general. Second, low-to-high network L21 trained produce the super resolution image. Novel training methods were utilized reproduce better results. As shown in Fig. (c), the HR image generated by our method can han- dle the noise better than ESRGAN as illus- trated by the blue patch. Moreover, better details can be reproduced as shown in the red patch. To sum up, the proposed SR method can effectively deal with the real distortions and restore fine details.
(a) Low-resolution image
(b) High High-resolution result by ESRGAN
(c) High- resolution resalt proposed method
Figure 1: (a) Example of an mobile low-rcsolution image, which is utilized the input of super resolution (SR) meth- ods. (b) Examples of super-resolution result via state-of-the-
method ESRGAN [Wang al. (2018b). (c) Examples of
SR result via the proposed method. (Best viewed in color.)
The contributions of work can be summarized into three categories. First, general SR problem explored which straightly face the mismatch between real mobile images and commonly utilized training data. H2L network is proposed to solve the problems effectively. Second, an end-to-end framework is proposed, which includes two parts, high-to-low network (H2L) and super-resolution network (L.2H). During training, dual Generative Adversarial Network (GAN) is utilized to optimize the parameters. Third, a large-scale General Mobile Super Resolution dataset, GMSR, is generated by high-to-low network. This dataset can support further research on the super resolution domain, and can be utilized either for training as fair benchmark.
2 RELATED WORK
Single image super resolution (SISR) is an important topic and has been developed for a long time. Early methods Li & Orchard (00T): Allebach & Wong
( (196)) & Wu (2006): (uchon (1979) that are based on the interpolation theory can be very fast, however usually yield over-smooll results. Methods rely on neighbor embedding and sparse representation Chang et al. (2004); Kim Kwon
(2100); Timofte et al. (2013) 2014); Yang et al. (2010); Zeyde et al. (2010a) targeted at learning the mapping between L.R and HR. Some example-bass based approaches used image self-similarity property to reduce the amount of training data needed Glasner et al. (2009); Freedman & Fattal (2011) Gao
ctt al. (2012), and increased the size of the limited internal dictionary Huang et al. (0155)
With the development of deep learning technology, methods based on convolution neural network have shown great potential in solving super resolution problems. Chao Dong (2016) first proposed end-to-end convolutional neural network to learn the mapping between HR and LR. Kim et al. (2016a) improved the reconstruction accuracy by using very deep convolutional network and resid- ual learning, and they further utilized recursive structure and skip-connection improve perfor-
2

---

Under review as a conference paper at ICLR 2020
X34
I1R1 Block
BR Block
HRDD Block
(
T H1L.
ItRe
LR (Rs, N:40)
Figure 2: Pipeline of the proposed SRDGAN framework, including #22 and L22H. The high-to- low network T H2L is proposed to generate realistic HR/LR image pairs. The low-to-high network T L2H is proposed to produce the super resolution image.
mance without introducing new parameters Kim et al. (0116)). To increase inference efficiency,
methods with low-resolution images input Lai al. 2017); Dong al. (2016); (Shi et al. (2016);
Lim et al. (2017) were proposed. In Lai et al. (2017). the authors proposed the Laplacian Pyramid network structure reduce the computational complexity and improve the performance, Dong al. 2016) and Shi et al. (2116) introduced different deconvolution methods to upscale the LR to HR.
Most recently, GAN based super resolution methods have been proposed and achieved great progress. Ledig et al. (2017) first applied GAN to super resolution task and got highly photo- realistic results. Wang et al. (2188) improved the performance by modifying the network structure and loss based on SRGAN. Network conditioning was used in Wang et al. (2018a) to combine the category prior with the generative network to get more realistic textures. Yuann et al. (2018) devel- oped unsupervised learning with GAN, where unpaired HR-LR data were utilized. Until now, most super resolution methods use the bicubic downsampled LR image as training data. However, since bicubic downsampling is quiet different from the degradation in real-world, it makes the trained model not effective to the wooldworld cases. Inspired by ESRGAN Wang et al. (2018b). We followed its main structure and proposed deeper network named the low-to-high([2HH) network solve the more complicated real-world cases, and build a large-scale General Mobile Super Resolution Dataset GMSR, which is potentially applicable to other image reconstruction methods. BBlltt et al. (0018) adopted degradation network to generate LR image first, then use it to train the model and get good result. However, they focus face, which has much less complexity and diversity than general images. Hence, we extended their work to the general image super resolution. We pro- posed high-to-low(H2L) network learn the degradation from HR image LR image, so that the "realistic" LR image can be generated once we have the HR image. Zhho et al. (2018) proposed a degradation network to generate 'realistic' L.R image and use train the degradation and SR construction network. However, they only one discriminative network to train the degradation network. Different from them, two discriminative networks were utilized to train H2L and L2H network respectively in this work. Moreover, the two networks were jointly optimized well.
PROPOSED FRAMEWORK
The proposed framework aims solving real-world super resolution problems. In the framework, high-to-low network is first proposed to generate realistic image pairs, which is trained with GAN using real-word images captured by mobiles. (Section 3.2). Secondly, super resolution method

---

Under review as a conference paper at ICLR 2020
Table 1: Notations used in the paper.
H2L
High-to-low Network
I pair
(HR HR,]
paired clear image HR; and noisy image HR,
LRR In Inc GMSR Leim
Low-resolution image downsampled from HR,,
Mobile captured noisy images. No corresponding clear images Cropped LR image from I,.
Proposed general mobile super resolution dataset Image pairs in dataset GMSR
HR,
Clear high-resolution images as input of O H22 Random noise gementtwwsset added to HR,
N
L22H
Low-to-high Network, i.c. super-resolution network
with a novel structure and training strategy is explored (Section captured or generated, which are also introduced in this section.
3.3 Besides, new datasets are
3.1 DATASETS
Three novel datasets were captured and generated to train the proposed framework, including Lprii I, and GMSR. [pair and I, were captured to train the & #2 network. Dataset GMSR is simulated by 422 and utilized to train the super resolution model L22H- Details are given as follows.
1. In order to learn the noise distribution of real world images, we applied smart phone to capture the dataset, named ppair Re., HR, Blackberry Key2 is selected to do this task due to its representative camera module. The resolution of images Ipoir are 4032 x 3024. It contains 447 pairs of images, which are randomly divided into training, validation and test set with number 400, 27 and 20 respectively. Each pair contains a noisy image H Rn and corresponding clear image H R.. To generate the clear image HR,, burst of 20 noisy images were captured with the mobile phone fixed on a tripod.
A multi-frame denoising method was used to generate clear high-resolution image HR, by adaptively fusing the 20 noisy images. Since every noisy images can be base image for fusion, 20 pairs of images down-sampled from H R,,, and constitutes an image pair with HR.-
be generated for each scene. During training, LR, is
2. Dataset I contain 206 captured noisy image by Blackberry key2 with resolution 4032 x 3024. Contents of the images are different from the ones in 4pair- We set In as the reference image for the discriminative network to help the high-to-low network to learn the actual noise distribution.
3. General Mobile Super resolution GMSR is generated by H22 and contains 1 1153 image pairs. Isim represent image pairs dataset GMSR. The HR images in GMSR are collected from the Internet.
3.2 HIGH-TO-LLOW NETWORK
The difficulty of artificially generating HR-LR image pairs is mainly two-fold. One is to model distortions in LR images, the other is to generate image pairs with pixel-wise correspondence. To solve the problems, a high-to-low network H2L is proposed. Notations used in this paper are summarized in Table T LR image is generated based on clear H R., follows:
LR = 222 (HR,
(1)
where HR, is a clear high-resolution image, the generated low-resolution image LR is supposed to have similar noise distribution with real-world image. Gaussian random noise N is added to simulate the randomness the distortions in LR. The mean and standard deviation of the Gaussian noise is 0 and 0.05, respectively. The values be adjusted according to the distortion level of different mobiles. The noise transition between Gaussian distribution and that in real-world images accomplished by d H22 and learned through GAN.
4

---

Under review as a conference paper at ICLR 2020
D2
LR
1211
IIC
HIRe
Pix/Fea loss
Pix/Fea loss
Iia
(a) Train 1111
(b) Train
Training procedure of the proposed framework with dual GAN. (a) introduces the training process of H22. (b) the training procedure of & L2HH-
Figure 3:
3.2.1 MODEL STRUCTURE
PH2z is able to model the image distortion of real mobile images caused in the image capturing and processing pipeline. The structure of & #22. network is shown in Fig. which consists of 4 Resblocks and 1 convolutional layer. Each resblock consists of two convolution layers and two activation functions. order to further build the global connection Ahn et al.] (2018), multiple shortcut connections is added as indicated by the red lines. The final convolution layer aims at decreasing the resolution by setting the downscale factor as the stride.
3.2.2 TRAINING PROCEDURE 1121.
The training strategy of F22, is summarized in Fig. 3 During the training process, dataset Ipai [H,, R. and I, are used. In order to get the ground-truth of LR, noisy image H HR, is down- sampled to generate LRa, which is used to calculate the pixel and feature losses. However, the noise distribution of LRn, is not consistent with HR, after the down-sampling Therefore, Generative Adversarial Network (GAN) is used to further learn the noise distribution mobile images. The loss function and training strategy are introduced follows.
Loss function: Pixel loss, feature loss (i.e. perceptual loss) and GAN loss are used to train $ H2L-
loss H2L a (2)) + (2oosss + as clooss H2L
GAN
(2)
where a1, a2 and azy are weighting parameters.
Pixel loss loss H2L aims at pixel-wisely comparing the image difference between LRn and the net- work output LR, as
1 FLR (ILLR(K) - LR,, (k))
loss 1121.
(3)
W k=1
where nhe corresponds to the number of pixels in LR.
Feature loss aims at comparing the difference between high-level features of two images, which has been proved to effective in many previous works Ledig al. (2017); Wang et al. (2018b); He et al. (2016); Johnson et al. (2016). Inspired by these works, VGG /Simonyan & Zisserman (2014) is utilized as feature extraction network yp:
AE Σ (LR())))( ((LR))(+)111) L 1=1
Toss H2L
Where ((LR) is the extracted high-level features of input image LR by sp. k represents the index in the extracted features. GAN is then used to learn the distribution by discriminating unpaired images, The and I. R.,, where The is the low-resolution image randomly cropped from T..
GAN
loss 121.
GAN
Σ
llg (H
(5)
GAN
1
n=1
5

---

Under review as a conference paper at ICLR 2020
LR where GAN is the number of batch size, Dø, is the discriminative model with parameters 82. The
goal of Dø, to well distinguish the generated LR image and real image patch Inc. Dø, is trained iteratively with the generative model 112r. shown in Equ.
ming, maza, EllogDo, (Inc)]
(6)
Ellog(1 Da H2 (H (H N;0,)))
In this work, nearest neighbour is explored as the down-sampling method to better simulate the real-world ill-informative situation of mobile images.
3.3 LOW-TO-IIGHI NETWORK
Network LL2H is proposed to implement super resolution based the low-resolution image LR with parameter 0g as in Equation
HR L2H (LR; 0z)
3.3.1 NETWORK STRUCTURE
The structure of & L2H shown in Fig. 2 RRDB block from Wang et al. (2018b) used as the basic network unit. deeper structure with 25 RRDB blocks is proposed to model the more complex task. The basic unit not limited RRDB block, other basic block like residual block and dense block can also work in the proposed network structure.
3.3.2 TRAINING PROCEDURE OF O 12H
The training strategy of L2H is summarized in Fig. The loss function of $ L121 is similar to that of & H2L. which is described Section 3.2.2
GAN 10881 = (1os00 + ar5 (5(08/5(22)) + arg (loss L2H
(8)
where 04,, at and ag weighting parameters. The main difference of the training strategy between H22 and & L2H is training data. For + L2H two categories of training samples are utilized. The most important one is sam from dataset GMSR that simulates real-world distortions. Furthermore, the popular dataset DIV2K Timofte et al. (0177) is utilized to further increase the data diversity.
Finally, global fine-tuning is performed the whole network structure including 1 1221 and the discriminative networks, to jointly optimize the performance.
4 EXPERIMENTS
Eexperiments were conducted to evaluate the effectiveness of the whole proposed framework and its constituents including the H2L network and the training strategy. The parameters for training proposed framework are described as follows. Batch-size 16 with training patch of size 192 x 192.
Table 2: Quantitative evaluation of SR algorithms (PSNR/SSIM)
Method
Set5
Set14
Set pasr (test)
Bicubic
28.42/0.8104
26.00/0.7021 7027
35.85/0.9674
SRCNN
30.07/0.8627
27.50/0.7513
39.09/0.9716
SRGAN
29.40/0.8472
26.02/0.7397
38.43/0.9596
ESRGAN
30.55/0.8677
26.39/0.7246
38.55/0. 9572
Ours
30.37/0.8766
27.35/0.7767
39.93/097782
Following ESRGAN Wang et al. (2018b), the learning rate is initialized with x 10 10-4, and halved at [50k, 100k, 200k, 300k] iterations. Adam & Ba (2014) optimizer is utilized with 3n 0.9,
32 = 0.999 without weight decay.
4.1 SUPER RESOLUTION RESULTS
Quantitative and Subjective experiments were conducted evaluate the performance of the pro- posed method. Table shows the quantitative evaluation results of the popular and state-of-the-an GAN-based super resolution methods with an upscale factor of

---

Under review as a conference paper at ICLR 2020
43340
(3566099713
LR,, example from datasel
SRGAN (B8 350,897)
ESRGAN
OutS
CTA (1NA510,599)
UM 961-99777
Bicabic (37 3770000000 skag
(PSNRISSIM)
(47 sWi))
champple from Iaar
SRGAN
ESRGAN (770
OuT (01
Subjective comparison between the proposed method and some state-of-the-art methods. Left is the mobile low-resolution image example and detail patch. Right the super-resolution result of different methods and high-resolution ground-trute -ound (HR). (Best viewed in color)
Figure 4:
Both general datasets (set5 Bevilacqua et al. (2012), 14 Zeyde et al. (2010b)) and the
Table 3: Subjective evaluation on real mobile images
Mobile
Iphone7p
Xiaomi
IphoneXs
All
mobile-specific dataset (1 (porr) are utilized. PSNR and SSIM [Wang et al. (2004) are adopted for measuring the effectiveness of the proposed method, where higher value in-
Bicubic
1.78
1.35
1.69
1.61
ESRGAN
1.83
2.31
1.98
2.04
Ours
2.39
2.34
2.32
2.35
dicates better performance. From Table [] shown that the proposed method outperforms the other state-of-the-art methods in terms of SSIM in all circumstances, demonstrating its effective- ness on reconstructing structure. Especially, the improvements achieved on Set Tpair are significant, showing its good performance on mobile images. The best performing results are shown in bold. Subjective comparison examples were presented as shown in Fig. 4 Real-world mobile images are utilized. PSNR and SSIM results are presented the bottom for reference, which evaluated on Y channel. In many situations, mobile images suffer from noise and detail missing. Most methods cannot handle these cases well, including bicubic, SRCNN Chao Dong (2016). SRGAN et al. (017), ESRGAN Wang et al. (2018b). The noise is still very obvious, and even increased in SR results. Instead, our method can deliver good results by removing the noise and well-reserve edges and details.
Moreover, order to verify the generalization ability of the proposed method, subjective evaluation experiments were conducted based on real phone-captured images. 24 images captured by three different mobile phones (Iphone7p. Xiaomi Mix2s and Iphone Xs Max) were used as inputs. We compared the performance of three SR methods: Bicubic, ESRGAN and the proposed SRDGAN. Totally 21 subjects without expert knowledge participated the experiment, who evaluated each up- sampled image subjectively using ranging from one three. Higher score represents better

---

Under review as a conference paper at ICLR 2020
visual quality. The experimental results are shown in Table 3 which confirms that in the real-world case, the proposed method is applicable, and it largely outperforms Bicubic and the seite-of-theaasd art method ESRGAN.
4.2 RESULTS OF 122
With PH2L. aim to simulate LR images that is a better ensemble of real mobile images and to generate large-scale dataset. To be fair, a train-
Table 4:
Quantitative evaluation of SRa and ours algoribmn(PPSNR/SSMM)
Method
Sot5
Set14
Set (pair(test)
19.72/0.8404 Ours 30.37/0.8766
SRa
27.05/0.7357
39 99.0440.9736
ing dataset that pre-processed with
27.35/0.7767
39.93/0.9782
artificial degradation methods is gen-
erated and serves as baseline. Specifically, high-quality HR images of GMSR dataset and DIV2K are downsampled with bicubic interpolation, Gaussian blurred with random intensity to simulate out-of-focur converted into Bayer data, added with Gaussian noise, and finally debayer-filtred to generate the LR images. We used this baseline dataset to train the proposed SR model and named the new super resolution model SRa. Quantitative results are shown Table Our model trained on GMSR outperforms SR in all cases, demonstrating the effectiveness of the proposed H2L network and the GMSR dataset.
4.3 EXPLORING TRAINING STRATEGIES FOR L22H
Experiments were conducted to evaluate the effectiveness of the new dataset, new training strategies and the proposed model structure, shown in Fig. [5] 'ESRGAN" represents the original ESRGAN method with GAN-based structure and DIV2 TTmofte et al. (2017) (bicubic downsampling) as training data.
PSNR
SSIM
"ESRGAN" represents the new-trained method with the same model structure but new datasets for training, includ- ing DIV2K Timofte al. (2017) (Near- est neighbour downsampling) and GMSR. 'Ours' the proposed method with the new model structure and new datasets for training. Comparison between 'ES- RGAN' and 'ESRGAN" can evaluate the effectiveness of new training datasets. While performance comparison between 'ESRGAN# and 'Ours' evaluates the im- pacts in the model structure.
3855
1842
39.4
1.1
0.5572
0,0761
087822
0.9
0761
07707
6.7216
0.6
Set14
Set /ptl
Se114
Set fpair
ESRGAN
ESRGAN
ows
ESRGAN
Figure 5: Comparing ESRGAN with the proposed method
PSNR and SSIM Wang et al. (2004) are utilized, where larger value represents better performance. The test was conducted on both general dataset 4 4 and the mobile image dataset Set pair- It can be seen that the 'ESRGAN" outperforms 'ESRGAN' by changing training data. It proves that the new dataset(GMMRR and training strategy can be easily generalized to other model structures and improve the performance on both general data and mobile data. Moreover, 'Ours" further improves the performance compared with "ESRGAN*" It proves the effectiveness of the model structure in L22H-
5 CONCLUSION
In this work, novel framework SRDGAN is proposed to solve the noise prior super resolution problem with dual generative adversarial network. general degradation network H2L is proposed and is able to learn the noise prior of real-world LR image. By using our proposed training strategy, the H2L network is trained to generate the 'realistic" LR image paired with the HR image, and A large-scale benchmark general mobile super resolution dataset, GMSR, is generated from it. The GMSR can help super resolution methods to be well applied in the real-world mobile taken images. Meanwhile, super resolution network L2H is proposed, which contains structure and training strategics. Especially, these training strategies are proved to be effective for other SR model, such ESRGAN, and improve the SR performance.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Namhyuk Ahn, Byungkon Kang, and Kyung-Ah Sohn. Fast, accurate, and, lightweight super- resolution with cascading residual network. Proc. ECCV, 2018.
Jan Allebach and Ping Wah Wong. Edge-directed interpolation. Proc. ICIP, volume 3, pp. 707- 710. IEEE, 1996.
Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie Line Alberi-Morel. Low- complexity single-image super-resolution based on nonnegative neighbor embedding. 2012.
Adrian Bulat, Jing Yang, and Georgios Tzimiropoulos. To learn image super-resolution, use a gan to learn how to do image degradation first. In Proc. ECCV, pp. 185 185-200, 2018.
Hong Chang, Dit- Yan Yeung, and Yimin Xiong. Super-resolution through neighbor embedding. In Proc. CVPR, pp. 275-282. IEEE, 2004.
Xiaoou Tang Chao Dong, Chen Change Loy. Accelerating the super-resolution convolutional neural network. Proc. ECCV, 2016.
Chao Dong, Chen Change Loy, and Xiaoou Tang. Accelerating the super-resolution convolutional neural network. In Pruc. ECCV, pp. 391-407. Springer, 2016.
Claude E Duchon. Lanczos filtering in one and two dimensions. Journal of applied meteorology, 18(8):1016-1022 1979.
Gilad Freedman and Raanan Fattal. Image and video upscaling from local self-examples. ACM Transactions on Graphics (TOG), 30(2):12, 2011.
Xinbo Gao, Kaibing Zhang, Dacheng Tao, and Xuelong Li. Joint learning for single-image super- resolution via a coupled constraint. IEEE Trans. Image Processing, 21(():469 480, 2012.
Daniel Glasner, Shai Bagon, and Michal Irani. Super-resolution from a single image. In Proc. ICCV, pp. 349-356. IEEE, 2009.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Proc. NeurlPS, pp. 2672- 2680, 2014.
Jingwei Guan and Wai-Kuen Cham. Quality estimation based multi-focus image fusion. 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1987- 1991. IEEE, 2017.
Jingwei Guan, Shuai Yi, Xingyu Zeng. Wai-Kuen Cham, and Xiaogang Wang. Visual importance and distortion guided deep image quality assessment framework. IEEE Transactions on Multime- 25200, 2017.
dia,
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. Proc. CVPR, June 2016.
Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super-resolution from trans- formed self-exemplars. In Proc. CVPR. pp. 5197-5206, 2015.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In Proc. ECCV, pp. 694-711. Springer, 2016.
Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Accurate image super-resolution using very deep convolutional networks. In Proc. CVPR, pp. 1646-1654, 2016a.
Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Deeply convolutional network for image super-resolution. In Proc. CVPR, pp. 1637-1645, 2016b.
Kwang In Kim and Younghee Kwon. Single-image super-resolution using sparse regression and natural image prior. IEEE transactions on pattern analysis and machine intelligence. 32(6): 1127 1133, 2010.

---

Under review as a conference paper at ICLR 2020
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang. Deep laplacian pyramid networks for fast and accurate super-resolution. In Proc. CVPR, pp. 624 632, 2017.
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic single image super-resolution using a generative adversarial network. In Proc. CVPR, pp. 4681-4690, 2017.
Xin Li and Michael T Orchard. New edge-directed interpolation. IEEE Trans. Image Processing, 10(10)15211-157, 2001.
Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep resid- ual networks for single image super-resolution. In Proc. CVPR Workshops, pp. 136-144, 2017.
Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using efficient sub-pixel convolutional neural network. In Proc. CVPR, pp. 1883-1883, 2016.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409, 1556, 2014.
Radu Timofte, Vincent De Smet, and Lue Van Gool. Anchored neighborhood regression for fast example-based super-resolution. In Proc. CVPR, pp. 1920-1927, 2013.
Radu Timofte, Vincent De Smet, and Lue Van Gool. A+: Adjusted anchored neighborhood regres- sion for fast super-resolution. In Asian conference computer vision, pp. 111-126. Springer, 2014.
Radu Timofte, Eirikur Agustsson, Lue Van Gool, Ming- Yang, and Lei Zhang. Ntire 2017 challenge single image super-resolution: Methods and results. In Proc. CVPR Workshops, pp- 114-125, 2017.
Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy. Recovering realistic texture in image super-resolution by deep spatial feature transform. In Proc. CVPR, pp. 606-615, 2018a.
Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. Esrgan: Enhanced super-resolution generative adversarial networks. In Proc. ECCV, pp. 63-79. Springer, 2018b.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. JEEE Trans. Image Processing, 13(4):600-612, 2004.
Jianchao Yang. John Wright, Thomas S Huang, and Yi Ma. Image super-resolution via sparse representation. IEEE Trans. Image Processing, 19(1 (1111):261128777 2010.
Yuan Yuan, Siyuan Liu, Jiawei Zhang, Yongbing Zhang, Chao Dong, and Liang Lin. Unsuper- vised image supor-resolution using cycle-in-cycle generative adversarial networks. In Proc. CVPR Workshops, pp. 701-710, 2018.
Roman Zeyde, Michael Elad, and Matan Protter. On single image scale-up using sparse- representations. In International conference on curves and surfaces, pp. 711-730. Springer, 2010a.
Roman Zeyde, Michael Elad, and Matan Protter. On single image scale-up using sparse- representations. In International conference on curves and surfaces, pp. 711-730. Springer, 2010b.
Lei Zhang and Xiaolin Wu. An edge-guided image interpolation algorithm via directional filtering and data fusion. IEEE Trans. Image Processing, 15(8):2226-2335 2006.
Wei Zhang, Youmeii Zhang, Lin Ma, Jingwei Guan, and Shijie Gong. Multimodal learning for facial expression recognition. Pattern Recognition, 48( (0100)3311-30000 2015.
10

---

Under review as a conference paper at ICLR 2020
Wei Zhang, Chenfei Qu, Lin Ma, Jingwei Guan, and Rui Huang. Learning structure of stereoscopic image for no-reference quality assessment with convolutional neural network. Pattern Recogni- tion, 59:176-187, 2016.
Xuaner Zhang, Qifeng Chen, Ren Ng, and Vladlen Koltun. Zoom learn, learn zoom. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3762- 3770, 2019.
Tianyu Zhao, Changqing Zhang, Wenqi Ren, Dongwei Ren, and Qinghua Hu. Unsupervised degra- dation learning for single image super-resolution. CoRR,
2018.


Reviews:
[{"review_id": "SkgLCEuTKB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes a framework for real world super resolution, which includes two networks: high-to-low and low-to-high. With high-to-low network, a general mobile super resolution dataset is proposed. According to the proposed framework and dataset, promising results are achieved for realistic image super-resolution.\n\n[Strengths]\n- This paper is well-written and easy to follow.\n- A new dataset is proposed for real world image super-resolution task.\n\n[Weaknesses]\n- The novelty of this work is relatively limited. The idea to use a high-to-low network to model distribution of real-world low resolution images has already been investigated in several works, like Bulat et al. (2018) and Zhao et al. (2018). However, no comparisons are conducted with these two works. Besides, the network structures are very similar to SRGAN and ESRGAN, which again shows little novelty.\n\n- The authors claimed the recent SOTA optical based method from Zhang et al. (2019) suffers from many problems. However, no direct comparisons are conducted to verify their viewpoints, and it's difficult to distinguish which one is better.\n\n- Is there any visualization for the distributions of real work noisy images and the learned low resolution images? Besieds, since only one mobile device is used to capture the images, why does the proposed method generalize well on other mobile phones? Any distribution illustrations?", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HkevFaACKB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a single image super-resolution method. The motivation is real and practical.  In almost all existing methods the low-resolution training images are artificially synthesized using the available high-resolution images with bicubic downsampling, which allows LR images to carry more information than real demosaic-upscaled LR images. This mismatch between training and realistic\ndata hinders the practicability of such solutions (not limited to mobile scenarios). \n\nYet, the proposed remedy in the paper is not an answer either. It simply adds a noise image to the HR image, passes it through a convolutional network to construct an LR version, which is then fed into a deconvolutional network. Both these networks are build by residual blocks. Even worse, the paper simply uses zero-mean Gaussian random noise with a standard deviation of 0.05 to simulate the distortions. This is myopic and questionable. \n\nThe introduced dataset is very small, it contains only 447 pairs of images, with an additional 206 images as reference images. These pairs consist of a normal image (considered to be noisy) and a computed image by aggregating 20 images (considered to be clean) using a Blackberry Key2. In other words, the dataset only models a single device and a single camera lens. For practical purposes, this is not much different than using a bicubic sampling and adding noise drawn from a fixed distribution.  \n\nThe paper fails to provide comparisons with the top methods in the NTIRE 2019 single image super-resolution challenge leaderboard. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BygtgrvnKS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In the paper, the authors proposed an end-to-end single image super-resolution framework, which is composed of two parts. First, a high-to-low network is trained to generate realistic HR/LR image pairs for training super-resolution models. Using this network, the authors designed a large-scale General Mobile Super Resolution Dataset, GMSR, which can be utilized for training or as a benchmark for super-resolution methods. Second, a low-to-high network is trained to produce a super-resolution image. Beneﬁting from the GMSR dataset and novel training strategies, the proposed low-to-high network could thus deal with the real distortions and restore ﬁne details.\n\nThe paper is easy to follow. The proposed framework is logical and the experiments proved to be effective. However, there are some problems as follows:\n\n1. In the high-to-low network of framework, Gaussian random noise is added to clean high-resolution images to simulate the randomness of the distortions in LR. Why can do this in this way? The rationality of this setting should be given. Are there other kinds of noises be applied? As far as I knew, the in-camera processing pipeline will involve  Poisson-Gaussian noise rather than Gaussian only and other factors such as demosaicing, Gamma correction, and JPEG compression. It is hard to only consider Gaussian noise to stimulate the whole in-camera process. \n\n2. The experimental part is remarkable insufficient to an ICLR paper. There only have a quantitative evaluation of the proposed framework with other super-resolution algorithms, while lacking the qualitative evaluation of the framework itself. \n\n3. There are some formatting issues in the paper,\n   1. Page 8, figure 5, here some errors in the label color of the figure.\n   2. Page 1, introduction, line 14, \"However, In most practical...\" -> \"However, in most practical...\";\n   3. Page 2, line 10, \"...many problems, such as..., which is hard to avoid\" -> \"...many problems, such as..., which are hard to avoid\";\n   4. The last paragraph of the introduction, line 6, \"...parameters.Third,...\" -> \"...parameters. Third,...\"\n      ...", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agree that the authors have done a great job identifying weaknesses with the current SOTA in super-resolution.   However, there is also agreement that the proposed approach may be too simple to accurately capture a range of real camera distortions, and more comparisons to the SOTA are needed.   While this paper certainly has merits and opens the door for strong work in the future, there is not enough support to accept the paper in its current form.
Decision: Reject