Title: One-to-many Approach for Improving Super-Resolution
Authors: Sieun Park
Caleb Eunho Lee
Venue: NeurIPS 2021
Keywords: Super resolution
Deep learning

Abstract: Super-resolution (SR) is a one-to-many task with multiple possible solutions. However, previous works were not concerned about this characteristic. For a one-to-many pipeline, the generator should be able to generate multiple estimates of the reconstruction, and not be penalized for generating similar and equally realistic images. To achieve this, we propose adding weighted pixel-wise noise after every Residual-in-Residual Dense Block (RRDB) to enable the generator to generate various images. We modify the strict content loss to not penalize the stochastic variation in reconstructed images as long as it has consistent content. Additionally, we observe that there are out-of-focus regions in the DIV2K, DIV8K datasets that provide unhelpful guidelines. We filter blurry regions in the training data using the method of [10]. Finally, we modify the discriminator to receive the low-resolution image as a reference image along with the target image to provide better feedback to the generator. Using our proposed methods, we were able to improve the performance of ESRGAN in x4 perceptual SR and achieve the state-of-the-art LPIPS score in x16 perceptual extreme SR.

TL;DR: We propose a piepline capable of generating multiple estimates of the reconstruction, and a loss not penalizing similar but equally realistic images for a one-to-many approach for super-resolution.

Review discussion:
Pros:
- Noise generator to address one-to-many problem.
- New content loss not to penalize high-frequency gradients.
- Using laplacian to select non-blur area for training.
- Better subjective and objective quality than ESRGAN
Cons:
- Limited comparisons to state-of-the-arts, such as SRFlow and its improvments.
- Blured images compared with ESRGAN.
No
Thank you for providing us constructive feedback.
Regarding the quote "Limited comparisons to state-of-the-arts, such as SRFlow[2] and its improvements.", we agree that our paper lacked comparisons with some key papers, including SRFlow mentioned by reviewers including you. We will upload an updated version of the paper regarding the comparisons with other S.O.T.A SR.
Regarding the quote "Blurred images compared with ESRGAN.", we understood the sentence as that we didn't compare the effects of filtering blurred images on ESRGAN in Table 1. The results without blur detection are denoted as 'ESRGAN (Official)' and all other experiments were done with the filtering algorithm. Especially, the experiment on config (c) only modified the blur detection algorithm from the original ESRGAN. We agree that there was potential confusion in the layout, and have revised it.
Regarding the comments related to "adding the proposed techniques to other GAN-based methods", GAN-based methods in super-resolution are not diverse. They stem from the work of SRGAN and typically use a combination of content, perceptual(VGG score), and GAN loss as the training objective of the generator. Most modifications are made to the network architecture and modifications to the training objectives are minor. Recent works on GAN-based SR is typically based on the ESRGAN pipeline, and we believe that the effects of the techniques in this paper can sufficiently generalize to other GAN-based methods in SR. The results of the experiments on x16 SR using RFB-ESRGAN can be an example.
Also, we note that SRFlow uses a very different approach from GAN-based methods and most of our proposed techniques are inapplicable to SRFlow.
Regarding the quote "The new content cycle loss is interesting but up-sampling and down-sampling may induce some information loss in generation and discriminator.", this 'weak-content loss' is actually the basis of our work. We supposed that the generation of high-frequency, photo-realistic details should be done purely on the perceptual and GAN loss. The role of the content loss is to simply make sure that the reconstructed image is consistent with the original image, and thus won't interfere with other objectives being 'creative' on generating pixel-level differences. We believe that our proposed content loss based on the cycle-consistency loss is the optimal form for ensuring consistency while not interfering with other perceptual objectives.
In short, our work claims that the loss of information in the content loss is actually the key to relying more on perceptual objectives. We also added some clarification on the paper on why we adopted a weaker content loss.
Thank you again for your generous feedback.
[1] Wang, X., Yu, K., Wu, S., Gu, J., Liu, Y., Dong, C., ... & Change Loy, C. (2018). Esrgan: Enhanced super-resolution generative adversarial networks. In Proceedings of the European conference on computer vision (ECCV) workshops (pp. 0-0).
[2] Lugmayr, A., Danelljan, M., Van Gool, L., & Timofte, R. (2020, August). Srflow: Learning the super-resolution space with normalizing flow. In European Conference on Computer Vision (pp. 715-732). Springer, Cham.
Strengths
1. The idea of the paper is clearly written and empirical results demonstrate its effectiveness.
2. The paper presented many interesting ideas, e.g., soft content loss, new data augmentation with blur detection, one-to-many generator, and conduct extensive ablation study for each of the terms.
3. There are additional results in Table 2 of the paper that validate the key idea for x16 SR.
Weaknesses
1. The authors cannot claim this paper as the first to consider SR as a one-to-many problem.
Actually many previous papers have already model SR in a one-to-many way.
For example, [1] gave a solid theoretical explaination of SR problems and give an effective solution. And GAN based SR methods, use discriminator loss for the same reason.
[1] Amortised MAP Inference for Image Super-resolution
2. The motivation of adding noise to intermediate layers seems lacking theoretical support.
Indeed, adding noise can generate various outputs.
However, candidate multiple HR images are not just random. They lie in some `null` space  of the downsample operator. And adding random noise cannot ensure to generate proper HR results.
3. The major weakness is the novelty of the proposed method, although the ablation studies validate the effectives of each technique on several datasets. The novelty is limited, and there are few new techniques for me. In my view, the proposed method seems to stack many tricks for improving the performances, such as a new data augmentation and adding the pixel-wise noise into the extracted features.
4. Table 1, The comparison to STDA methods do not include the recent state of the art methods like " Enhancing Perceptual Loss with Adversarial Feature
5. Matching for Super-Resolution", IJCNN’20; “Journey Towards Tiny Perceptual
6. Super-Resolution”, ECCV’20; “Generative adversarial network-based image super-resolution using perceptual content losses”, ECCV’18 (and many others).
7. The manuscript is filled with informal words and needs a rewrite.
28: don’t -> do not
98: “->” -> “\rightarrow”
125: won’t -> will not
224: don’t -> do not
241: “x16 “ -> “\times 16”
No
Thank you for your constructive feedback.
After reviewing more references suggested by reviewer 44pW and the SRFlow paper pointed out by other reviewers, we decided to change the objective of our paper from "proposing a one-to-many approach for SR" to "adapting the concepts of SRFlow to improve GAN-based SR by properly implementing the one-to-many property" since our method did improve the perceptual quality of GAN-based SR methods(ESRGAN, RFB-ESRGAN).
Although our work doesn't seem to meet the standards of the submitted to NeuralIPS, we thank all reviewers for providing great constructive feedback and enabled us to improve our work and learn.
# Originality
Other recent works have noted that there should be multiple solutions to the SISR problem and developed their own solutions [A]. These methods are not mentioned nor compared against.
The others innovations in this work don't seem particularly novel.
[A] Lugmayr, Andreas, et al. "Srflow: Learning the super-resolution space with normalizing flow." European Conference on Computer Vision. Springer, Cham, 2020.
# Quality
The paper provides few high quality images and those that are included aren't presented clearly: Is Figure 4 demonstrating 4x or 16x superresoltuion results? I think 16x because both methods have major artifacts.
The comparisons are each limited to a single method that is no longer SOTA. See [A].
Much of the paper is motivated by the idea the one LR image should produce multiple plausible HR images.
However, the main text provides no examples of this occuring.
User studies are preferable to relying on LPIPS alone.
# Clarity
The authors motivate the L1 penalty in the LR domain as part of a cycle consistency loss. Howevever, cycle consistency seems fundamentally at odds with the goals of the paper; generating a diverse set of HR outputs form a LR input. A true cycle consistent loss would ensure that the LR image formed by a single HR image would always produce that same HR image.
# Significance
The paper set out to develop a one-to-many approach to super-resolution, which is arguably an important problem. Unfortunately, there's little evidence this was accomplished.
No
After reviewing more references including the SRFlow paper pointed out by reviewer rx3L and other reviewers, we decided to change the objective of our paper from "proposing a one-to-many approach for SR" to "adapting the concepts of SRFlow to improve GAN-based SR by properly implementing the one-to-many property" since our method did improve the perceptual quality of GAN-based SR methods(ESRGAN, RFB-ESRGAN).
We did want to point out that our work does share the same conceptual ideas with SRFLow, but is practically very different from SRFlow. There seem to be misunderstandings about our cycle-consistency loss. Conceptually, we train the generator to generate high-frequency details with the GAN and perceptual loss, while our content loss simply ensures the reconstructed image to be consistent with the original image. Our content loss will penalize the various images while the semantic content is consistent with the low-resolution image.
Also, with regard to your comments related to user studies, the 2020 NTIRE challenges suggest that LPIPS scores are highly consistent with user studies on evaluating the perceptual quality of super-resolved images.
Although our work doesn't seem to meet the standards of the submitted to NeuralIPS, we thank all reviewers for providing great constructive feedback and enabled us to improve our work and learn. We had a great experience throughout the review process and are hoping to participate in later NeurlIPS conferences with a better paper. We thank reviewers again for constructive feedback.

Meta-review:
role: PC
data: comment: The paper proposes a one-to-many approach to super-resolution. All reviewers and the AC agree that this is an important problem and a very interesting research direction. 

The original manuscript missed several important prior work (as listed by Reviewers rx3L and 44pW), which also treat SR as a one-to-many problem. This significantly reduced the novelty aspect of the work. In their rebuttal the authors acknowledged these similarities and proposed to change the objective of work to "adapting the concepts of SRFlow to improve GAN-based SR by properly implementing the one-to-many property". The AC agrees with the authors that the paper does improve performance and make interesting contributions (see below). However, incorporating these changes, adding new baselines and reformulating the work would certainly require another round of revisions.

The paper makes several practical contributions (soft content loss, pixel-wise noise after each RRDB block, filter blurry regions using Laplacian activation) that seem to work well in practice. This was highlighted by Reviewer 44pW.

Reviewer rx3L points out that the paper does not clearly show the ability of the model of generating multiple HR images from a given LR image. The AC agrees that this would strengthen the work significantly.

The AC agrees with the concerns raised by the reviewers and finds no basis for overruling their recommendation. The AC appreciates the constructive rebuttal and encourages the authors to incorporate the reviewers feedback and resubmit to a future conference.


Decision: Reject