AND: Adversarial Neural Degradation for Learning Blind Image Super-Resolution
Authors: Fangzhou Luo
Xiaolin Wu
Yanhui Guo
Venue: NeurIPS 2023 poster (2023)
Keywords: Blind Image Super-Resolution

Abstract: Learnt deep neural networks for image super-resolution fail easily if the assumed degradation model in training mismatches that of the real degradation source at the inference stage. Instead of attempting to exhaust all degradation variants in simulation, which is unwieldy and impractical, we propose a novel adversarial neural degradation (AND) model that can, when trained in conjunction with a deep restoration neural network under a minmax criterion, generate a wide range of highly nonlinear complex degradation effects without any explicit supervision. The AND model has a unique advantage over the current state of the art in that it can generalize much better to unseen degradation variants and hence deliver significantly improved restoration performance on real-world images.

AND: Adversarial Neural Degradation for Learning
Blind Image Super-Resolution
Fangzhou Luo McMaster University luofl @eemamtte......
Xiaolin Wu"
Yanhui Guo McMaster University guoy mnoyy13@memaster.co
McMaster University xwußece, www.Bece.ncasste.co
Abstract
Learnt deep neural networks for image super-resolution fail easily if the assumed degradation model in training mismatches that of the real degradation source at the inference stage. Instead of attempting exhaust all degradation variants in simulation, which is unwieldy and impractical, we propose a novel adversarial neural degradation (AND) model that can, when trained in conjunction with deep restoration neural network under a minmax criterion, generate a wide range of highly nonlinear complex degradation effects without any explicit supervision. The AND model has a unique advantage over the current state of the art in that it can generalize much better to unseen degradation variants and hence deliver significantly improved restoration performance on real-world images.
Introduction
Deep learning has made great strides in the applications of image restoration. It has demonstrated superior performances over traditional methods on almost all common image restoration tasks, including super-resolution [ denoising [52], compression artifacts removal (6. deblurring [38], etc. But the margin of performance gains made by deep learning methods of image restoration decreases sharply if the degradation processes assumed in training mismatch those of the real world images inference stage 33 well known that, for any real-world problems, the efficacy of machine learning technique relies not only on the design of the technique itself but also, sometimes even more critically, on the statistical agreement between the training and test data 42
In reality, it either intractable or highly expensive to obtain both degraded images and the corre- sponding latent images (ground truth). The most common practice in literature is model to generate paired degraded and ground truth images for training the restoration network-
degradation
26 24 This synthesis approach cannot accurately simulate the realistic digital imaging pipeline that is affected by multiple complex and compounded degradation sources; for instances, insufficient sampling rate, color demosaicing errors, sensor noises, camera jitters, compression distortions and etc. In this paper, we focus on the task of super-resolution, namely assuming that the dominant degradation cause is insufficient sampling rate, which is compounded by other degradation sources in the imaging pipeline. The said complex nonlinear phenomena often defy explicit analytical modeling. brute force approach may be build multiple simpler parametric degradation models, one for each type of degradation (c.g., downsampling, noises, compression, motion, etc.) and apply them in different combinations, orders and parameter setting to generate training data, in hope to simulate as wide a range of degradations encountered in practice possible. This amounts, however, fighting a losing battle because it impossible exhaust degradation variants, many of which are not even known understood.
51 4
Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

observation 1:
observation 2:
degradation -- CNN layer
lim degradation idantity
convolution layer
noise
pooling layer
identity
activation layer
JPEG
noise injection
layer
Figure 1: We observe two properties in most image degradations. Firstly, almost all types of image degradation could find a corresponding operation a standard convolutional neural network. Secondly, almost all moderate image degradations could be considered small deviations from the identity transformation. The neural degradation prior proposed for our real-world super-resolution method is inspired by these observations.
This work represents a fundamental departure from the current ways of coping with mismatches between the simulated training and real-world image data. Instead of attempting exhaust all degradation types simulation, we propose a novel adversarial neural degradation (AND) model that can, when trained in conjunction with deep restoration neural network under minmax criterion, generate a wide range of highly nonlinear complex degradation effects without any explicit supervision. Adversarial attack and defense (training) [12] 32] is proven learning strategy to vaccinate neural network models of signal classification against being misled by imperceptible disturbances in input signals. Previous research in adversarial learning for image restoration tasks [49 only borrows the earlier research in adversarial training for image classification tasks, but does not account for the differences between the classification and restoration tasks. The AND model, the main contribution of this paper, demonstrates for the first time how adversarial learning can effectively boost the robustness of deep networks for signal restoration. In particular, we adopt the minmax optimization criterion when training the AND model, aiming to withstand the attacks by the most difficult but nuance degradations that otherwise defy modeling. unique advantage over the current state of the art in being generic in terms of degradation types. It can generalize much better to unseen degradation types and variants and hence deliver significantly improved restoration performance on real-world images.
result, the AND model enjoys a
Our insight of the AND model comes from the following observations. We observe properties in most image degradations, as shown in Fig. Firstly, almost all types of image degradation could find corresponding operation in standard convolutional neural network. For example, blur and ringing could correspond to a convolution layer, downsampling could correspond to pooling layer, color fading and posterization 35 could correspond non-linear activation layer, sensor noise and film grain could correspond to a noise injection layer 19]. Secondly, most moderate image degradations could be considered small deviations from the identity transformation. For example, blur, noise and lossy image compression all obviously tend to the identity transformation pointwise as the degradation level approaches the slightest level. As the degradation level get higher, those degradations gradually deviate from the identity transformation. Inspired by the two observations of image degradations, we initialize untrained convolutional neural networks the identity transformation, make parameters of these networks slightly deviated from the start, and take them as prior for various real-world image degradations. When we train a SR model with HR and LR image pairs constructed by the proposed degradation prior, we adversarially search small deviations which could make the SR model perform the worst, and optimize the SR model based on worst degradation case to achieve good lower performance bound for various real- world image degradations.

---

2
Related Work
Single Image Super-Resolution The first convolutional neural network for single image super- resolution is proposed by Dong et al. [7] called SRCNN, and achieved superior performance against previous works. Since that the field has witnessed variety of developments. Shi et al. [36] firstly proposed real-time super-resolution algorithm ESPCN by proposing the sub-pixel convolution layer. Lim et al. [27] removed batch normalization layers in the residual blocks, and greatly improved the SR effect. Zhang et al. [56] introduced the residual channel attention to the SR framework. To achieve photo-realistic results with detailed textures, Ledig et al. 22 introduced the generative adversarial network [TT] into the SR framework, and employed it as loss supervisions to push the SR solutions closer to the natural manifold. Wang et al. [44] later improved the GAN based SR method, and achieved better SR visual quality with more realistic and natural textures.
Blind Image Resolu-esolution. The field is also named as real-world image super-resolution. Different from the classical SR field which assumes that the image degradation model is an ideal bicubic downsampling, the blind SR field aims to solve SR problems with unknown degradation. Researchers tried to solve the problem by implicitly or explicitly estimating the degradation model. Gu et al. [13 proposed a method to iteratively estimate the blur kernel. Kligler et al. 2 introduced KemnelGAN, which trains solely on the LR test image at test time, and learns its internal distribution of patches. Researchers also built complex models for image degradation, augment the robustness of the SR model. Zhang et al. [51] designed a complex degradation model that consists of randomly shuffled blur, downsampling and noise degradations. Wang et al. 43 used high-order degradation model to better simulate complex real-world degradations.
Zhang et al. [55] indicated that when the degradation distribution during both training and testing perfectly matches, the SR model exhibits favorable generalization and achieves high performance simultaneously. The binning method is utilized to adjust the joint distribution of the three parameters of a widely-used image degradation model, helping the training degradation distribution to better match the testing distribution. Both their work and our work aim to align degradation distributions to achieve good performance and generalization. However, they concentrate on aligning the distributions of degradation parameters within given model, while we focus developing more suitable family of degradation functions.
Adversarial Training. Adversarial training improves the model robustness by training on adversarial examples generated by gradient-based method [12] Madry et al. [32] studied the adversarial robustness of neural networks through the lens of robust optimization. Tramer et al. [41] proposed an
ensemble adversarial training on adversarial examples generated from a number of pretrained models. Kolter and Wong (47) developed a provable robust model that minimizes worst-case loss over convex outer region. Athalye et al. [ demonstrated that adversarial training on PGD adversarial examples was to be the state-of-of-art defense model. Yue et al. [49] and Castillo et al. [4] both used adversarial examples during training to improve SR model's 's capacity to process noisy inputs, which is different with our method because they only considered adversarial noise rather than general real-world degradations.
Domain Generalization. by using only source data for model learning. Most existing approaches belong to the category of domain alignment [44, where the central idea is to minimize the difference among source domains for learning domain-invariant representations. Meta-learning (9) are also used to solve domain generalization by exposing model to domain shift during training with a hope that the model can better deal with domain shift in unseen domains. In the context of domain generalization, the work most related to ours is RandConv [48). It is based on the idea of using randomly initialized, single-layer convolutional neural network to transform the input images to novel domains. Since the weights are randomly sampled from Gaussian distribution at each iteration and no learning is performed, the transformed images mainly contain random color distortions, which do not contain meaningful variations and are best to be mixed with the original images before passing to the task network.
Domain generalization aims to achieve -iittii-iitiittioonn generalization
Identity Mapping in Deep Learning. Identity mappings are widely used in deep learning methods, typically as network layers rather than degradation priors. Sun et al. [14 used identity mappings as the skip connections and after-addition activation, to make the training easier and improves
3

---

HR put
degradation network
step 2:
LeakyReLU
adversarial
noise injection
adversarial LR
degradation
perturbation
degradation
basic block
restoration network
avg pool
degradation parameters
degradation paramnleers
basic block
SR output
basic block
1: iddenitty degradation initialization
step
restoration
discriminator network
ideal LR
discriminator
training
Figure 2: Illustration of the training procedure of our real-world super-resolution method with the proposed adversarial neural degradation model. Every single optimization step of the whole network can be divided into four sub-steps, and we highlight the internal state of the degradation network in the first two sub-steps.
model generalization. Zhang al. (0 used identity mapping task study memorization and generalization of overparameterized networks the extreme cases.
3 Adversarial Neural Degradation for Blind Super-Resolution
Before discussing our new robust real-world SR method, we would like to emphasize once again that the degradation prior employed in our approach is inspired by the following two observations of image degradations.
1. Almost all types of image degradation could find corresponding operation in standard convolutional neural network.
2. Almost all moderate image degradations could be considered as small deviations from the identity transformation.
Once we identify the commonalities among various image degradations, we naturally propose a simple and elegant degradation prior that encompasses all these degradations. That is, we take slightly deviated identity convolutional neural networks prior for various real-world image degradations.
We illustrate the entire training procedure of our SR method with the proposed degradation prior Fig. The entire neural network can be divided into three parts: degradation network, restoration network, and an optional discriminator network. A single optimization step of the entire network can further divided into the following four sub-steps. First, we initialize the degradation network to the identity transformation. Next, we adversarially search for a small deviation of the initialized degradation network that would cause the restoration network to perform the worst. Then, we optimize the restoration network based on the identified degradation case. Finally, we upgrade the discriminator network to distinguish restoration outputs from real images. We repeat the optimization steps of the entire network multiple times during training. Once the training is complete, we discard the degradation network and the discriminator network, and only use the trained restoration network for inference.
In the following subsections, we will first describe the degradation network architecture and the reason why it can function prior to incorporate various image degradations. Then, we will explain method used to initialize the degradation network to the identity transformation. Next, we will discuss the approach to perturb the degradation network, enabling it to represent complex image degradations. Subsequently, we will outline the adversarial training procedure of the SR model. Finally, we will analyze the training and inference efficiency of the method.

---

3.1 Degradation Network Architecture
Since we argue that almost all types of image degradation could find a corresponding operation in standard convolutional neural network, the degradation network does not need much design for its architecture to work as a prior to include various image degradations. We can simply concatenate common convolutional neural network layers which could represent these image degradations.
Convolution layer, which is the most common layer type, is used in the degradation network to represent filter related degradations. like blur and ringing. These degradation types are also very common in the real world. Blur can be caused by camera movement or out of focus, and ringing can be caused by image compression or image sharpening technique.
Non-linear activation layer is used the degradation network to represent global non-linear color changes, like color fading and posterization [35]. Color fading can be caused by inaccurate color response of old films, and posterization can be caused by color quantization in image compression.
Both convolution layer and activation layer can only represent spatially homogeneous image degrada- tions, and their abilitics are limited by the space invariant property of normal convolutional neural network. To represent spatially heterogeneous degradations like block artifacts in compressed images or dust spots in old images, we need to use relatively less common layer called noise injection layer [19], which adds noise to its input in pixel-wise manner. Combined with other layers, noise injection layer makes the degradation network able to represent complex spatially heterogeneous degradations.
Pooling layer is the last layer type we would like to discuss, and it can directly represent a downsam- pling process. We use anti-aliased average pooling layer [53 rather than a normal average pooling layer to avoid aliasing in the downsampling process.
For convolution, activation and noise layer, we would like to use multiple layers of the same type in the degradation network, make the network able to represent complicated and higher order degradations. But only one pooling layer is used in the degradation network, since it is hard to break one pooling layer with integer downsampling factor down into multiple ones with non- non-integer factor. We combine a 3 x 3 convolution layer, a LeakyReLU activation layer [31] and a noise layer to form our basic block, put 5 basic blocks before and after an average pooling layer, and put a 3 x convolution layer at the end. Number of channels in the degradation network are all 64, except for the input channels of the first convolution layer and the output channels of the last convolution layer, which are both 3 to take RGB images as input and output of the degradation network.
3.2 Identity Degradation Network Initialization
Due to overparameterization of neural networks, there are infinitely many parameter solutions to make a network represent the identity transformation, even the network architecture is fixed [50]. However, two different parameter solutions, which could identically represent the same function, may have totally different behaviors of functions in their own parameter neighbourhood. We take slightly deviated identity neural networks prior for various real-world image degradations. If all these networks are deviated from one or a few parameter solutions, their behaviors would be severely biased and cannot cover a variety of image degradations. Therefore, we need fast initialization method to generate a lot of identity neural networks with different parameters.
The most straightforward initialization method, which trains networks on the identity mapping task by minimizing error using gradient descent, is way too slow for our SR model training. We propose a fast method that can initialize the degradation network to the identity transformation. Our method only takes a few small matrix multiplications and one singular value decomposition, while the most straightforward initialization method takes millions of training steps.
Before discussing our method initialize the degradation network to the identity transformation, we would like to first clarify the meaning of the identity transformation in this work. For convolution, activation and noise layer in our degradation network, the definition of the identity transformation is strictly applicable, since the size of their input is the same as the size of their output. But for the pooling layer, the input feature is downsampled by a scale factor, so the strictly defined identity transformation no longer exists. In this work, we treat the ideal downsampling operation as a visually
5

---

identity transformation and use the anti-aliased average pooling layer [53] as an approximation of the ideal downsampling This interpretation is reasonable, because an image and its ideally downsampled counterpart are very similar from the perspective of the human visual system.
To make the degradation network represent the identity transformation, it must first be linear. So first we remove all nonlinearities in the network, by setting the negative slopes of all LeakyReLU activation layers [3]] to one. And we set all additive noises in all noise injection layers to zero. An output pixel of an identity network can only be affected by the counterpart pixel of the input. So then we only fill the center slices of all convolution kernels with nontrivial values, and set all other values of all convolution kernels to zero. By this way, we essentially simulate 1 x 1 convolutions with 3 x convolutions. We initialize the center slices of all but the last convolution kernels with Xavier Initialization [0] of 1 x 1 convolutional fan mode, to make all layers of the degradation network have a stable variance of responses. To finalize the identity network initialization, we merge all but the last 1 x 1 convolutions into one 1 x 1 convolution, by multiply center slices of convolution kernels as matrices. We compute the Moore-Penross pseudoinverse of the merged matrix, and fill the result into the center slice of the last 3 x 3 convolution kernel.
We did not mention the anti-aliased average pooling layer in our initialization method. That is because it is a strictly defined linear operator, and we already claim it as a visually identity transformation. As long as the remaining part of the network is a strictly defined identity transformation, the whole network would be a visually identity transformation, or in another word, would be the ideal downsampling.
3.3 Adversarial Degradation Perturbation
Once we initialize the degradation network to the identity transformation, we are at the starting point to various image degradations. The next thing we need to do is to perturb the identity degradation network, and the slightly deviated identity degradation network would represent a real-world image degradation case. We can use the network to quickly generate abundant perfectly aligned HR and LR image pairs, by taking HR images as input of the degradation network and collecting the outputs. And finally, we can train SR model with the collected HR and LR image pairs. Since the degradation prior includes various real-world image degradations, the SR model trained by this way could reconstruct various real-world degraded images well.
So how do we perturb the identity degradation network? The most straightforward way is add small random numbers to all parameters of the degradation network, and that means to take a small random step from the original identity transformation on the degradation space. A lot of degradation networks which are independently perturbed by this way would cover neighbourhood of the identity transformation on the degradation space. If we train a SR model with HR and LR image pairs constructed by many of those networks, the trained model would have good average performance on the covered degradation set.
However, instead of having a good average performance for regular degradations, we want our real-world SR model to be as robust as possible. In other word, we want our SR model have good worst-case performance. That is because the real-world situation is always more complicated than laboratory situations. We want our real-world SR model to keep having satisfactory performance, even for images might have been suffered from rare or unpredictable real-world degradations.
To achieve such goal. we perturb the identity degradation network adversarially instead of randomly. That means, we adversarially search small perturbations on all parameters of the degradation network, which could make the SR model perform the worst on HR and LR image pairs constructed by the degradation network. During SR model training, we keep searching those worst cases dynamically, and keep optimizing the SR model based on the worst degradation case for the moment. By this way, the worst-case performance of the SR model would be gradually improved, and will finally converge to the robust model with the highest lower bound.
'More formally, a function f is a visually identity transformation if for every image X, there exists scale factor such that f(X)= = D(X;s) s), where D the ideal downsampling function.

---

3.4 Super-Resolution Model Training
To better show the advantage of the proposed neural degradation prior and the adversarial degradation training, we adopt the ESRGAN [44] as our SR model. The SR model training procedure solves the following optimization problem:
min H mmx Leone OFES
B, 0c. 0F)
0G
+ max Eyua [max LGAN (T" ,0, PD. 0r)]}
OFES
where = [0]]0 lidl|2 <, and Fau is the identity transformation). and D are generator (restoration network) and discriminator of the SR model respectively. F is the degradation network. stands for parameter of network. JHR represents the high-resolution images. Tecnnt and LGAN are the content loss and the GAN loss (22) respectively. À is the coefficient to balance the two loss terms. The content loss Leont is the sum of the 1-norm loss and the VGG loss [17]
Leont HB, 00,000 =11 (eo (1
[ellos(()))) 6j(Gna (Fo (I"R)))
where 6j is the feature map of jth convolution layer of the VGG network [37]] and ey is the coefficient for term of the jth layer. The GAN loss LGAN is:
L 0c, 0F) log Dap (1 HR)
log Dap (Gog (For (R)
The optimization problem in Equation 1 iss more complicated than the ordinary two-player minimax problem in previous GAN-based SR methods [2] 44 43). There are three networks involved in the optimization, and each network tries to compete or cooperate with other two networks. The restoration network (i.e., generator) takes LR output of the degradation network as its input, generates SR images with realistic and natural texture, and tries to fool the discriminator network into believing the generated SR images are actually natural HR images. The discriminator network tries to distinguish SR output of the restoration network from natural HR images. The degradation network takes natural HR images its input, generates LR images with moderate but also complicated image degradations, tries to make the restoration network to generate unsatisfying results and to make easier for the discriminator network to distinguish. The optimization procedure drives the restoration network to improve, until it is robust enough and can keep generating perceptually satisfying SR results, even its LR input images suffered from complicated real-world degradations.
3.5 Training and Inference Efficiency
Compared with previous SR methods, our SR model needs cost more time during training phase. When we train our GAN-based SR method with adversarial neural degradation, we need to perturb the identity degradation network, optimize the restoration network and the discriminator network, in an alternating manner. The adversarial perturbation of the identity degradation network is done by taking gradient steps. In this work, before each operation step of the SR model, we use gradient steps for adversarial degradation perturbation. That requires additionally 5 forward and backward passes through the whole network, including the degradation network and the SR model. Thankfully, the degradation network is much smaller than the SR model. That is reasonable because degradation is much easier than restoration, which is a general property for all inverse problems. So the degradation network itself does not cost much, most of the additional training cost is due to the adversarial training procedure. As we mentioned before, we adopt the ESRGAN as our SR model. It would cost 8.92 TFLOPs for one training step of ESRGAN our SR model training with the same training setting. So our method need an increase in training time of factor of 6.49.
training batch, while would cost 57.85 TFLOPs for
However, what is more important for our real-world method inference efficiency. That because once finish the training, the robust SR model we get would not need retraining or finetune for
7

---

Table 1: Quantitative comparison with state-of-the-ar methods real-world blind image super- resolution benchmarks. The best and second best results are highlighted red and blue, respectively.
RealSR x x 131
DRealSR
x 4) [46]
SupER x 4) 21
ImgPairs x [18]
Method
PSNR1
LPIPS
PSNR1
LPIPS
PSNR1
LPIPS
PSNR
LPIPS
KernelGAN 12]
25.13
0.3349
28.56
0.3978
25.65
0.3445
26.74
0.3340
DAN [66
27.80
0.4114
30.59
0.4111
32.19
0.2064
28.56
0.2802
BSRNet [51
27.35
0.3084
29.49
0.3411
32.11
0.2532
28.59
0.3915
BSRGAN [5
26.51
0.2685
28.35
0.2929
29.18
0.2181
28.13
0.3346
Real- -eal-ESNNet
43
26.79
0.2939
28.50
0.3257
30.89
0.2496
28.34
0.3858
Real-Eas Beal-ESRGAN
43
25.85
0.2728
27.92
0.2818
27.55
0.2046
28.12
0.3679
SwinlR-Re -eal 226
26.43
0.2515
28.29
0.2739
28.27
0.1889
28.11
0.3464
DCLSS (2
27.83
0.4080
28.32
0.4760
32.71
0.1985
28.64
0.2844
PDM-SRGAN 28
21.96
0.3717
24.32
0.3668
25.31
0.2710
26.11
0.3788
FeMaSR
25.42
0.2937
26.59
0.3374
25.45
0,2419
27.03
0.3400
DASR [25
27.18
0.3113
29.72
0,2962
29.73
0.1476
28.34
0.3412
ReDegNet [2
24.77
0.2800
26.24
0.2995
26.60
0.1785
27.06
0.3930
ANDNet (ours)
28.47
0.2599
30.97
0.3381
32.96
0.2125
28.75
0.2786
ANDGAN (ours)
26.34
0.2326
28.95
0.2610
29.85
0.1372
27.78
0.2598
unpredicted image degradations. Once the training done, we can discard the degradation network and the discriminator network, and only use the trained restoration network for inference. That means, compared with previous SR methods, our SR method does not need additional computational and storage cost during inference phase.
4 Experiments
4.1
Datasets
Widely used datasets for SR evaluation, like Urban [15] and DIV2K (3), are suitable for the study of real-world SR, because they only contain HR images and their LR counterpart need to be generated by bicubic downsampling. We perform experiments on four datasets constructed for real-world SR evaluation: RealSR (3), DRealSR (46), SupER (21). and ImagePairs [18]. RealSR and
DRealSR are datasets containing HR and LR image pairs captured on the same scene by adjusting the focal length of digital cameras. SupER includes HR and LR image pairs constructed by camera hardware binning, which aggregates adjacent pixels on the sensor array. ImagePairs includes HR and LR image pairs captured by HR camera and LR camera, which aligned and mounted rig with a beam splitter. These datesets are constructed in different ways, so they could provide a comprehensive evaluation for world SR methods.
4.2 Quantitative Metrics
We use four quantitative metrics for quality assessment of SR images: PSNR, SSIM 45 LPIPS [4], and NIQE [33] PSNR and SSIM calculated Y channel of transformed YCbCr space for fair They are more focused on low-level pixel-wise image differences, and they are metrics suitable for PSNR-oriented SR models. LPIPS learned metric for full-reference image quality assessment. We could use the preceding three full-reference metrics, since all datasets we used the experiments have pixel-wise aligned HR and LR image pairs. Considering that GAN-base SR methods may generate detailed textures, which is although realistic but different from the ground truth, we also use NIQE, a no-reference metric for image quality evaluation. Both LPIPS and NIQE agree better with human visual perception, and they are metrics suitable for perceptual quality-oriented SR models.
comparison 40
4.3 Training Details
We use DIV2K 39
Flickr2K 27 and WED 30 as HR image datasets for training. The training
HR patch size set to 256 the batch size set 48. Following BSRGAN [5I) and Real- ESRGAN 43 train two SR models with method: PSNR- model noted ANDNet.

---

HR
Bicubic
KernelGAN 2
DAN 16
BSRGAN 51
Canon_Ok _02 (4x)
SwinlR Real 26
DCLS 29
FeMaSR S
DASR 25
ANDGAN
HR
Bicubic
KernelGAN 2
DAN 16
BSRGAN 511
Nikon_012 (1x)
SwinlR Real [26
DCLS 29
FeMaSR 5
DASR 25
ANDGAN
Figure 3: Qualitative comparisons real-world images from RealSR dataset with scale factor 4.
and a perceptual quality-oriented model noted as ANDGAN. First, we train ANDNet with the L1 loss only, for 10 iterations with 10-4 learning rate. Then initialization for the generator of the ANDGAN, and train the whole ANDGAN model with both the content loss and the GAN loss Equation | which are balanced by À = 0.1, for 5 x 10 iterations with 1 x 10-4 learning rate. We use Adam optimizer training.
the trained ANDNet an
20 for both generator and discriminator
We use projected gradient descent method 32 to adversarially search a small deviation of the identity degradation network. Before every training step for the restoration network, we initialize the degradation network to the identity transformation. and run 5 iterations of projected gradient descent with step size of 6 and perturbation size 20. We also need balance the weights for convolutional degradation, noise and nonlinearity, to make their respective intensity close to real-world degradation. When calculate the L.2 norm | gid ||2 determine the perturbation set S in Equation we use scale factors of 1, 10, 50 for term convolutional degradation, noise and nonlinearity respectively. Note that a larger scale factor means a stronger suppression for the degradation type.
4.4 Comparisons with Prior Works
We compare both our PSNR-oriented model and the perceptual quality-oriented model with several state-of-the-are methods. Quantitative results are shown Table and visual comparison between different methods are shown in Fig. Due to limited space, only provide most relevant results here. More detailed results can be found in the supplementary material.
4.5 Ablation Study
In order to study the effects of each component in the proposed blind SR method, gradually modify the AND model and compare their quantitative performances. The comparisons are shown in Table For cases where neural degradation is removed, we either use additive noise model for adversarial training or employ bicubic downsampling for non-adversarial training. In situations where identity initialization is eliminated, randomly initialize all x 3 convolution kernels in the degradation network using Xavier Initialization. We can observe that all three major components, namely adversarial perturbation, neural degradation, and identity initialization,
necessary.
The first column of Table labeled as "Configuration", is name and an explanation for particular ablation setting. If we do not use adversarial perturbation and neural degradation at all, our method

---

Table 2: Comparisons showing the effects of each component in the AND model, tested on the RealSR [
dataset with scale factor of 4.
Adversarial Perturbation
Neural Degradation
Identity Initialization
PSNRT
LPIPS of ANDNet
Configuration
ANDGAN
Classical SR training
26.53
0.4245
Adversarial noise training
26.60
0.4194
Synthetic data augmentation
X
/
27.31
0.3089
Severe random style shift
X
11.84
Complete AND model
/
28.47
0.2326
would become classical SR training method 8 27 56]. which assumes that the image degradation model is an ideal bicubic downsampling. If we retain solely the adversarial perturbation without inducing neural degradation, it implies the utilization of an adversarial noise training method similar to 4. 49] If we employ neural degradation with identity initialization but without adversarial perturbation, method becomes similar to SR training with synthetic data augmentation [5I 43 If we only remove the identity initialization from our method, the generated LR patches would no longer be visually similar to the HR patches. While the skeleton of the LR patches would remain unaffected, their color and texture would undergo dramatic change 48). We can observe that all three major components, namely adversarial perturbation, neural degradation, and identity initialization, are necessary.
5 Conclusions
We propose a neural degradation prior that encompasses various image degradations in the real world. Specifically, an untrained convolutional neural network, which deviates slightly from the identity transformation, can serve as a prior for various real-world image degradations. We employ adversarial searches to find small deviations in the degradation network during the training of the SR model. This approach allows the restoration model to continuously optimize itself on the worst degradation case, thus achieving robustness.
Acknowledgement
This work supported by the Natural Sciences and Engineering Research Council of Canada.
References
[1] Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Cireumventing defenses adversarial examples. In International conference machine learning, pages 274 283. PMLR, 2018
[2] Sefi Bell-Kligler, Assaf Shocher, and Michal Trani. Blind super-resolution kernel estimation using an internal-gan. Advances in Neural Information Processing Systems, 32, 2019.
[3] Jianrui Cai, Hui Zeng, Hongwei Yong, Zisheng Cao, and Lei Zhang. Toward real-world single image resoluteso A new benchmark and new model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3086-300 3095, 2019.
[4] Angela Castillo, María Escobar, Juan C Pérez, Andrés Romero, Radu Timofte, Lue Van Gool, and Pablo Arbelaez. Generalized real-world super-resolution through adversarial robustness. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1855-1866 2021.
[5] Chaofeng Chen, Xinyu Shi, Yipeng Qin, Xiaoming Li, Xiaoguang Han, Tao Yang, and Shihui Guo. Real-world blind super-resolution via feature matching with implicit high-resolution priors. Proceedings of the 30th ACM International Conference on Multimedia, pages 1329-1338. 2022.
[6] Chao Dong. Yubin Deng, Chen Change Loy, and Xiaoou Tang. Compression artifacts reduction by a deep convolutional network. In Proceedings of the IEEE International Conference on Computer Vision, pages 776-584, 2015.
10

---

[7] Chao Dong, Chen Change Loy. Kaiming He, and Xiaoou Tang. Learning deep convolutional network for image super-resolution. European conference on computer vision, pages 184-199. Springer, 2014.
[8] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolutional networks. IEEE transactions on pattern analysis and machine intelligence, 38(2):295-307, 2015.
[9] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pages 126-1135. PMLR, 2017.
[10] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks, In Proceedings of the thirteenth international conference artificial intelligence and statistics, pages 249-256. JMLR Workshop and Conference Procoedings, 2010.
[11] Ian Goodfellow, Jcan Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139- 144, 2020.
[12] lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and hameessing adversarial examples. arXiv preprint arX.v:1412.6572, 2014.
[13] Jinjin Gu, Hannan Lu, Wangmeng Zuo, and Chao Dong. Blind super-resolution with iterative kernel correction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1604-1613, 2019.
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Computer ECCV 2016: 14th European Conference, Amsterdam. The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pages 630-645. Springer, 2016.
[15] Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super-resolution from transformed self-exemplars. Proceedings of the IEFE conference on computer vision and pattern recognition, pages 5197-5206, 2015.
[16] Yan Huang, Shang Li. Liang Wang, Tieniu Tan, al. Unfolding the alternating optimization for blind super resolution. Advances in Neural Information Processing Systems,
2020.
[17] Justin Johnson, Alexandre Alahi, and Fei-Fei. Perceptual losses for real-time style transfer and super- resolution. In Computer Vision ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 694-711. Springer, 2016.
[18] Hamid Reza Vaezi Joze. Ilya Zharkov, Karlton Powell, Carl Ringler, Luming Liang, Andy Roulston, Moshe Lutz, and Vivek Pradeep. Imagepairs: Realistic super resolution dataset via beam splitter camera rig. In Proceedings of the IEEL/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 518-519, 2020.
[19] Tero Karras, Samuli Laine, and Timo Aila. A style-hased generator architecture for generative adversarial networks. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 4401-4410, 2019.
[20] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1412v:1412.6900 2014.
[21] Thomas Köhler, Michel Bätz, Farzad Naderi, André Kaup, Andreas Maier, and Christian Riess. Toward bridging the simulated-to -eaall gap: Benchmarking super-resolution real data. IEEE transactions on pattern analysis and machine intelligence, 42114)294442995, 2019.
[22] Christian Ledig, Lucas Theis, Ferene Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic single image super- resolution using a generative adversarial network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4681-4690, 2017.
[23] Xiaoming Li, Chaofeng Chen, Xianhui Lin, Wangmeng 7 and Lei Zhang. From face natural image: Learning real degradation for blind image super-resolution. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XVIII, pages 376-392. Springer, 2022.
[24] Yawei Li, Yuchen Fan, Xiaoyu Xiang, Denis Demandolx, Rakesh Ranjan, Radu Timofte, and Luc Van Gool. Efficient and explicit modelling of image hierarchies for image restoration. arXiv preprint 203.0:2 2023,
11

---

[25] Jie Liang, Hui Zeng, and Lei Zhang, Efficient and degradation-adaative network for real-world image super-resolution. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Purt XVIII, pages 574-591. Springer, 2022.
[26] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF international conference on computer vision. pages 1833- 1844, 2021.
[27] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 136-144, 2017.
[28] Zhengxiong Luo, Yan Huang, Shang Li, Liang Wang. and Tieniu Tan. Learning the degradation distribution for blind image super-resolutee resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 6063-666 6063-6072, 2022.
[29] Ziwei Luo, Haibin Huang, Lei Yu. Youwei Li, Haoqiang Fan, and Shuaicheng Liu. Deep constrained least squares for blind image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17642-17652, 2022.
[30] Kede Ma, Zhengfang Duammu, Qingbo Wu, Zhou Wang. Hongwei Yong. Hongliang Li, and Lei Zhang. Waterloo exploration database: New challenges for image quality assessment models. IEEE Transactions on Image Processing, 26(2): 1001):0004000000 1016, 2016.
[31] Andrew L Maas, Awni Y Hannun, Andrew Y Ng. et al. Rectifier nonlinearities improve neural network acoustic models. In Proc. ieml, volume 30, page 3. Atlanta, Georgia, USA, 2013.
[32] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arX6::/706.06005 2017.
[33] Anish Mittal, Rajiv Soundararajan, and Alan C Bovik, Making a "completely blind" image quality analyzer. IEEE Signal processing letters, 209-2220- 2012.
[34] Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In Internationnal conference on machine learning, pages 10-18. PMLR, 2013.
[35] Evgenia Rusak, Lukas Schott. Roland S Zimmermann, Julian Bitterwolf, Oliver Bringmann, Matthias Bethge, and Wieland Brendel. simple way to make neural networks robust against diverse image corruptions. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, pages 53-69. Springer, 2020.
[36] Wenzhe Shi, Jose Caballero, Ferene Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop. Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEFF conference on computer vision and pattern recognition, pages 1874-1883, 2016.
[37] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint arXiv: 14051115661. 2014.
[38] Jian Sun, Wenfei Cao, Zongben Xu, and Jean Ponce. Learning convolutional neural network for non- uniform motion blur removal. In Proceedings of the IEFE conference computer vision and pattern recognition, pages 769-777, 2015.
[39] Radu Timofte, Eirikur Agustsson, Luc Van Gool, Ming-Hsuan Yang, and Lei Zhang. Ntire 2017 challenge on single image super-resolution: Methods and results. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 114-125, 2017.
[40] Radu Timofte, Vincent De Smet, and Luc Van Gool. A+: Adjusted anchored neighborhood regression for fast super-resolution. In Computer Vision-ACCV 2014: 12th Asian Conference on Computer Vision, Singapore, Singapore, November 1-5, 2014, Revised Selected Papers, Part IV 12, pages 1 111-166. Springer, 2015.
[41] Florian Tramer, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Bonch, and Patrick McDaniel. Ensemble adversarial training: Attacks and defenses. arXiv preprint arXiv: 2017.
[42] Mei Wang and Weihong Deng. Deep visual domain adaptation: A survey. Neurocomputing, 312:135-1521 2018.
12

---

[43] Xintao Wang. Liangbin Xie, Chao Dong, and Ying Shan. Real-esrgan: Training real-world blind super- resolution with pure synthetic data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1905-1914, 2021.
[44] Xintao Wang. Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. Esrgan: Enhanced super-resolution generative adversarial networks. Proceedings of the European conference on compuder vision (ECCV) workshops, pages 0-0, 2018.
[45] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600-6) 612, 2004.
[46] Pengxu Wei, Ziwei Xie, Hannan Lu, Zongyuan Zhan, Qixiang Ye. Wangmeng Zuo, and Liang Lin. Component divide-and-conquer for world image super-resolution. In Computer VisionEECCV 2020: 16th European Conference, Glasgow, UK, August 23 23-28, 2020, Proceedings, Part VIII 16, pages 101-117. Springer, 2020.
[47] Eric Wong and Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International conference machine learning, pages 5286-5295. PMLR, 2018.
[48] Zhenlin Xu, Deyi Liu, Junlin Yang, Colin Raffel, and Mare Niethammer. Robust and generalizable visual representation learning via random convolutions. arXiv preprint arXiv:2007.130033 2020.
[49] Jiutao Yue, Haofeng Li. Pengxu Wei, Guanbin Li, and Liang Lin. Robust real-world image super-resolution against adversarial attacks. In Proceedings of the 29th ACM International Conference on Multimedia, pages 5148 5157, 2021.
[50] Chiyuan Zhang. Samy Bengio, Moritz Hardt, Michael C Mozer, and Yoram Singer. Identity crisis: Memorization and generalization under extreme overparameterization. arXiv preprint arXiv.190..04695 2019.
[51] Kai Zhang, Jingyun Liang, Luc Van Gool, and Radu Timofte. Designing a practical degradation model for deep blind image super-resolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4791 2021.
[52] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE Transactions on Image Processing, 26(7):3142- 3155, 2017.
[53] Richard Zhang. Making convolutional networks shift-invariant again. In International conference on machine learning, pages 7324-7334 PMIR, 2019.
[54] Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unrcasonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 586-595, 2018.
[55] Ruofan Zhang, Jinjin Gu, Haoyu Chen, Chao Dong, Yulun Zhang, and Wenming Yang. Crafting training degradation distribution for the
trade-off in weal-world super-resolution. 2023.
[56] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image super resertresolution using very deep residual channel attention networks. In Proceedings of the Earopean Conference Computer Vision (ECCV), pages 286-301, 2018
13


Reviews:
[{"review_id": "fJjqasJgI1", "reviewer": "Reviewer_9QoZ", "summary": "The paper presented a blind super-resolution algorithm, in which a neural network is used to represent image degradation, followed by the incorporation of adversarial learning mechanisms to study \"hard cases\". The technique to represent image degradation involves initializing a neural network, ensuring the initialization is an identity mapping, then inducing specific perturbations in the network's filtering and noise, resulting in irregular image degradation. Adversarial learning primarily involves monitoring difficult cases during training and subsequently increasing the penalty, establishing an adversarial approach.", "questions": "See weakness.", "limitations": "Description is not clear", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The main contribution of the paper is the application of neural networks to express forms of degradation, which is relatively innovative. This approach is more intricate than previous higher-order degradation and random degradation. Adversarial training, to some extent, may be beneficial as it promotes generality, though its innovativeness is not notably outstanding.", "weaknesses": "Despite the novelty of using a neural network to represent degradation, there are several concerns. \n\nFirst, a neural network represent degradation as a distribution, which actually has a range. An ICML paper [R1] pointed out that the range of this distribution could directly impact its performance and generalization abilities. The paper doesn't describe the distribution's range, nor is it clear how this range compares to that of other methods like RealESRGAN or BSRGAN.\n\n[R1] Crafting Training Degradation Distribution for the Accuracy-Generalization Trade-off in Real-World Super-Resolution. ICML 2023.\n\nSecondly, I have concerns about the experiment due to the possible limited range of degradation in the network. It may perform better in specific ranges, as RealESRGAN and BSRGAN may have a much wider range of degradation.\n\nRelatedly, the paper doesn't discuss the network's generalization performance for degradation. If degradation is a range within a distribution, there must be a range, and neither enlarging nor narrowing it is inherently better. The paper does not offer any control for this range.\n\nLastly, the article's writing is problematic. Many descriptions are long and repetitive, like section 3.1 and 3.2, which could have been resolved with a simple diagram, but the verbose writing makes it difficult to communicate a straightforward idea.\n\nFurthermore, the approach to representing degradation with a neural network seems intended to argue that this design of versatile degradation is more similar to real-life situations, particularly for low-resolution images that could be sampled from mobile ISPs. However, the paper lacks any descriptions or visualizations of low-resolution images and doesn't present experimental results to demonstrate the closeness of their design to actual cases. Hence, I have reservations about the nature of the low-resolution images they've created.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for bringing the ICML 2023 paper to our attention, and we will discuss the contribution of the ICML paper in the revised version of our paper. As several of your questions are rooted in this paper, please let us first elaborate on the connection between the ICML paper and our research.\n\nThe ICML paper indicates that when the degradation distribution during both training and testing perfectly matches, the SR model exhibits favorable generalization and achieves high performance simultaneously. The binning method is utilized to adjust the joint distribution of the three parameters of a widely-used image degradation model, helping the training degradation distribution to better match the testing distribution.\n\nWe fully agree with the indication in the ICML paper regarding the importance of aligning training and testing degradation. However, we would like to emphasize that the ultimate objective should be the alignment of distributions across the set of degradation functions, whereas the ICML paper focuses solely on aligning the distributions of degradation parameters within a given model.\n\nThis nuanced yet critical distinction can be elucidated through a simple analogy: Just as people cannot perfectly align a circle with a square by solely adjusting the circle's location and radius, they must also modify its shape. In this analogy, the location and radius correspond to the degradation parameters in the ICML paper, while modifying the shape corresponds to transitioning to a new family of functions, as demonstrated in our paper. In other word, if the blur-noise-JPEG model cannot generate LR image like the testing images, only using the binning method to adjust the distribution of parameters are not able to align the distributions. So our neural degradation model and the ICML paper both attempt to align degradation distributions. However, they focus on distinct and parallel areas.\n\n***\n**Q1:** The paper doesn't describe the distribution's range, nor is it clear how this range compares to that of other methods like RealESRGAN or BSRGAN.\n\n**A1:** We have described the feasible region of our model in Section 4.3 of our paper and in Section A.2 of our supplementary material. However, because of the distinct nature of the function family employed in our paper compared to other methods, conducting a quantitative comparison of their ranges poses challenges. In the revised version of our paper, we will present a visual comparison of LR patches.\n\n***\n**Q2:** The SR model may only perform better in limited range of degradation.\n\n**A2:** Your assertion is certainly true for any SR model, including ours, as the ICML paper indicates the importance of aligning training and testing degradation. However, we believe that the SR model which performs better within the real-world degradation range is more valuable than one that works better in another range. Quantitative comparison could demonstrate that our method generalizes better across all four real-world SR datasets. The range limitation is inherent to its nature rather than a drawback.\n***\n\n**Q3:** The article's writing is problematic. Many descriptions are long and repetitive, like section 3.1 and 3.2.\n\n**A3:** Thank you for your advice. We will revise those two sections to make them more concise.\n***\n**Q4:** The paper lacks visualizations of LR images.\n\n**A4:** Thank you for your advice. We will include visualizations of LR images in the revised version of our paper.\n***\n**Q5:** Description of limitations is not clear\n\n**A5:** We did discuss the limitations of our method in the Limitations section of our supplementary material (page 3)."}, {"review_id": "ABzk7iA8yX", "reviewer": "Reviewer_vm11", "summary": "This paper proposed a new image degradation system for blind image super-resolution tasks. The proposed method uses a neural network system to learn and capture the image degradation operations, combined with a image restoration network, the proposed method achieved satisfactory performance. ", "questions": "+ Can the author help visualize the degradation system's filter after the training, and compare it with real-world blur/noise/gamma kernel and effects?", "limitations": "The paper discussed some real-world use cases. The ablation study provides some insight into the system's different behavior. ", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed method introduced an image degradation system captured by a neural network system. The neural network is intuitive and has shown effectiveness on capture real-world degradation based on experimental results and analysis in the paper. \n\nThe presentation of the paper is good and the materials are well organized.  ", "weaknesses": "The paper is focusing on the degradation system and only adapted an existing SR model to be trained with the proposed system. It is not a weakness per see, but a new GAN system designed with the degradation system could have better performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** The paper is focusing on the degradation system and only adapted an existing SR model to be trained with the proposed system. It is not a weakness per see, but a new GAN system designed with the degradation system could have better performance.\n\n**A1:** Thank you for your advice. This work is inspired by two observations of image degradations. These two properties can be easily used in degradation model design, but for the time being, we cannot figure out how to use them in restoration model design. That is why we only adapted an existing SR model.\n***\n\n**Q2:** Can the author help visualize the degradation system's filter after the training, and compare it with real-world blur/noise/gamma kernel and effects?\n\n**A2:** Thank you for your advice. We will include visualizations of the degradation system's filter in the revised version of our paper. However, for the current model, these visualizations can hardly contain any information. This is because there are several layers in the neural degradation, and each layer is only responsible for a very small portion of the degradation. We cannot simply combine them together due to the presence of non-linear activation layers between them. We intend to retrain the model with a much smaller degradation model, consisting of only two or three layers. This adjustment aims to make the visualizations more informative. If you have any suggestions, please do not hesitate to share them with us.\n"}, {"review_id": "iVKJ1L4Yqf", "reviewer": "Reviewer_JnSR", "summary": "This work proposed a novel blind SR method via adversarial neural degradation. Utilizing the proposed adversarial neural degradation model can generate various nonlinear degradations effects and no supervisions are required. This makes the proposed method can deal with various real SR datasets. Experiments also verify the effectiveness of the proposed method. ", "questions": "See the weaknesses. ", "limitations": "The authors discussed the limitations. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tUtilizing adversarial neural degradation model for degradation synthesis is novel. \n2.\tThe performance is promising. \n", "weaknesses": "1.\tIn table 1, the authors are expected to provide the SR results with full supervisions by sota sr methods. The proposed method is not required to outperform these methods since they are full-supervised but the proposed method is zero-shot. This can help the readers know the gap between the proposed method and supervised methods. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** In table 1, the authors are expected to provide the SR results with full supervisions by sota sr methods. The proposed method is not required to outperform these methods since they are full-supervised but the proposed method is zero-shot. This can help the readers know the gap between the proposed method and supervised methods.\n\n**A1:** Thank you for your advice. We will incorporate it into the revised version of our paper. However, obtaining both degraded images and the corresponding latent images (ground truth) is expensive in reality. As a result, the number of images in real-world SR datasets is typically fewer than the number of images in the training set used in classical SR. We are curious if state-of-the-art SR methods trained with full supervision on such a small training set could serve as a reliable upper bound."}, {"review_id": "an9hQnRQ3w", "reviewer": "Reviewer_Qwb9", "summary": "This paper proposes an adversarial approach for blind image super-resolution. Instead of using combinations of synthetic degradations (e.g., Gaussian blur, JPEG compression), this paper proposes to use a degradation network to construct LR patches from HR patches during training. The degradation network is optimized with a constraint that the network parameters fall within a local region of that of an identity mapping. The proposed approach outperforms existing works on common datasets.", "questions": "While the proposed approach achieves promising performance, the underlying mechanism and motivation are unclear. It is advised to dress the concerns in the weakness section. I would be happy to adjust the rating if the aforementioned concerns are resolved. ", "limitations": "The authors addressed the limitations in the supplementary material.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed ANDNet and ANDGAN achieve promising performance on multiple common datasets, outperforming existing works.\n2. The idea of using a degradation network is interesting. \n", "weaknesses": "1. The working mechanism of the proposed approach is not clear. While the empirical results show the effectiveness of this approach, more explanation of the mechanism is necessary. Specifically, \n\n    **a)** Without additional constraint, why do the degradation network and restoration network correspond to degradation and restoration respectively? Is it possible that the degradation network attempts to enhance the HR input and the restoration network degrades back?\n \n    **b)** Why does the learned degradation network reveal the real-world degradations? In theory, there could be unlimited possibilities. How does it work?\n\n2. The claim in Ln.118 that `Almost all types of image degradation could ﬁnd a corresponding operation in a standard convolutional neural network.` is inaccurate. In practice, there are many degradations that cannot be represented by standard CNN operations. For example, the camera shot noise and read noise corrupt the input in the raw image domain, which cannot be represented by common operations. Similarly for multiplicative noise and JPEG compression. While CNN can be used to approximate the degradations, the above statement is over-claimed.\n\n3. The ablation studies (Sec. 5) is confusing. The configuration in Table 2 is difficult to understand. Please explain the ablation studies clearly. Specifically,\n\n    **a)** What is classical SR training? \n\n    **b)** What is synthetic data augmentation? \n\n    **c)** What is severe random style shift? (The difference to b) is the identity initialization, why does it related to style?)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** Why do the degradation network and restoration network correspond to degradation and restoration respectively? Is it possible that the degradation network attempts to enhance the HR input and the restoration network degrades back?\n\n**A1:** As demonstrated in Algorithm 1 of our supplementary material, the degradation network and restoration network are optimized alternatively. With a temporarily fixed restoration network, the degradation network continually seeks to increase the loss, intensifying image degradation. Conversely, with a temporarily fixed degradation network, the restoration network consistently endeavors to minimize the loss, enhancing image restoration quality. Therefore, we do not believe that the scenario where the degradation network enhances while the restoration network degrades would result in a stable convergence point. While this situation might appear feasible in theory, in practice, even a minor random disturbance could quickly propel the entire system out of that situation.\n***\n\n\n**Q2:** Why does the learned degradation network reveal the real-world degradations? In theory, there could be unlimited possibilities. How does it work?\n\n**A2:** We do not require any perturbed degradation network to correspond exactly with a real-world degradation. This is neither necessary nor feasible for our method. We only need to guarantee that the entire feasible region of the degradation network could cover most cases of real-world degradation. In other words, we only need to guarantee that most instances of real-world degradation could be represented by a degradation network. If we can establish a solid lower performance bound for the entire feasible region of the degradation network through adversarial training, this lower bound also applies to most real-world degradation cases, thus achieving robust restoration.\n***\n\n**Q3:** The claim that \"Almost all types of image degradation could find a corresponding standard CNN operation\" is inaccurate. There are many degradations that cannot be represented by standard CNN operations, such as noise in the raw image domain, multiplicative noise, and JPEG compression.\n\n**A3:** Thank you for pointing this out. We will revise the claim to make it more rigorous. We completely agree with your point that the degradations you mentioned cannot be accurately represented by standard CNN operations. In fact, we addressed similar issues in the Limitations section of our supplementary materials. The example of noise in the raw image domain, which you referred to, is similar to the halftoning degradation we discussed. Similarly, the scenario involving multiplicative noise closely resembles a low-light environment. Perhaps we could claim that \"Most dominated image degradations in real-world SR tasks could find corresponding standard CNN operations\"? If you have any suggestions, please don't hesitate to share with us.\n\n***\n**Q4:** The ablation studies is confusing. The configuration in Table 2 is difficult to understand. Please explain the ablation studies clearly.\n\n**A4:** We apologize for any confusion that may have arisen. The ablation studies are intended to investigate the effects of the three major components of our method: adversarial perturbation, neural degradation, and identity initialization. In each ablation setting, we retain only specific components and assess the method's performance. The first column, labeled as \"Configuration\" in Table 2, is a name and an explanation for a particular ablation setting.\n\nIf we do not use adversarial perturbation and neural degradation at all, our method would become a classical SR training method, which assumes that the image degradation model is an ideal bicubic downsampling. Most SR researches are with this setting, such as SRCNN, EDSR and RCAN.\n\nIf we retain solely the adversarial perturbation without inducing neural degradation, it implies the utilization of an adversarial noise training method similar to \"Generalized real-world super-resolution through adversarial robustness\" (ICCVW2021). The perturbation under this setting is additive noise, following most adversarial training researches on image classification tasks.\n\nIf we use neural degradation with identity initialization, but without adversarial perturbation, our method is then a SR training with synthetic data augmentation, working like \"Designing a practical degradation model for deep blind image super-resolution\" (ICCV2021). During model training, the neural degradation would be random sampling near the identity initialization rather than adversarial sampling, and the generated LR patches works like synthetic data augmentations.\n\nIf we employ neural degradation with identity initialization, but exclude adversarial perturbation, our approach becomes analogous to SR training with synthetic data augmentation, akin to the method presented in \"Designing a practical degradation model for deep blind image super-resolution\" (ICCV 2021). Throughout the model training process, neural degradation involves random sampling near an identity mapping, rather than adversarial sampling. Consequently, the generated LR patches function as synthetic data augmentations.\n\nIf we only remove the identity initialization from our method, the generated LR patches would no longer be visually similar to the HR patches. While the skeleton of the LR patches would remain unaffected, their color and texture would undergo a dramatic change, as the mapping would no longer be an identity mapping. This phenomenon is referred to as \"severe random style shift\" in our experiments."}, {"review_id": "WRNGcf6vn3", "reviewer": "Reviewer_BEQG", "summary": "This paper proposes a method of AND to learn neural degradation for the task of blind image super-resolution. The core idea is learning to degrade HR images by neural networks, trying to synthesize real-world degradations. Based on the synthesized data, a restoration model can be well trained. The proposed AND model has a unique advantage over the current state of the art in that it can generalize much better to unseen degradation variants. Experimental studies on public datasets show its effectiveness.", "questions": "1, the qualitative results are not promising and are still vague, compared with other methods. This needs more explanations.", "limitations": "The authors did not address the limitations.", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "+ The paper is generally well written, and it is easy to follow.\n\n+ The paper conducts extensive studies including ablation studies, showing effective results.", "weaknesses": "There are several issues in this paper, including:\n\n- The idea of learning degradation to better learn blind image super-resolution is not new, and has been studied in existing works, like \"To learn image super-resolution, use a GAN to learn how to do image degradation first\" (ECCV2018). The contribution can hardly be important.\n\n- The claim \"But regrettably, adversarial learning has not been applied to neural network models of signal restoration\" is not valid. As a fact, there have been many works in this direction. For example, \"Robust Real-World Image Super-Resolution against Adversarial Attacks\"(ACMMM2021). There are more that are not given as examples.\n\n- The experimental results need more clarification and explanations.\n\n- Minor: L238, \"By this way\", L278, \"So our method need an ...\"\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** The idea of learning degradation to better learn blind image super-resolution is not new, and has been studied in existing works, like \"To learn image super-resolution, use a GAN to learn how to do image degradation first\" (ECCV2018).\n\n**A1:** You seem to misunderstand the main contribution of our paper. It may help for us to clarify further.\n\nThe proposed neural degradation prior is an untrained neural network, which does not learn to imitate any specific degradation effects, hence not requiring degraded LR training images at all.  The untrained network is able to work as an image degradation prior, because it is designed to satisfy the following two common properties of a wide range of image degradations. First, the degradation effects can be modeled by CNN operations. Second, most degradations of interest can be considered as a small deviation from the identity transformation.\n\nOn the other hand, the ECCV2018 paper uses a trained network to imitate the degradation effects using a given LR image set; the learnt degradation is then used to train the super-resolution model. This is a totally different methodology from ours, and the methodology difference leads to the inferior  generalization ability of the former to the latter. Their method would fail easily if the degradation of the chosen LR training images mismatches the real degradation at the inference stage, while our method can generalize much better to unseen degradation variants with the untrained neural degradation prior.\n\nWe would like to show the generalization ability gap by experiments. However, the ECCV2018 paper focuses only on facial image SR, which is by itself a weakness, so we cannot compare it fairly with our method. Luckily, there is another paper published in ECCV2022, named \"From face to natural image: Learning real degradation for blind image super-resolution\". This ECCV2022 paper, like the ECCV2018 paper, also uses GAN to learn degradation from an LR image set but it can be applied to natural image SR task, so we can compare its performance with our method. The name of the ECCV2022 method is ReDegNet, and its performance is already shown in Table 1 of our paper (page 8) and in Table 1 of our supplementary material (page 4). The quantitative comparison proves that our method does generalize better on all four real-world SR datasets.\n***\n\n**Q2:** The claim \"But regrettably, adversarial learning has not been applied to neural network models of signal restoration\" is not valid. As a fact, there have been many works in this direction. For example, \"Robust Real-World Image Super-Resolution against Adversarial Attacks\" (ACMMM2021).\n\n**A2:** Thank you for pointing this out. We will tune down our statement accordingly, and discuss the contribution of the ACMMM2021 paper in the revised version of our paper. However, here we would like to emphasize the difference between the ACMMM2021 paper and our paper.\n\nThe ACMMM2021 paper borrows the earlier research in adversarial training for image classification tasks. But it does not account for the differences between the classification and restoration tasks. In the previous research on adversarial training for image classification tasks, Out-of-Distribution (OOD) perturbations are introduced through the deliberate efforts of malicious attackers. This approach utilizes pixelwise additive high-frequency noise as a concealed and effective perturbation attack. Note that the adversarial attack is very different from the type of signal degradations in restoration tasks. Specifically, the perturbations in super-resolution tasks encompass a mixture of blur, noise, and nonlinear transformations. As a result, for restoration tasks noise no longer predominantly influences the OOD perturbation as in classification tasks. Therefore, the proposed neural degradation prior is a more suitable perturbation model for real-world image restoration.\n\nThe experimental settings of the ACMMM2021 paper and our paper also differ due to different understandings of real-world image SR tasks. In the ACMMM2021 paper's experiment, real-world LR images were not used directly as input for the SR model. Instead, they were first manipulated by a malicious attacker, which may not represent the most crucial scenario for real-world image SR. Conversely, our approach utilizes real-world LR images directly as input and demonstrates promising performance on real-world image SR datasets.\n\n***\n**Q3:** The qualitative results are more vague, compared with other methods.\n\n**A3:** Yes, your observation is accurate. However, we must emphasize that the enhanced sharpness in the image outputs from other methods results from an artifact known as \"watercolor-like artifacts\" or \"painterly artifacts\". This artifact can occur when SR algorithms, especially GAN-based methods, generate images that exhibit characteristics similar to brush strokes or painterly textures, rather than natural or realistic details. The boundaries between two brush strokes would be very sharp, even sharper than the HR images. It is considered an artifact because it deviates from the original intention of SR, which is to enhance the detail and clarity of the image without introducing artistic or stylized characteristics. Compared to other methods, our SR output more closely resembles a natural HR image.\n\n***\n**Q4:** The authors did not address the limitations.\n\n**A4:** We did discuss the limitations of our method in the Limitations section of our supplementary material (page 3).\n\n***\nSince there are several misunderstandings regarding our paper, we would greatly appreciate it if you could review it again. If you have any further questions, please don't hesitate to reach out. We are more than willing to provide clarifications both within our discussion and in the paper itself."}]

Author rebuttal:

Meta-review: Before the discussion and rebuttal, reviewers highlighted the following strengths and weaknesses of Submission967:

*Strengths:*
- The use of adversarial neural degradation for degradation synthesis.
- Demonstrated effectiveness in capturing real-world degradation.
- Performance of the proposed method outperforms existing works on multiple datasets.
- Extensive experimental studies, including ablation, that showcase the method's effectiveness.

*Weaknesses:*
- The paper lacks clarity on the distribution range of degradation by the neural network, and its comparison to methods like RealESRGAN or BSRGAN.
- Questions regarding the experiment's possible limited range of degradation.
- Absence of discussions on the network's generalization performance for degradation.
- Issues with the paper's writing, including verbosity and lack of visualization.
- Unclear working mechanism of the proposed approach.
- Over-claims made in the paper, like the potential of CNNs to represent all types of image degradation.
- Confusing ablation studies.
- The idea of learning degradation for super-resolution isn't new, and the paper's contribution may not be significant.

Post-rebuttal, authors have addressed most concerns raised by reviewers. Reviewer 9QoZ acknowledges the paper's value, though suggests better experimental work. Reviewer Qwb9 seeks deeper insights into why the adversarial approach mirrors real-world degradations. Authors should tone down the claim regarding the capabilities of CNNs in representing all image degradation and show the degraded images in the revised version of the paper. Reviewer BEQG points out similarities to the ECCV2018 paper. The authors have addressed this concern by explaining the differences against the ECCV2018 paper. The AC concurred with the authors. Based on the overall comments, the decision is to accept the paper.
Decision: Accept (poster)