SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \min _ {\theta_ {G}} \{\mathbb {E} _ {I ^ {H R}} [ \max _ {\theta_ {F} \in S} L _ {c o n t} (I ^ {H R}; \theta_ {G}, \theta_ {F}) ] \\ + \lambda \max  _ {\theta_ {D}} \mathbb {E} _ {I ^ {H R}} \left[ \max  _ {\theta_ {F} \in S} L _ {G A N} \left(I ^ {H R}; \theta_ {G}, \theta_ {D}, \theta_ {F}\right) \right] \} \tag {1} \\ \end{array}
$$
where  $S = \{\theta \| \theta -\theta^{id}\| _2 < \varepsilon$ , and  $F_{\theta^{\text{id}}}$  is the identity transformation}.  $G$  and  $D$  are generator (restoration network) and discriminator of the SR model respectively.  $F$  is the degradation network.  $\theta$  stands for parameter of network.  $I^{HR}$  represents the high-resolution images.  $L_{cont}$  and  $L_{GAN}$  are the content loss and the GAN loss [22] respectively.  $\lambda$  is the coefficient to balance the two loss terms. The content loss  $L_{cont}$  is the sum of the 1-norm loss and the VGG loss [17]:
$$
\begin{array}{l} L _ {c o n t} \left(I ^ {H R}; \theta_ {G}, \theta_ {F}\right) = \left\| I ^ {H R} - G _ {\theta_ {G}} \left(F _ {\theta_ {F}} \left(I ^ {H R}\right)\right) \right\| _ {1} \\ + \sum_ {j} c _ {j} \| \phi_ {j} (I ^ {H R}) - \phi_ {j} \left(G _ {\theta_ {G}} \left(F _ {\theta_ {F}} (I ^ {H R})\right)\right) \| _ {2} ^ {2} \tag {2} \\ \end{array}
$$
where  $\phi_{j}$  is the feature map of  $j$ th convolution layer of the VGG network [37], and  $c_{j}$  is the coefficient for term of the  $j$ th layer. The GAN loss  $L_{GAN}$  is:
$$
\begin{array}{l} L _ {G A N} \left(I ^ {H R}; \theta_ {G}, \theta_ {D}, \theta_ {F}\right) = \log D _ {\theta_ {D}} \left(I ^ {H R}\right) \\ - \log D _ {\theta_ {D}} \left(G _ {\theta_ {G}} \left(F _ {\theta_ {F}} \left(I ^ {H R}\right)\right)\right) \tag {3} \\ \end{array}
$$
The optimization problem in Equation 1 is more complicated than the ordinary two-player minimax problem in previous GAN-based SR methods [22, 44, 43]. There are three networks involved in the optimization, and each network tries to compete or cooperate with other two networks. The restoration network (i.e., generator) takes LR output of the degradation network as its input, generates SR images with realistic and natural texture, and tries to fool the discriminator network into believing the generated SR images are actually natural HR images. The discriminator network tries to distinguish SR output of the restoration network from natural HR images. The degradation network takes natural HR images as its input, generates LR images with moderate but also complicated image degradations, tries to make the restoration network to generate unsatisfying results and to make it easier for the discriminator network to distinguish. The optimization procedure drives the restoration network to improve, until it is robust enough and can keep generating perceptually satisfying SR results, even if its LR input images are suffered from complicated real-world degradations.
3.5 Training and Inference Efficiency
Compared with previous SR methods, our SR model needs to cost more time during training phase. When we train our GAN-based SR method with adversarial neural degradation, we need to perturb the identity degradation network, optimize the restoration network and the discriminator network, in an alternating manner. The adversarial perturbation of the identity degradation network is done by taking gradient steps. In this work, before each operation step of the SR model, we use 5 gradient steps for adversarial degradation perturbation. That requires additionally 5 forward and backward passes through the whole network, including the degradation network and the SR model. Thankfully, the degradation network is much smaller than the SR model. That is reasonable because degradation is much easier than restoration, which is a general property for all inverse problems. So the degradation network itself does not cost much, most of the additional training cost is due to the adversarial training procedure. As we mentioned before, we adopt the ESRGAN as our SR model. It would cost 8.92 TFLOPs for one training step of ESRGAN on a training batch, while would cost 57.85 TFLOPs for our SR model training with the same training setting. So our method need an increase in training time of a factor of 6.49.
However, what is more important for our real-world SR method is inference efficiency. That is because once we finish the training, the robust SR model we get would not need retraining or finetune for
Table 1: Quantitative comparison with state-of-the-art methods on real-world blind image superresolution benchmarks. The best and second best results are highlighted in red and blue, respectively.
Method	RealSR(×4) [3]	DRealSR(×4) [46]	SupER(×4) [21]	ImgPairs(×2) [18]
PSNR↑	LPIPS↓	PSNR↑	LPIPS↓	PSNR↑	LPIPS↓	PSNR↑	LPIPS↓
KernelGAN [2]	25.13	0.3349	28.56	0.3978	25.65	0.3445	26.74	0.3340
DAN [16]	27.80	0.4114	30.59	0.4111	32.19	0.2064	28.56	0.2802
BSRNet [51]	27.35	0.3084	29.49	0.3411	32.11	0.2532	28.59	0.3915
BSRGAN [51]	26.51	0.2685	28.35	0.2929	29.18	0.2181	28.13	0.3346
Real-ESRNet [43]	26.79	0.2939	28.50	0.3257	30.89	0.2496	28.34	0.3858
Real-ESRGAN [43]	25.85	0.2728	27.92	0.2818	27.55	0.2046	28.12	0.3679
SwinIR-Real [26]	26.43	0.2515	28.29	0.2739	28.27	0.1889	28.11	0.3464
DCLS [29]	27.83	0.4080	28.32	0.4760	32.71	0.1985	28.64	0.2844
PDM-SRGAN [28]	21.96	0.3717	24.32	0.3668	25.31	0.2710	26.11	0.3788
FeMaSR [5]	25.42	0.2937	26.59	0.3374	25.45	0.2419	27.03	0.3400
DASR [25]	27.18	0.3113	29.72	0.2962	29.73	0.1476	28.34	0.3412
ReDegNet [23]	24.77	0.2800	26.24	0.2995	26.60	0.1785	27.06	0.3930
ANDNet (ours)	28.47	0.2599	30.97	0.3381	32.96	0.2125	28.75	0.2786
ANDGAN (ours)	26.34	0.2326	28.95	0.2610	29.85	0.1372	27.78	0.2598
unpredicted image degradations. Once the training is done, we can discard the degradation network and the discriminator network, and only use the trained restoration network for inference. That means, compared with previous SR methods, our SR method does not need additional computational and storage cost during inference phase.
4.1 Datasets
Widely used datasets for SR evaluation, like Urban100 [15] and DIV2K [39], are not suitable for the study of real-world SR, because they only contain HR images and their LR counterpart need to be generated by bicubic downsampling. We perform experiments on four datasets constructed for real-world SR evaluation: RealSR [3], DRealSR [46], SupER [21], and ImagePairs [18]. RealSR and DRealSR are datasets containing HR and LR image pairs captured on the same scene by adjusting the focal length of digital cameras. SupER includes HR and LR image pairs constructed by camera hardware binning, which aggregates adjacent pixels on the sensor array. ImagePairs includes HR and LR image pairs captured by a HR camera and a LR camera, which are aligned and mounted on a rig with a beam splitter. These datasets are constructed in different ways, so they could provide a comprehensive evaluation for real-world SR methods.
4.2 Quantitative Metrics
We use four quantitative metrics for quality assessment of SR images: PSNR, SSIM [45], LPIPS [54], and NIQE [33]. PSNR and SSIM are calculated on Y channel of transformed YCbCr space for fair comparison [40]. They are more focused on low-level pixel-wise image differences, and they are metrics suitable for PSNR-oriented SR models. LPIPS is a learned metric for full-reference image quality assessment. We could use the preceding three full-reference metrics, since all datasets we used in the experiments have pixel-wise aligned HR and LR image pairs. Considering that GAN-based SR methods may generate detailed textures, which is although realistic but different from the ground truth, we also use NIQE, a no-reference metric for image quality evaluation. Both LPIPS and NIQE agree better with human visual perception, and they are metrics suitable for perceptual quality-oriented SR models.
4.3 Training Details
We use DIV2K [39], Flickr2K [27] and WED [30] as HR image datasets for training. The training HR patch size is set to 256 and the batch size is set to 48. Following BSRGAN [51] and Real-ESRGAN [43], we train two SR models with our method: a PSNR-oriented model noted as ANDNet,
Bicubic
Canon_042  $(4\times)$
Nikon_012  $(4\times)$
Figure 3: Qualitative comparisons on real-world images from RealSR [3] dataset with scale factor 4.
KernelGAN [2]
DAN [16]
BSRGAN [51]
SwinIR-Real [26]
DCLS [29]
FeMaSR [5]
DASR [25]
ANDGAN
HR
SwinIR-Real [26]
Bicubic
DCLS [29]
KernelGAN [2]
FeMaSR [5]
DAN [16]
DASR [25]
BSRGAN [51]
ANDGAN
and a perceptual quality-oriented model noted as ANDGAN. First, we train ANDNet with the L1 loss only, for  $1 \times 10^{6}$  iterations with  $1 \times 10^{-4}$  learning rate. Then we use the trained ANDNet as an initialization for the generator of the ANDGAN, and train the whole ANDGAN model with both the content loss and the GAN loss in Equation 1, which are balanced by  $\lambda = 0.1$ , for  $5 \times 10^{5}$  iterations with  $1 \times 10^{-4}$  learning rate. We use Adam optimizer [20] for both generator and discriminator training.
We use projected gradient descent method [32] to adversarially search a small deviation of the identity degradation network. Before every training step for the restoration network, we initialize the degradation network to the identity transformation, and run 5 iterations of projected gradient descent with step size of 6 and perturbation size  $\varepsilon = 20$ . We also need to balance the weights for convolutional degradation, noise and nonlinearity, to make their respective intensity close to real-world degradation. When we calculate the L2 norm  $\| \theta - \theta^{id} \|_2$  to determine the perturbation set  $S$  in Equation 1, we use scale factors of 1, 10, 50 for term of convolutional degradation, noise and nonlinearity respectively. Note that a larger scale factor means a stronger suppression for the degradation type.
4.4 Comparisons with Prior Works
We compare both our PSNR-oriented model and the perceptual quality-oriented model with several state-of-the-art methods. Quantitative results are shown in Table 1, and visual comparison between different methods are shown in Fig. 3. Due to limited space, we can only provide most relevant results here. More detailed results can be found in the supplementary material.
4.5 Ablation Study
In order to study the effects of each component in the proposed blind SR method, we gradually modify the AND model and compare their quantitative performances. The comparisons are shown in Table 2. For cases where neural degradation is removed, we either use an additive noise model for adversarial training or employ bicubic downsampling for non-adversarial training. In situations where identity initialization is eliminated, we randomly initialize all  $3 \times 3$  convolution kernels in the degradation network using Xavier Initialization. We can observe that all three major components, namely adversarial perturbation, neural degradation, and identity initialization, are necessary.
The first column of Table 2, labeled as "Configuration", is a name and an explanation for a particular ablation setting. If we do not use adversarial perturbation and neural degradation at all, our method
Table 2: Comparisons showing the effects of each component in the AND model, tested on the RealSR [3] dataset with a scale factor of 4.
Configuration	Adversarial
Perturbation	Neural
Degradation	Identity
Initialization	PSNR↑ of
ANDNet	LPIPS↓ of
ANDGAN
Classical SR training	X	X	-	26.53	0.4245
Adversarial noise training	✓	X	-	26.60	0.4194
Synthetic data augmentation	X	✓	✓	27.31	0.3089
Severe random style shift	✓	✓	X	11.84	-
Complete AND model	✓	✓	✓	28.47	0.2326
would become a classical SR training method [8, 27, 56], which assumes that the image degradation model is an ideal bicubic downsampling. If we retain solely the adversarial perturbation without inducing neural degradation, it implies the utilization of an adversarial noise training method similar to [4, 49]. If we employ neural degradation with identity initialization but without adversarial perturbation, our method becomes similar to SR training with synthetic data augmentation [51, 43]. If we only remove the identity initialization from our method, the generated LR patches would no longer be visually similar to the HR patches. While the skeleton of the LR patches would remain unaffected, their color and texture would undergo a dramatic change [48]. We can observe that all three major components, namely adversarial perturbation, neural degradation, and identity initialization, are necessary.
We propose a neural degradation prior that encompasses various image degradations in the real world. Specifically, an untrained convolutional neural network, which deviates slightly from the identity transformation, can serve as a prior for various real-world image degradations. We employ adversarial searches to find small deviations in the degradation network during the training of the SR model. This approach allows the restoration model to continuously optimize itself on the worst degradation case, thus achieving robustness.
This work is supported by the Natural Sciences and Engineering Research Council of Canada.
[1] Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In International conference on machine learning, pages 274-283. PMLR, 2018.
[2] Sefi Bell-Kligler, Assaf Shocher, and Michal Irani. Blind super-resolution kernel estimation using an internal-gan. Advances in Neural Information Processing Systems, 32, 2019.
[3] Jianrui Cai, Hui Zeng, Hongwei Yong, Zisheng Cao, and Lei Zhang. Toward real-world single image super-resolution: A new benchmark and a new model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3086-3095, 2019.
[4] Angela Castillo, María Escobar, Juan C Pérez, Andrés Romero, Radu Timofte, Luc Van Gool, and Pablo Arbelaez. Generalized real-world super-resolution through adversarial robustness. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1855–1865, 2021.
[5] Chaofeng Chen, Xinyu Shi, Yipeng Qin, Xiaoming Li, Xiaoguang Han, Tao Yang, and Shihui Guo. Real-world blind super-resolution via feature matching with implicit high-resolution priors. In Proceedings of the 30th ACM International Conference on Multimedia, pages 1329-1338, 2022.
[6] Chao Dong, Yubin Deng, Chen Change Loy, and Xiaou Tang. Compression artifacts reduction by a deep convolutional network. In Proceedings of the IEEE International Conference on Computer Vision, pages 576-584, 2015.
[43] Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan. Real-esrgan: Training real-world blind superresolution with pure synthetic data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1905-1914, 2021.
[44] Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. Esrgan: Enhanced super-resolution generative adversarial networks. In Proceedings of the European conference on computer vision (ECCV) workshops, pages 0-0, 2018.
[45] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600-612, 2004.
[46] Pengxu Wei, Ziwei Xie, Hannan Lu, Zongyuan Zhan, Qixiang Ye, Wangmeng Zuo, and Liang Lin. Component divide-and-conquer for real-world image super-resolution. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VIII 16, pages 101-117. Springer, 2020.
[47] Eric Wong and Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International conference on machine learning, pages 5286-5295. PMLR, 2018.
[48] Zhenlin Xu, Deyi Liu, Junlin Yang, Colin Raffel, and Marc Niethammer. Robust and generalizable visual representation learning via random convolutions. arXiv preprint arXiv:2007.13003, 2020.
[49] Jiutao Yue, Haofeng Li, Pengxu Wei, Guanbin Li, and Liang Lin. Robust real-world image super-resolution against adversarial attacks. In Proceedings of the 29th ACM International Conference on Multimedia, pages 5148-5157, 2021.
[50] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Michael C Mozer, and Yoram Singer. Identity crisis: Memorization and generalization under extreme overparameterization. arXiv preprint arXiv:1902.04698, 2019.
[51] Kai Zhang, Jingyun Liang, Luc Van Gool, and Radu Timofte. Designing a practical degradation model for deep blind image super-resolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4791-4800, 2021.
[52] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE Transactions on Image Processing, 26(7):3142-3155, 2017.
[53] Richard Zhang. Making convolutional networks shift-invariant again. In International conference on machine learning, pages 7324-7334. PMLR, 2019.
[54] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[55] Ruofan Zhang, Jinjin Gu, Haoyu Chen, Chao Dong, Yulun Zhang, and Wenming Yang. Crafting training degradation distribution for the accuracy-generalization trade-off in real-world super-resolution. 2023.
[56] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image super-resolution using very deep residual channel attention networks. In Proceedings of the European Conference on Computer Vision (ECCV), pages 286-301, 2018.