SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Wenlong Zhang $^{1,2}$ , Xiaohui Li $^{2,3}$ , Guangyuan Shi $^{1}$ , Xiangyu Chen $^{2,4,5}$ , Xiaoyun Zhang $^{3}$ , Yu Qiao $^{2,5}$ , Xiao-Ming Wu $^{1}$ , Chao Dong $^{2,5\dagger}$
<sup>1</sup>The HongKong Polytechnic University <sup>2</sup>Shanghai AI Laboratory
$^{3}$ Shanghai Jiao Tong University  $^{4}$ University of Macau
Shenzhen Institute of Advanced Technology, CAS
wenlong.zhang@connect.polyu.hk, xiao-ming.wu@polyu.edu.hk, chao.dong@siat.ac.cn
In this paper, we take a new look at real-world image super-resolution (real-SR) from a multi-task learning perspective. We demonstrate that the conventional formulation of real-SR can be viewed as solving multiple distinct degradation tasks using a single shared model. This poses a challenge known as task competition or task conflict in multi-task learning, where certain tasks dominate the learning process, resulting in poor performance on other tasks. This problem is exacerbated in the case of real-SR, due to the involvement of numerous degradation tasks. To address the issue of task competition in real-SR, we propose a task grouping approach. Our approach efficiently identifies the degradation tasks where a real-SR model falls short and groups these unsatisfactory tasks into multiple task groups. We then utilize the task groups to fine-tune the real-SR model in a simple way, which effectively mitigates task competition and facilitates knowledge transfer. Extensive experiments demonstrate our method achieves significantly enhanced performance across a wide range of degradation scenarios. The source code is available at https://github.com/XPixelGroup/TGSR.
Real-world image super-resolution (real-SR) aims to enhance the resolution and quality of low-resolution images captured in real-world scenarios. Real-SR algorithms enable improved image quality and better visual understanding, making them valuable in a wide range of applications [43, 46, 45, 14, 17]. Unlike non-blind SR [9, 56, 55] and classical blind SR [13, 18, 38] that assumes a simple degradation process, real-SR deals with complex and unknown degradations present in real-world imaging conditions, such as noise, blur, compression artifacts, and sensor limitations. The diversity of real-world degradations and unknown degradation parameters make it challenging to reverse the specific degradation effects and accurately recover high-resolution details.
Prior studies tackle real-SR by designing various degradation models to simulate the degradation process in real-world scenarios. For instance, they generate synthetic paired training data with a shuffled degradation model (BSRGAN [50]), a high-order degradation model (RealESRGAN [39]), or a three-level degradation model (DASR [25]). Existing methods commonly train a single real-SR network with training data generated by a sophisticated degradation model, aiming to cover as many degradation cases as possible during the training process. Under this framework, recent works have focused on enhancing various aspects of the real-SR network, including improving the backbone network architecture [26], optimizing inference efficiency [25], enhancing generalization capabilities [23], and improving modulation ability [31].
This study takes a new look at real-SR from a multi-task learning perspective. We show that the conventional formulation of real-SR essentially corresponds to a multi-task learning problem, which involves solving a large number of different degradation tasks simultaneously with a single shared model. Consequently, real-SR faces a well-known challenge in multi-task learning, namely task competition or task conflict [32]. It refers to the situation that tasks compete for model capacity, potentially resulting in certain tasks dominating the learning process and adversely affecting the performance of other tasks. This problem is severely aggravated in the case of real-SR, where a large number of degradation tasks (up to thousands) are typically involved. In our pilot experiment (Fig. 1), we have observed that a real-SR network trained with the multi-task objective fails to yield satisfactory results for a significant portion of degradation tasks.
We propose a task grouping approach (Fig. 2) to alleviate the negative impact of task competition for real-SR. Task grouping [48, 37, 35, 10] is an effective technique in multi-task learning that helps mitigate negative transfer by training similar tasks together. This approach typically involves learning the relationship or relevance between pairs of tasks through validation or fine-tuning, which becomes impractical in the context of real-SR where there is a large number of tasks to consider. Hence, we resort to design indirect measures to assess task affinity. Specifically, we introduce a performance indicator based on gradient updates to efficiently identify the degradation tasks that a real-SR model falls short of. Then, we propose an algorithm to group these unsatisfactory tasks into multiple groups based on a performance improvement score. Finally, we propose TGSR, a simple yet effective real-SR method, which leverages the identified task groups to fine-tune the pre-trained real-SR network. Our key contributions are summarized as follows:
- We take a new look at real-SR from a multi-task learning perspective and highlight the task competition problem.
- We propose a task grouping approach to effectively address the task competition problem for real-SR and develop a task grouping based real-SR method (TGSR).
- We conduct extensive experiments to validate the effectiveness of TGSR, and the results demonstrate its superior performance compared to state-of-the-art real-SR methods.
Image super-resolution. Since Dong et al. [9] first introduced Convolutional Neural Networks (CNNs) to the SR problem, a series of learning-based SR methods have made great progress, including deep networks [21], dense connections [56], channel attention of [55], residual-in-residual dense blocks [41], and transformer structure [26]. To reconstruct realistic textures, Generative Adversarial Networks (GANs) [24, 41, 53, 42] have been employed to generate visually pleasing results. However, these methods adopt a bicubic down-sampling degradation that is insufficient for real-world images.
Real-world super-resolution. To address the SR problem in real-world scenarios, classical blind SR methods primarily use Gaussian blur and noise to model the distribution of real-world images. Significant progress has been made through a variety of approaches, including the use of a single SR network with multiple degradations [51], kernel estimation [13, 18, 44, 2], and representation learning [38]. Additionally, BSRGAN [50] proposes a large degradation model that incorporates multiple degradations using a shuffled strategy, while RealESRGAN [39] employs a high-order strategy to construct a large degradation model. DASR [25] adopts a three-level degradation distribution (i.e., two one-order and one high-order degradation models) to simulate the distribution of real-world images. These works demonstrate the potential of large degradation models for real-world applications. In addition, recent works also improve real-SR in multiple dimensions, such as network backbone [26], efficiency [25], generalization [23, 54, 52] and modulation ability [31].
Multi-task learning. Multi-task learning methods can be roughly divided into three categories: task balancing, task grouping, and architecture design. Task balancing [16, 20, 47, 27, 34, 15, 8, 7, 19] methods address task/gradient conflicts by re-weighting the loss or manipulating the update gradient. Task grouping [48, 37, 35, 10] methods mainly focus on identifying which tasks should be learned together. Zamir et al. [48] provide a task taxonomy that captures the notion of task transferability. TAG [10] determines task groups by computing affinity scores that capture the effect between tasks. Architecture design methods can mainly be divided into hard parameter sharing methods [22, 28, 3] and soft parameter sharing methods [30, 33, 12, 11]. Hard parameter sharing methods require
(a) PSNR distance between the real-SR and single-task SR networks across 100 randomly selected degradation tasks.
Figure 1: Illustration of task competition. The jointly-trained multi-task real-SR network falls short in producing satisfactory results for almost half of the degradation tasks, on which the fine-tuned single-task networks obtain more than 0.4dB performance gain, indicating these tasks are dominated by other tasks during the learning process.
(b) Visual comparison between satisfactory and unsatisfactory tasks.
different decoders for different tasks, while soft parameter sharing methods do cross-talk between different task networks. We focus on a task grouping strategy in our TGSR. The strategy identifies unsatisfactory degradation tasks in a large degradation space. Subsequently, we can further improve the overall performance of the real-SR network by fine-tuning it in the unsatisfactory tasks.
3 Real-SR as a Multi-task Learning Problem
Problem Formulation Real-SR aims to restore a high-resolution (HR) image  $x$  from its low-resolution (LR) counterpart  $y$  that has undergone an unknown and intricate degradation process:
$$
y = \mathcal {D} (x) = \left(f _ {n} \circ \dots \circ f _ {2} \circ f _ {1}\right) (x), \tag {1}
$$
where  $f_{i}$  represents a degradation function such as Gaussian blurring with a kernel width [0.1, 2.4], adding Gaussian noise with a noise level [1, 25], a downsampling operation with a scale factor  $r$ , or applying the JPEG compression. Hence,  $\mathcal{D}$  represents a vast continuous degradation space that encompasses an infinite number of degradations. Existing real-SR models such as BSRGAN [50], RealESRGAN [39], and DASR [25] make different assumptions in the generation of  $\mathcal{D}$ , aiming to simulate the highly complex degradation process in real-world scenarios.
An SR task  $\tau$  can be defined as a training pair  $(x, y = d(x))$  formed by sampling a degradation  $d$  from the degradation space  $\mathcal{D}$  and applying it on an HR image  $x$  to generate an LR image  $y$ . Due to the infinite size of  $\mathcal{D}$ , it is impossible to consider every degradation case. Hence, a common approach is to sample a large number of degradations to sufficiently represent the degradation space. Given a set of high-resolution images  $\mathcal{X}$ , with  $N$  different degradations sampled from  $\mathcal{D}$  ( $N \gg |\mathcal{X}|$ ), we can generate a set of SR tasks:  $\mathcal{T} = \{\tau_i = (x_i, y_i)\}_{i=1}^N$ , where  $x_i \in \mathcal{X}$  and  $y_i$  is obtained by applying a degradation on  $x_i$ . A real-SR model is commonly trained by minimizing the empirical risk:
$$
\mathcal {L} _ {\text {t o t a l}} (\theta) = \sum_ {i = 1} ^ {N} \mathcal {L} _ {i} \left(\tau_ {i}; \theta\right), \tag {2}
$$
where  $\mathcal{L}_i$  is the loss on task  $\tau_{i}$ , and  $\theta$  are the trainable model parameters shared by all  $N$  tasks. Therefore, real-SR is essentially a multi-task learning problem, aiming to solve  $N$  different SR tasks with a single shared model. In the following, we refer to  $\tau_{i}$  as a degradation task.
Task Competition From the view of multi-task learning, real-SR is a challenging problem as it intends to solve a large number of different degradation tasks (e.g.,  $N = 10^{3}$ ) altogether with a single model, inevitably suffering from task competition that some tasks may dominate the training process leading to poor performance on other tasks [37].
To illustrate the impact of task competition, we randomly sample 100 degradation tasks with the degradation model in [39] and train a real-SR network with the multi-task objective in Eq. 2. Next, we fine-tune the real-SR network on each degradation task independently and obtain 100 fine-tuned
Figure 2: Overview of our proposed task grouping approach for real-SR.
models, which we call single-task networks. We then compare the performance between the real-SR network and the single-task networks on each degradation task by computing their PSNR distance. The results in Fig. 1 show that for nearly half of the degradation tasks, the PSNR distance between the real-SR network and single-task network exceeds  $0.4\mathrm{dB}$ , indicating that these tasks are not well solved by the real-SR network, which we refer to as unsatisfactory tasks. From an optimization perspective, the other tasks (those with PSNR distance less than  $0.4\mathrm{dB}$ ) dominate the learning process and are effectively solved by the real-SR network, which we refer to as satisfactory tasks.
4 Real-SR via Task Grouping
Our analysis in Sec. 3 shows that when a real-SR network is tasked with many degradation tasks, they may compete for model capacity or interfere with each other, resulting in a significant decline in performance for certain tasks. This phenomenon, commonly referred to as negative transfer [37], is a well-known challenge in the field of multi-task learning. An effective approach to mitigate negative transfer is task selection or task grouping [10, 37, 48]. By finding groups of tasks that may benefit from training together, the interference among tasks can be minimized.
4.1 Which Tasks Should Be Learned Together for Real-SR?
The results in Fig. 1 suggest that the satisfactory tasks should be grouped together. These dominant tasks may share certain characteristics or similarities that lead to minimal task conflicts, making them more prominent in the training process. More importantly, considering that the real-SR network has effectively tackled these tasks and reached a satisfactory performance level, we may focus on the unsatisfactory tasks that present greater potential for improvement.
Efficiently Identifying Unsatisfactory Tasks To identify the unsatisfactory tasks, we may use the approach described in Sec. 3 and Fig. 1, by comparing the performance between the jointly-trained multi-task real-SR network (referred to as the pre-trained real-SR network hereafter) and the fine-tuned single-task networks. However, the cost of adopting this approach is prohibitive due to the large number of degradation tasks (e.g., thousands or tens of thousands) involved in real-SR.
Therefore, to efficiently identify the unsatisfactory tasks, we propose a measure to assess whether a degradation task has been well solved by the real-SR network, which is done through fine-tuning the pre-trained real-SR network for a small number of iterations (e.g., 100) on the degradation task. Specifically, we assess the impact of the gradient update of task  $\tau_{i}$  on the shared parameters  $\theta$  (i.e., the pre-trained real-SR network), by comparing the loss of  $\tau_{i}$  before and after updating  $\theta$ . We develop a performance indicator defined as:
$$
z _ {i} ^ {t} = \frac {\mathcal {L} _ {i} \left(\tau_ {i} , \theta_ {i} ^ {t}\right)}{\mathcal {L} _ {i} \left(\tau_ {i} , \theta\right)}, \tag {3}
$$
where  $\theta_i^t$  are the parameters updated on task  $\tau_{i}$  at the time step  $t$ . Notice that a high value of  $z_{i}^{t}$  indicates the loss of task  $\tau_{i}$  is not significantly reduced after fine-tuning, suggesting that the pre-trained real-SR network has achieved good performance on this task.
Algorithm 1: Degradation Task Grouping for Real-SR
Input: A set of degradation tasks  $\mathcal{T} = \{\tau_1,\tau_2,\dots,\tau_n\}$ , the pre-trained real-SR model  $\theta$ , the number of groups  $c$ , and the threshold values  $t_0,t_1,\dots ,t_c$   
for any  $\tau_{i}\in \mathcal{T}$  do  
| Compute the average performance indicator  $\hat{z}_i$  with Eq. 4;  
end  
// Select unsatisfactory tasks  
Let  $\hat{\mathcal{T}} = \{\tau_i|\hat{z}_i > t_0\}$ ;  
// Group unsatisfactory tasks  
for  $i = 1,\ldots ,c$  do  
| Fine-tune the pre-trained real-SR network with all unsatisfactory tasks;  
for any  $\tau_{j}\in \hat{\mathcal{T}}$  do  
| Compute the performance improvement score  $s_j$  with Eq. 5;  
end  
Let  $G_{i} = \{\tau_{j}|s_{j} > t_{i}\}$ , where  $t_i$  is the threshold for group  $i$ ;  
Let  $\hat{\mathcal{T}} = \hat{\mathcal{T}}\backslash G_i$ ;  
end  
Output: Degradation task groups  $\mathcal{G} = \{G_1,G_2,\dots,G_c\}$ .
For stability, we consider the average performance indicator during the fine-tuning process on task  $\tau_{i}$ :
$$
\hat {z} _ {i} = \frac {1}{t _ {e} - t _ {s}} \sum_ {t = t _ {s}} ^ {t _ {e}} z _ {i} ^ {t}, \tag {4}
$$
where  $t_s$  and  $t_e$  represent the starting and end points of a time span, respectively. With the computed average performance indicator of each task, we can set a threshold to select the unsatisfactory tasks. The effectiveness of  $\hat{z}_i$  is validated in Sec. 5.2 and Sec. 5.3.
Grouping Unsatisfactory Tasks into Multiple Task Groups The number of identified unsatisfactory degradation tasks can still be significant, typically in the range of hundreds in our experiments. Given the potential variance among these tasks, it would be advantageous to further divide them into multiple task groups based on their similarity to reduce negative transfer. A common approach for task grouping in multi-task learning is to learn pairwise performance indicator between tasks [48, 10, 37], which is impractical for real-SR due to the large number of degradation tasks involved.
A feasible way is to learn an indirect measure of task similarity by fine-tuning the pre-trained real-SR network on each degradation task independently and computing the PSNR distance as described in Sec. 3 and Fig. 1. However, this approach is still time-consuming given the large number of unsatisfactory tasks. Another alternative of indirect measure is the average performance indicator in Eq. 4, which, although efficient, may not be accurate enough. Hence, we make a trade-off to fine-tune the pre-trained real-SR network  $\theta$  on all unsatisfactory tasks simultaneously through joint-training to obtain a new network  $\hat{\theta}$ . Then, we test the fine-tuned network  $\hat{\theta}$  on each degradation task with an available validation set and compute a performance improvement score (PIS) defined as:
$$
s _ {j} = I \left(\mathcal {D} _ {j} ^ {\text {v a l}}; \hat {\theta}\right) - I \left(\mathcal {D} _ {j} ^ {\text {v a l}}; \theta\right), \tag {5}
$$
where  $\mathcal{D}_i^{\mathrm{val}}$  represents the validation set of an unsatisfactory task  $\tau_{i}$ ,  $I$  is an IQA (image quality assessment) metric such as PSNR, and  $s_i$  is the PIS of task  $\tau_{i}$ . We then select the tasks with PIS larger than some threshold to form a task group, which should have small conflicts as they dominate the fine-tuning process. We repeat this process to find the rest of task groups as described in Alg. 1.
4.2 TGSR: A Task Grouping Based Real-SR Method
With the identified degradation task groups  $\mathcal{G} = \{G_1, G_2, \dots, G_c\}$ , we adopt a straightforward method to fine-tune the pre-trained real-SR network, referred to as TGSR. We first form  $\hat{\mathcal{G}} = \{G_0, \mathcal{G}\} = \{G_0, G_1, G_2, \dots, G_c\}$ , where  $G_0$  represents the entire degradation space and serves for preventing
Group 0
Group 1
Figure 3: Sample images from different degradation task groups in our DIV2K5G datasets.
catastrophic forgetting during the fine-tuning process. Then, we randomly select a group from  $\hat{\mathcal{G}}$  and randomly sample a task from the chosen group for fine-tuning. This approach effectively increases the inclusion of the unsatisfactory tasks during the fine-tuning process and weights their likelihood of being chosen based on their respective group size (i.e., tasks in smaller groups have a higher probability to be selected). In essence, it is similar to the task re-weighting approach commonly used in multi-task learning.
Datasets and evaluation. We employ DIV2K [1], Flickr2K [1] and OutdoorSceneTraining [40] datasets to implement our task grouping algorithm and train the TGSR network. For evaluation, we use DIV2K validation set to construct a DIV2K5G dataset consisting of 5 validation sets according to the divided 5 different degradation groups by the task grouping algorithm, as shown in Fig. 3. Each validation set contains 100 image pairs. In addition to the task group based test set, we employ the synthetic test set AIM2019 [29] and DIV2K_random for evaluation. The DIV2K_random is generated using the RealESRGAN degradation model on the DIV2K validation set. Furthermore, we incorporate the real-world test set, RealSR set [4], into our evaluation process. All evaluations are conducted on  $\times 4$  SR and PSNR is computed on the Y channel of YCbCr color space.
Implementation details. We adopt the high-order degradation model proposed by RealESRGAN [39] as the real-SR degradation model in our experiments. To compute the performance indicator,  $N$ $(4\times 10^{3})$  degradation tasks are sampled from the whole degradation space. The single-task network is fine-tuned from the pre-trained RealESRNet model for 100 iterations. The performance indicator is computed based on the average of the last 10 iterations considering the instability of the training procedure. To implement the degradation grouping, the degradation tasks with the top  $40\%$  higher performance indicators  $(1.6\times 10^{3}$  degradation tasks) are selected as unsatisfactory tasks. Then, we fine-tune the pre-trained RealESRNet for  $1\times 10^{4}$  iterations based on all unsatisfactory tasks. After that, we evaluate the model on Set14 [49] for each unsatisfactory degradation task. By using the proposed performance improvement score, we divide the degradation tasks into five groups based on the thresholds of [0.8, 0.6, 0.4, 0.2]. According to this set of thresholds, we obtain four groups with the number of tasks in each group being [14, 29, 84, 200] as groups1-4. Then, we label the entire degradation space beyond the four degradation groups as Group0. To avoid forgetting the satisfactory tasks, we uniformly sample degradation tasks from each group to fine-tune the pre-trained real-SR network. Other basic training settings follow RealESRGAN.
5.2 Comparison with State-of-the-Art
We compare our TGSR with the state-of-the-art methods, including SRGAN [24], ESRGAN [41], RDSR [23], MM-RealSR [31] BSRGAN [50], SwinIR [26], RealESRGAN [39], DASR [25] and HAT [6, 5]. Officially released pre-trained models are used for the compared methods.
Our TGSR improves the overall performance. The quantitative results of different methods are presented in Tab. 1. ESRGAN is based on a single-degradation model (i.e., Bicubic setting), so it performs worst under the multi-degradation evaluation system. RDSR is an MSE-based method, so it achieves the highest PSNR but performs rather poorly (the second worst) performance on LPIPS. BSRGAN, RealSwinIR, and DASR employ a one-order degradation model. They sacrifice performance on PSNR for better perceptual quality (reflected in low LPIPS). MM-RealSR and