Real-World Image Super-Resolution as Multi-Task Learning
Authors: Wenlong Zhang
Xiaohui Li
Guangyuan SHI
Xiangyu Chen
Yu Qiao
Xiaoyun Zhang
Xiao-Ming Wu
Chao Dong
Venue: NeurIPS 2023 poster (2023)
Keywords: Image super-resolution

Abstract: In this paper, we take a new look at real-world image super-resolution (real-SR) from a multi-task learning perspective. We demonstrate that the conventional formulation of real-SR can be viewed as solving multiple distinct degradation tasks using a single shared model. This poses a challenge known as task competition or task conflict in multi-task learning, where certain tasks dominate the learning process, resulting in poor performance on other tasks. This problem is exacerbated in the case of real-SR, due to the involvement of numerous degradation tasks. To address the issue of task competition in real-SR, we propose a task grouping approach. Our approach efficiently identifies the degradation tasks where a real-SR model falls short and groups these unsatisfactory tasks into multiple task groups. We then utilize the task groups to fine-tune the real-SR model in a simple way, which effectively mitigates task competition and facilitates knowledge transfer. Extensive experiments demonstrate our method achieves significantly enhanced performance across a wide range of degradation scenarios.

Real-Worke Image Super-Resolution as Multi-Task Learning
Wenlong Zhang! Xiaohui Li23, Guangyuan Shi', Xiangyu Chen Xiaoyun Zhang', Yu Qiao Xiao-Ming Wu'j Chao Dong 'The HongKong Polytechnic University SShanghai AI Laboratory Shanghai Jiao Tong University UUniversity of Macau SShenzhen Institute of Advanced Technology, CAS
wenlong zhang@connect wtwpeeooonnntttlol...ooeeeeeeme. polyu.hi xiao-ming. su@polyu edu. hk, chao. wewo.donndsitt.co
Abstract
In this paper, we take new look real-world image super-resolution (real-SR) from a multi-task learning perspective. We demonstrate that the conventional formulation of real-SR can be viewed solving multiple distinct degradation tasks using single shared model. This poses challenge known task competition or task conflict multi-task learning, where certain tasks dominate the learning process, resulting in poor performance on other tasks. This problem is exacerbated in the case of real-SR, due to the involvement of numerous degradation tasks. To address the issue of task competition in real-SR, propose task grouping approach. Our approach efficiently identifies the degradation tasks where a real-SR model falls short and groups these unsatisfactory tasks into multiple task groups. We then utilize the task groups to fine-tune the real-SR model in a simple way, which effectively mitigates task competition and facilitates knowledge transfer. Extensive experiments demonstrate our method achieves significantly enhanced performance across wide range of degradation scenarios. The source code is available at lhttps://ee tigghu.b..... com/ XP TePPPeeGrroo XPoxeeooooo/TTSR
Introduction
Real-world image super-resolution (real-SR) aims to enhance the resolution and quality of low- resolution images captured real-world scenarios. Real-SR algorithms enable improved image quality and better visual understanding, making them valuable in wide range of applications [43] 46
[4] [7]. Unlike non-bling SR [9] [6 (55) and classical blind SR [3 18 38) that assumes simple degradation process, real-SR deals with complex and unknown degradations present in real- world imaging conditions, such as noise, blur, compression artifacts, and sensor limitations. The diversity of real-world degradations and unknown degradation parameters make challenging reverse the specific degradation effects and accurately recover high-resolution details.
Prior studies tackle real-SR by designing various degradation models to simulate the degradation process real-world scenarios. For instance, they generate synthetic paired training data with a shuffled degradation model (BSRGAN [50)), high-order degradation model (RealESRGAN (39), or a three-level degradation model (DASR 251). ). Existing methods commonly train single real- SR network with training data generated by sophisticated degradation model, aiming to cover as many degradation cases as possible during the training process. Under this framework, recent works have focused enhancing various aspects of the real-SR network, including improving the backbone network architecture (26.. optimizing inference efficiency (25 enhancing generalization capabilities (3). and improving modulation ability B1]
*Coresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

This study takes a new look at real-SR from multi-task learning perspective. We show that the conventional formulation of real-SR essentially corresponds multi-task learning problem. which involves solving a large number of different degradation tasks simultaneously with a single shared model. Consequently, real-SR faces a well-known challenge in multi-task learning, namely task competition or task conflict [32]. It refers to the situation that tasks compete for model capacity, potentially resulting in certain tasks dominating the learning process and adversely affecting the performance of other tasks. This problem is severely aggravated in the case of real-SR, where large number of degradation tasks (up to thousands) are typically involved. In our pilot experiment (Fig. D we have observed that real-SR network trained with the multi-task objective fails to yield satisfactory results for a significant portion of degradation tasks.
We propase a task grouping approach (Fig. 2 to alleviate the negative impact of task competition for real-SR. Task grouping [48 37
is an effective technique in multi-task learning that helps mitigate negative transfer by training similar tasks together. This approach typically involves learning the relationship or relevance between pairs of tasks through validation or fine-tuning, which becomes impractical in the context of real-SR where there is large number of tasks to consider. Hence, we resort to design indirect measures to assess task affinity. Specifically, we introduce a performance indicator based on gradient updates to efficiently identify the degradation tasks that real-SR model falls short of. Then, we propose an algorithm to group these unsatisfactory tasks into multiple groups based on a performance improvement score. Finally, we propose TGSR, simple yet effective real-St SR method, which leverages the identified task groups to fine- tune the pre-trained real-SR network. Our key contributions summarized as follows:
35
10
We take a new look at real-SR from a multi-task learning perspective and highlight the task competition problem.
We propose task grouping approach to effectively address the task competition problem for real-SR and develop a task grouping based real-SR method (TGSR).
We conduct extensive experiments to validate the effectiveness of TGSR, and the results demonstrate its superior performance compared to state-of-the-art real-SR methods.
2 Related Work
Image super-resolution. Since Dong et al. [9] first introduced Convolutional Neural Networks (CNNs) to the SR problem, a series of learning- based SR methods have made great progress, including deep networks [2 dense connections [56]. channel attention of [55], residual-in -n--rsiduulll dense blocks [1], and transformer structure 26 To reconstruct realistic textures, Generative Adversarial Networks (GANs) 24 41 53 42 have been employed to generate visually pleasing results. However, these methods adopt a bicubic down-sampling degradation that insufficient for real-world images.
Real-world super-resolution. To address the SR problem in real-world scenarios, classical blind SR methods primarily use Gaussian blur and noise to model the distribution of real-world images. Significant progress has been made through a variety of approaches. including the use of a single SR network with multiple degradations [11] kernel estimation 13
and representation learning [38]. Additionally, BSRGAN 50 proposes a large degradation model that incorporates multiple degradations using a shuffled strategy, while RealESRGAN (39 employs a high-order strategy to construct large degradation model. DASR (5) adopts a three-level degradation distri- bution (i.e., two one-order and one high-order degradation models) to simulate the distribution of real-world images. These works demonstrate the potential of large degradation models for weal-world applications. In addition, recent works also improve real-SR in multiple dimensions, such as network
18
backbone [26). efficiency (25] generalization (23 54 52 and modulation ability [1].
Multi-task learning. Multi-task learning methods can be roughly divided into three categories: task balancing, task grouping, and architecture design. Task balancing [16] 20 47 27
34
[3 g 7 [9]
methods address task/gradient conflicts by re-whighting the loss or manipulating the update gradient.
Task grouping [48 37
35 10 methods mainly focus on identifying which tasks should be learned
together. Zamir et al. [88] provide a task taxonomy that captures the notion of task transferabilityy TAG [10] determines task groups by computing affinity scores that capture the effect between tasks. Architecture design methods can mainly be divided into hard parameter sharing methods [2]]28] 28
and soft parameter sharing methods [30 33 12
Hard parameter sharing methods require

---

Ianar usilly (bowc laght PSNKI
Sanehhmrry th degrdation
1.25
1.00
Unsatisdette
0.50
RC ehaeeyy Tidks Sane crer Teke
RR network
Single-osk netwerk
(a) PSNR distance between the real-SR and single-task SR networks across 100 randomly selected degradation tasks.
(b) Visual comparison between satisfac- tory and unsatisfactory tasks.
Figure I: Illustration of task competition. The jointly-trainee multi-task real-SR network falls short in producing satisfactory results for almost half of the degradation tasks, on which the fine-tuned single-task networks obtain more than 0.4dB performance gain, indicating these tasks are dominated by other tasks during the learning process.
different decoders for different tasks, while soft parameter sharing methods do cross-talk between different task networks. We focus on a task grouping strategy in our TGSR. The strategy identifies unsatisfactory degradation tasks in large degradation space. Subsequently, we can further improve the overall performance of the real-SR network by fine-tuning it in the unsatisfactory tasks.
Real-SR as a Multi-task Learning Problem
Problem Formulation Real-SR aims to restore high-resolution (HR) image from its low- resolution (LR) counterpart that has undergone an unknown and intricate degradation process:
y (x)) (fn
e fz)
where /: represents a degradation function such as Gaussian blurring with kernel width 10. 2.4]. adding Gaussian noise with noise level [1, 25], downsampling operation with scale factor or applying the JPEG compression. Hence, D represents vast continuous degradation space that encompasses an infinite number of degradations. Existing real-SR models such as BSRGAN (50). RealESRGAN [3]] and DASR (25) make different assumptions in the generation of D, aiming to simulate the highly complex degradation process in real-world scenarios.
An SR task 7 can be defined as a training pair (x: y = d(x)) formed by sampling a degradation from the degradation space D and applying it on an HR image I to generate LR image y. Due the infinite size of 'D it is impossible to consider every degradation case. Hence, common approach sample large number of degradations set of righutesolution images X, with N different degradations sampled from D (N >> |2'(), we can generate a of SR tasks:
sufficiently represent the degradation space. Given
{T, (I,, 3/4 J where I; X and yı is obtained by applying degradation on zy A real-SR model is commonly trained by minimizing the empirical risk:
Lioaal (0)
Σ C,(T))), 1=1
(2)
where L, is the loss on task T,, and 0 are the trainable model parameters shared by all N tasks. Therefore, real-SR is essentially multi-task learning problem, aiming solve N different SR tasks with single shared model. In the following, we refer to t; as a degradation task.
Task Competition From the view of multi-task learning, real-SR is challenging problem intends to solve large number of different degradation tasks (e.g.. N = 10²) altogether with single model, inevitably suffering from task competition that some tasks may dominate the training process leading to poor performance on other tasks [37].
To illustrate the impact of task competition, we randomly sample 100 degradation tasks with the degradation model in [39] and train a real-SR network with the multi-task objective in Eq. 2 Next, fine-tune the Sell-SR network each degradation task independently and obtain 100 fine-tuned

---

Jolnntl Degradation Tasks with a Single Network
Idcatity Unsatisfactory Degrudation Tasks
Group Unsatisfactory Degradation Taskk
Continue SR Network with Degradation Task Groups
Pre-trained
Netwurks
O
Traising
Cumpune the sverage performance indieatoo with Equ
Fine tuning
performarce imponement with Eq
Task Grouping Based
SR Nerwork (TGSR)
Degradation
Sariehetory degrdation tode Uneriafictery degralation tndos
Deprodaton space
Figure 2: Overview of our proposed task grouping approach for real-SR.
models, which we call single-task networks. We then compare the performance between the real-SR network and the single-task networks on each degradation task by computing their PSNR distance. The results in Fig. show that for nearly half of the degradation tasks, the PSNR distance between the real-SR network and single-task network exceeds 0.4dB, indicating that these tasks are not well solved by the real-$R network, which we refer to as unsatisfactory tasks. From an optimization perspective, the other tasks (those with PSNR distance less than 0. 0.4dB) dominate the learning process and are effectively solved by the real-SR network, which we refer to as satisfactory tasks.
Real-SR via Task Grouping
Our analysis in Sec. 3 shows that when a real-SR network is tasked with many degradation tasks, they may compete for model capacity interfere with each other, resulting in significant decline in performance for certain tasks. This phenomenon, commonly referred to as negative transfer (37), is well-known challenge in the field of multi-task learning, An effective approach mitigate negative transfer is task selection or task grouping (10 37 48 By finding groups of tasks that may benefit from training together, the interference among tasks can be minimized.
4.1 Which Tasks Should Be Learned Together for Real-SR?
The results in Fig. suggest that the satisfactory tasks should be grouped together. These dominant tasks may share certain characteristics or similarities that lead minimal task conflicts, making them more prominent in the training process. More importantly, considering that the real-SR network has effectively tackled these tasks and reached a satisfactory performance level, we may focus on the unsatisfactory tasks that present greater potential for improvement.
Efficiently Identifying Unsatisfactory Tasks To identify the unsatisfactory tasks, we may use the approach described in Sec. 3lc Fig. [ by comparing the performance between the jointly-traine multi-task real-SR network (referred as the pre-trained real-SR network hereafter) and the fine- tuned single-task networks. However, the cost of adopting this approach is prohibitive due to the large number of degradation tasks (c.g., thousands or tens of thousands) involved in real-SR.
Therefore, to efficiently identify the unsatisfactory tasks, we propose measure to assess whether degradation task has been well solved by the real-SR network, which done through fine-tuning the pre-trained real-SR network for small number of iterations (e.g.. 100) on the degradation task. Specifically, we assess the impact the gradient update of task т, on the shared parameters (i.e., the pre-trained real-SR network), by comparing the loss of T; before and after updating 0. We develop a performance indicator defined as:
Li(Te, 0%) #
C,(T;, 0)
where 0f are the parameters updated task т; at the time step I. Notice that high value of of indicates the loss of task 7; not significantly reduced after fine-tuning, suggesting that the pre-trained real-SR network has achieved good performance this task.

---

Algorithm 1: Degradation Task Grouping for Real-SR
Input: set of degradation tasks T = {TI, T2,-..; J. the pre-trained real-SR model 0, the number of groups c, and the threshold values to, (1, " fe.
for any 7, E T do
Compute the average performance indicator 2 with Eq. 4
end
// Select unsatisfactory tasks
Let / = [ > to):
// Group unsatisfactory tasks
for i =
Fine-tune the pre-trained real-SR network with all unsatisfactory tasks; for any Tj f do Compute the performance improvement score 8j with Eq. end
Let G; [Tyls} > 44h, where & is the threshold for group i;
Let T = TG;;
end
Output: Degradation task groups g {G1, G2. Goj.
For stability, we consider the average performance indicator during the fine-tuning process on task Tj:
2,
(4)
t=t.
where 1, and le represent the starting and end points of time span, respectively. With the computed average performance indicator of each task, we can set threshold to select the unsatisfactory tasks.
The effectiveness of 2; validated in Sec. 5.2 and Sec. 5.3
Grouping Unsatisfactory Tasks into Multiple Task Groups The number of identified unsatisfac- tory degradation tasks can still be significant, typically in the range of hundreds in our experiments. Given the potential variance among these tasks, it would be udvantageous to further divide them into multiple task groups based on their similarity to reduce negative transfer. common approach for task grouping in multi-task learning is to learn pairwise performance indicator between tasks 48 10, 37
which is impractical for real-$R due to the large number of degradation tasks involved.
A feasible way is to learn an indirect measure of task similarity by fine-tuning the pre-trained real-SR network on each degradation task independently and computing the PSNR distance described in Sec. 3 and Fig. [ However, this approach is still time-consuming given the large number of unsatisfactory tasks. Another alternative of indirect measure the average performance indicator in Eq. [ which, although efficient, may not be accurate enough. Hence, we make a trade-off to fine-tune the pre-trained real-SR network
all unsatisfactory tasks simultaneously through voint-training to obtain a new network Ô. Then, we test the fine-tuned network on each degradation task with an available validation set and compute performance improvement score (PIS) defined as:
8 = [DD) - (D)) 0),
(5)
where Dial represents the validation set of unsatisfactory task T. I is IQA (image quality assessment) metric such as PSNR, and 8; is the PIS of task т, We then select the tasks with PIS larger than some threshold form task group, which should have small conflicts as they dominate the fine-tuning process. We repeat this process to find the rest of task groups as described in Alg.
4.2 TGSR: A Task Grouping Based Real-SR Method
With the identified degradation task groups (G1, G2, .., Go), we adopt straightforward method to fine-tune the pre-trained real-SR network, referred to as TGSR. We first form ğ (Go,9) = (Go, G1, G2, Ge}. where Go represents the entire degradation space and serves for preventing
5

---

GT
Group 0
Group ]
Group 2
Group 3
Group 4
Figure 3: Sample images from different degradation task groups our DIV2K5G datasets.
catastrophic forgetting during the fine-tuning process. Then, we randomly select a group from 9 and randomly sample task from the chosen group for fine-tuning. This approach effectively increases the inclusion of the unsatisfactory tasks during the fine-tuning process and weights their likelihood of being chosen based on their respective group size (i.e., tasks smaller groups have a higher probability to be selected). In essence, it is similar to the task re-weighting approach commonly used multi-task learning.
5 Experiments
5.1 Experimental Setup
Datasets and evaluation. We employ DIV2K D. Flickr2K I and ButdoorSceneTrriniie datasets to implement our task grouping algorithm and train the TGSR network. For evaluation, we use DIV2K validation set to construct a DIV2K5G dataset consisting of 5 validation sets according to the divided different degradation groups by the task grouping algorithm, as shown in Fig. 3 Each validation set contains 100 image pairs. In addition to the task group based test set, we employ the synthetic test set AIM2019 29 and DIV2K random for evaluation. The DIV2K random is generated using the RealESRGAN degradation model on the DIV2K validation set. Furthermore, we incorporate the real-world test set, RealSR set 4 into our evaluation process. All evaluations are conducted on x4 SR and PSNR is computed on the Y channel of YCbCr color space.
40
Implementation details. We adopt the high-order degradation model proposed by RealESRGAN (39) as the real-SR degradation model in our experiments. To compute the performance indicator, N (4 x 10²) degradation tasks are sampled from the whole degradation space. The single-task network is fine-tuned from the pre-trained RealESRNet model for 100 iterations. The performance indicator computed based on the average of the last 10 iterations considering the instability of the training procedure. To implement the degradation grouping, the degradation tasks with the top 40% higher performance indicators (1.6 x 10y degradation tasks) are selected as unsatisfactory tasks. Then, we fine-tune the pre- praitre RealESRNet for 1 x 10* iterations based on all unsatisfactory tasks. After that, we evaluate the model on S [49 for each unsatisfactory degradation task. By using the proposed performance improvement score, we divide the degradation tasks into five groups based on the thresholds of [0.8, 0.6, 0.4, 0.2). According to this set of thresholds, we obtain four groups with the number of tasks in each group being [14, 29, 84, 200] groups1-4 Then, we label the entire degradation space beyond the four degradation groups as GroupO. To avoid forgetting the satisfactory tasks, we uniformly sample degradation tasks from each group to fine-tune the pre-trained real-SR network. Other basic training settings follow RealESRGAN,
5.2
Comparison with State-of-the-Art
We compare our TGSR with the of methods, including SRGAN 24), ESRGAN 41)
RDSR [23], MM-RealSR [3I] BSRGAN [50]. SwinIR (26). RealESRGAN 39), DASR 25 and
HAT 6 Officially released pre-trained models are used for the compared methods.
Our TGSR improves the overall performance. The quantitative results of different methods are presented in Tab. [ ESRGAN is based on single-degradation model (i.e., Bicubic setting), so it performs worst under the multi-degradation evaluation system. RDSR is an MSE-based method, so it achieves the highest PSNR but performs rather poorly (the second worst) performance on LPPPS. BSRGAN, RealSwinIR, and DASR employ a one-order degradation model. They sacrifice performance on PSNR for better perceptual quality (reflected in low LPIPS).
and

---

Table 1: Quantitative results of different methods on DIV2K5G. Group0 denotes the validation set with satisfactory degradation tasks, and Groupl represent the validation sets with unsatisfactory degradation tasks. The ESRGAN trained on the non-blind setting and RDSR trained on the MSE- based setting are marked in gray.
GroupO
Groupl
Group2
Group3
Group4
PSNR
L.IPS
PSNR
I.PISS
PSNR
LPIPS
PSNR
LPIPS
PSNR
LPIPS
ESRGAN
21.49
0.6166
20.29
0.6697
20.95
0.6530
22.42
0.5940
22.02
0.5837
RDSR
25.00
0.5196
21.18
0.6167
23.06
0.5586
25.00
0,4993
25.58
0.4814
MM-RealSE
23.49
0.4549
19.74
0.5326
21.76
0.4731
23.37
0,4205
23.97
0.3989
DASR
23.87
0.4683
19.97
0.5752
22.31
0.5093
24.00
0.4474
24.85
0.4225
BSRGAN
23.99
0.4549
20.07
0.5799
22.45
0.4961
24.34
0,4388
24.80
0.4210
ReaISRGAN
23.88
0.4599
20.10
0.5586
22.12
0.5019
23.85
0.4499
24.48
0.4270
RealSRGAN-TG
23.95
0.4617
21.14
0.5323
23.06
0.4802
24.69
0.4248
25.05
0.4112
RealESRGAN
23.85
0,4325
20.10
0,5355
22.07
0,4701
24.30
0,4147
24.58
0,3970
RealESRGAN-TG
23.99
0,4286
21.10
0,5056
23.15
0.4494
24.62
0,3975
25.03
0,3851
RealSwinIR
23.35
0.4468
19.60
0.5624
21.65
0.4905
23.66
0,4265
24.16
0.4077
RealSwinIR- -GG
23.90
0.4168
20.62
0.4925
22.70
0.4377
24.56
0.3815
24.98
0.3670
RealHAT
24.26
0.4084
20.64
0,5022
22.44
0.4468
24.42
0.3918
25.02
0.3734
RealHAT-TG
24.31
0.4110
21.41
0,4932
23.20
0.4395
24.87
0.3841
25.32
0.3674
STATE BANK MARKET
STATE BANK MARKET
STATE BANK MARKET
STATE BANK MARKET
STATE BANK MARKET
STATE BANK MARKET
STATE BANK MARKET
STATE BANK MARKET
Inpor
USRGAN
R.DSR
MM ReUSR
BSRGAN
RealswatR
DANR
ReOlESRGAN
RealERRGAN-TE
(ouzs)
Figure 4: Qualitative results different methods. Zoom in for details.
RealESRGAN utilize a more complex high-order degradation. As result, they achieve better LPIPS for real-SR evaluation. However, the two approaches obtain low PSNR performance due to the great difficulty of optimizing. Notably, we can enhance the performance by fine-tuning it on the identified Task Groups (TG). This improvement is particularly significant in Groups 1-4, For instance, on RealESRGAN-TG, we can observe a maximum boost of I dB PSNR and 0.03 in LPIPS.
pre-trained real-SR network
Our TGSR demonstrates significant superiority when applied large degradation space. It is noticeable that RealESRGAN-TG outperforms RealESRGAN even in GroupO. This indicates that our approach enhances real-SR performance across nearly the entire degradation space, not just the specifically addressed Groups 1-4. In addition to the identified task groups, our TGSR can also achieve performance gains on the randomized synthetic test sets DIV2K_randon and AIM2019 test set as shown in Tab. 2 Additionally, our method achieves gain on the real scene test set RealSRset. These results clearly demonstrate that our approach does not compromise the performance of certain degradation tasks to enhance the performance of others.
7

---

Table 2: Quantitative results of different methods on the real-world test set. The ESRGAN trained on the non-blind setting and RDSR trained on the MSE-basee setting are marked in gray.
DIV2K
random LPIPS
AIM2019
RealSRsct-Nikon
RealSRset-Cam
PSNR
PSNR
LPIPS
PSNR
LPIPS
PSNR
LPIPS
ESRGAN
20.63
0.6345
23.16
0.5500
27.40
0.4132
27.73
0.4054
RDSR
24.61
0.5268
24.44
0,4803
26.39
0,4053
26.93
0.3795
MM-RealSR
23.17
0.4394
23.48
0.3917
23.78
0.3841
24.42
0.3664
DASR
23.52
0.4832
23.76
0.4210
26.68
0.3972
27.68
0.3792
BSRGAN
23.76
0.4622
24.20
0.4000
26.11
0.3900
26.90
0.3648
SwinIR
23.13
0.4432
23.89
0.3870
26.20
0.3616
26.68
0.3469
RealESRGAN
23.54
0.4423
23.89
0.3960
25.62
0.3820
26.06
0,3629
ReaISRGAN
23.58
0,4710
23.72
0.4247
24.71
0,4159
25.42
0.3902
ReaISRGAN-TG
23.79
0.4705
23.97
0.4174
25.18
0.4018
25.93
0.3759
RealESRGAN
23.54
0,4423
23,89
0.3960
25.62
0,3820
26.06
0.3629
RealESRGAN-TG
23.84
0.4368
24.27
0.3899
26.01
0,3819
26.33
0.3637
RealSwinIR
23.67
0,4216
23.98
0.3804
25. 25.70
0.3700
26.43
0.3506
RealSwinIR-TE TG
23.74
0.4190
24.10
0.3766
26.36
0.3751
27.18
0.3585
RealHAT
24.04
0.4156
24.19
0.3742
25.88
0.3532
26.56
0.3339
RealHAT-TG
24.21
0,4189
24.41
0.3723
26.12
0.3635
26.69
0.3451
Enpux
RealLLAT
RealHATTGG
GT
Figure 5: RealSwinIR and RealHAT may unreasonably generate unpleasant artifacts or semantic textures that shouldn't be there. However, our method does have this problem.
Our TGSR obtains better visual results than other methods. The visual results of different methods are shown in Fig. and Fig. TGSR significantly improves existing methods by removing various degradation artifacts and noticeable unpleasant artifacts. For the first four rows of images with complex degradation, the other methods cannot remove the degradation generate unacceptable artifacts. In contrast, our TGSR handles the degradation well and produces visually pleasant results. For the image on the last row, our method can generate more realistic results than other methods with clear textures. In the RealSwinIR results of Fig. [5] we can observe the unpleasant artifacts in the flat area of Flower image and the fur of cat image. In the RealHAT results, we also find semantic textures appearing where they shouldn't 't be. For example, tree texture appears on the train, and fur-like textures are present the background of the Wolf image. However, our method can remove these artifacts by fine-tuning the real-SR network on identified task groups.
5.3 Ablation and Analysis
(a) and (b), we present the PSNR improvement achieved by fine-tuning the real-SR network on individual tasks with lower and higher performance indicators. can be observed that tasks with higher performance indicators exhibit significant PSNR improvement (about 0.4- D.4B), while tasks with lower performance indicators
Effectiveness the performance indicator. 1) In Fig. 6

---

1-2-
1.2
PSNR (itunce(
1.0-
PSNR distance
1.0
0.8-
0.8
0.4
0.4
0.2
0.2
0.0
0.0
10
Degradation
Degradation
SR aetwoods
Single
(a) Lower performance indicator (b) Higher performance indicator
(c) Visual comparison
Figure 6: Performance comparison of real-SR networks that trained on the degradation tasks with (a) lower performance indicators and (b) higher performance indicators. The degradation tasks with a lower performance indicator can be further improved in quantitative and qualitative results.
1.
TGSR
0.30
TGSR: Rusioo
0.8
0.20
I
2.6
0015
0.4
0.2
Salderion
Valibalen
ResDESKAAN
TOSH
(a) Comparison on PSNR.
(b) Comparison on LPIPS.
(c) Visual comparison
Figure 7: Performance comparison of RealESRGAN and our TGSR/ TGSR with random grouping on (a) PSNR and (b) LPIPS. The results indicate random grouping achieves limited improvement compared with our proposed task grouping. (c) Visual results demonstrate that random grouping chooses a satisfactory degradation task that is not required for further training, while our task grouping method finds an unsatisfactory degradation task that needs be further improved.
show only a small PSNR improvement (about 0.2 dB). This finding indicates that our proposed performance indicator can effectively distinguish between satisfactory and unsatisfactory tasks for a real-SR model in a large degradation space. Additionally, (c) illustrates that significant visual improvements can be achieved with the unsatisfactory degradation task. 2) Directly fine-tuning a model for each task incurs high computational cost (empirically requiring least 10,000 iterations). However, our performance indicator requires only 100 iterations of fine-tuning. It suggests that utilizing the performance indicator can be 100 times faster than direct fine-tuning for distinguishing the unsatisfactory and satisfactory degradation tasks. This further illustrates the superiority of the proposed performance indicator.
Effectiveness of our task grouping algorithm. We compare the performance RealESRGAN-TO with our task grouping and random grouping to demonstrate the effectiveness of our method. As shown in Fig. 7 (a) and (b), random grouping can only bring very limited performance gains of about 0.2dB on PSNR and about 0.01 on LPIPS. In contrast, the groups generated by our task grouping algorithm show a significant improvement up to 1dB on PSNR and 0.03 on LPIPS. These results demonstrate the significance of well-designed task grouping approach. Furthermore, Fig. 7 (c) shows that random grouping can not select the unsatisfactory task, while our task grouping algorithm can find it.
Study of the performance upper bound. (1) We randomly select 100 single tasks and then add the corresponding small range of similar degradation parameters to each task, named single-task network with range (see details in appendix). Fig. shows that the existence of related tasks can improve the upper bound of some tasks (about 40%). However, there has been performance drop for other tasks due to similar degradation parameters that may not strictly represent similar tasks. (2) To further show the superiority of our TGSR, we fine-tune the RealESRGAN model on each group to obtain

---

Table 3: Ablation experiments the performance upper bound. We fine-tune RealESRGAN directly on each task group to get the corresponding performance upper bound, denoted RealESRGAN-SG.
Staglocall setevid
Mg A Aeot py papon
Metries
Model
Group0
Groupl
Group2
Group3
Group4
RealESRGAN
23.85
24.58
24.00
22:07
20.10
Valcsse 1911
PSNR (t)
RealESRGAN- SG
23.63
24.58
24.39
22.67
20.46
RealESRGAN-TO (ours)
23.88
25.08
24.62
23.11
21.06
Figure 8: Performance compari- son of SR network with single task and a range of tasks.
RealESRGAN
0.4325
0.3970
0.4147
0.4701
0.5355
(1)
RealESRGAN
0.3792
0.4478
RealESRGAN- (ours)
0.4291
0.3845
0.3981
0.4524
0.5087
their empirical performance upper bound for these degradation groups, and the results are denoted as RealESRGAN-SG. As presented in Tab. the fine-tuned models exceed the baseline models and obtain significant performance improvement on LPIPS. Although RealESRGAN-TG cannot surpass the empirical upper bound performance on LPIPS, it still achieves comparable performance. Moreover, our RealESRGAN-TG obtains higher PSNR compared to RealESRGAN-SG. This further shows the superiority of our method and suggests that multi-task learning enhances the performance of the specific task.
Table 4: Quantitative results of different methods DIV2K8G. Groupo denotes the validation set with satisfactory degradation tasks, and Group1-7 represents the validation sets with unsatisfactory degradation tasks. The ESRGAN trained on the non-blind setting and RDSR trained on the MSE- based setting are marked in gray.
Metrics
ESRGAN
RDSR
BSRGAN
Real-SwinIR
DASR
MM-RealSR
RealESRGAN
RealESRGAN (ours)
Group0
PSNR (f)
21.39
25.12
24.19
23.61
23.97
23.61
24.08
24.10
LPIPS (1)
0.6251
0,5181
0.4567
0.4430
0.4697
0.4335
0.4297
0.4262
Groupl
(1)
19.96
19.77
19.52
LPIPS ())
0.6812
0.6325
0.6151
0.5954
0.6012
0,5572
0.5584
0.5249
Group2
PSNR (t)
21.10
22.55
21.91
21.16
21.77
21.39
21.67
22.72
LPIPS (1)
0.6636
0.5755
0.5091
0.4988
0.5175
0.4814
0.4823
0,4652
Group3
PSNR (f)
21.02
23.82
23,24
22.66
23.06
22.54
22.94
23.93
I.PIS ())
0.6301
0.5225
0.4657
0.4528
0.4795
0.4445
0.4390
0.4188
Group4
PSNR (f)
23.96
23,20
LPIPS ())
0.6221
0,5171
0.4617
0.4429
0.4596
0.4335
0.4294
0..0997
Groups
PSNR (f)
22.78
25.30
24.64
23,99
24.49
23.61
24.36
25.14
LPIPS (1)
0,5705
0.4873
0.4248
0.4113
0.4245
0.4072
0.4043
0.3821
Group6
PSNR (f)
22.42
25.32
24.58
23.87
24.52
23.71
24.38
24.86
LPIPS (1)
0.5849
0.4917
0.4255
0.4153
0.4269
0.4060
0.4039
0.3889
Group7
PSNR (1)
25.37
24.50
23.85
24.43
23.74
24.76
LPIPS (1)
0,5868
0.4955
0.4337
0.4227
0.4398
0,4110
0.4083
0.3976
Impact of the number of task groups. In Tab. 4 we group the unsatisfactory tasks into more
compact new groups using new thresholds of [0.8. 0.7, 0.6, 0.5, 0.4, 0.3, 0.2], with the corresponding number of degradation tasks being [14, 15, 14, 31, 53, 116, 258]. The results demonstrate that our TGSR method exhibits steady improvement on the unsatisfactory tasks in Groups 1-7, while achieving comparable performance on the satisfactory tasks in Group O. This indicates that our approach can be applied with a more fine-grained division of unsatisfactory tasks based on user requirements.
Conclusion
We have re-examined the real-SR problem through the lens of multi-task learning and introduced novel approach, TGSR, to address task competition in real-SR. TGSR aims identify and enhance the degradation tasks where a real-SR model underperforms. It involves a task grouping method and a simple fine-tuning approach using the identified task groups. Comprehensive evaluation confirms the effectiveness of our approach in producing high-quality super-resolution results in real-world scenarios. One potential limitation of our method the need for adequate sampling from the degradation space.
10

---

Acknowledgments and Disclosure of Funding
This work was supported in part by the National Natural Science Foundation of China under Grant (62276251,62272555) the Joint Lab of CAS-HK, the National Key RD Program of China (NO. 20222D0160100,, and in part by the Youth Innovation Promotion Association of Chinese Academy of Sciences (No. 2020356).
References
[1] Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR) Workshops, volume 3, page 2, 2017.
[2] Sefi Bell-Kligler, Assaf Shocher, and Michal Irani. Blind super-resolution kernel estimation using an internal-gan. Advances in Neural Information Processing Systems, 32, 2019.
[3] Felix JS Bragman, Ryutaro Tanno, Sebastien Ourselin, Daniel C Alexander, and Jorge Cardoso. Stochastic filter groups for multi-task cnns: Learning specialist and generalist convolution kernels. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1385 1394, 2019.
[4] Jianrui Cai, Hui Zeng, Hongwei Yong, Zisheng Cao, and Lei Zhang. Toward real-world single image super-resolution: new benchmark and a new model. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 3086-3095, 2019.
[5] Xiangyu Chen, Xintao Wang, Wenlong Zhang, Xiangtao Kong, Yu Qiao, Jiantao Zhou, and Chao Dong. Hat: Hybrid attention transformer for image restoration, arXiv preprint arXiv:2309.055: 05239, 2023.
[6] Xiangyu Chen, Xintao Wang, Jiantao Zhou, Yu Qiao, and Chao Dong. Activating more pixels in image super-resolution transformer. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 22367-22377 2023.
[7] Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. Gradnorm: Gra- dient normalization for adaptive loss balancing in deep multitask networks. In International conference on machine learning, pages 794-803. PMLR, 2018.
[8] Zhao Chen, Jiquan Ngiam, Yanping Huang, Thang Luong, Henrik Kretzschmar, Yuning Chai, and Dragomir Anguelov. Just pick a sign: Optimizing deep multitask models with gradient sign dropout. Advances in Neural Information Processing Systems, 33:2039- 2050, 2020.
[9] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Learning deep convolutional network for image super-resolution. European conference on computer vision, pages 184-199. Springer, 2014.
[10] Chris Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn. Efficiently identifying task groupings for multi-task learning. Advances in Neural Information Processing Systems, 34:27503-272 27516, 2021.
[11] Yuan Gao, Haoping Bai, Zequn Jie, Jiayi Ma, Kui Jia, and Wei Liu. Mtl-nas: Task-agnostic neural architecture search towards veneral-purposs. multi-task learning. In Proceedings of the IEEE/CVE Conference on computer vision and pattern recognition, pages 11543-11552, 2020.
[12] Yuan Gao, Jiayi Ma, Mingbo Zhao, Wei Liu, and Alan L Yuille. Nddr-enn: Layerwise feature fusing in multi-task cnns by neural discriminative dimensionality reduction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3205-3214, 2019.
[13] Jinjin Gu, Hannan Lu, Wangmeng Zuo, and Chao Dong. Blind super-resolution with iterative kernel correction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1604-1613, 2019.

---

[14] Yuchao Gu, Xintao Wang, Liangbin Xie, Chao Dong, Gen Li, Ying Shan, and Ming-Ming Cheng. Vqfr: Blind face restoration with vector-quantized dictionary and parallel decoder, In Computer Vision-ECCC 2022: 17th European Conference, Tel Avix, Israel, October 23-27, 2022, Proceedings, Purt XVIII, pages 126-143. Springer, 2022.
[15] SHI Guangyuan, Qimai Li, Wenlong Zhang, Jiaxin Chen, and Xiao- Xiao-Ming Wu. Recon: Reducing conflicting gradients from the root for multi-task learning. In The Eleventh International Conference on Learning Representations.
[16] Michelle Guo, Albert Haque, De-An Huang, Serena Yeung, and Li Fei-Fei. Dynamic task prioritization for multitask learning. In Proceedings of the European conference on computer vision (ECCV), pages 270-287, 2018.
[17] Weidong Hu, Wenlong Zhang. Shi Chen, Xin Lv, Dawei An, and Leo Ligthart. deconvolution technology of microwave radiometer data using convolutional neural networks. Remote Sensing, 10(2):275, 2018.
[18] Yan Huang, Shang Li, Liang Wang, Tieniu Tan, al. Unfolding the alternating optimization for blind super resolution. Advances in Neural Information Processing Systems, 33:5632-5643, 2020.
[19] Adrián Javaloy and Isabel Valera. Rotograd: Gradient homogenization in multitask learning. In ICLR, 2022.
[20] Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semanties. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 7482-7491, 2018.
[21] Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1646-1654, 2016.
[22] lasonas Kokkinos. Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 61 6129-6138, 2017.
[23] Xiangtao Kong, Xina Liu, Jinjin Gu, Yu Qiao, and Chao Dong. Reflash dropout in image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6002-6012, 2022.
[24] Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew P Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic single image super-resolution using generative adversarial network. In CVPR, volume 2, page 4, 2017.
[25] Jie Liang, Hui Zeng, and Lei Zhang. Efficient and iegradattionadaapiin network for real- world image rusor-resolution. In Computer Eitio-ECCVV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVIII, pages 574-591. Springer, 2022.
[26] Jingyun Liang. Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In IEEE International Conference on Computer Vision Workshops, 2021.
[27] Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. Advances in Neural Information Processing Systems, 34: 18:88898-18880, 2021.
[28] Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Philip S Yu. Learning multiple tasks with multilinear relationship networks. Advances in neural information processing systems, 30. 2017.
12

---

[29] Andreas Lugmayr, Martin Danelljan, Radu Timofte, Manuel Fritsche, Shuhang Gu, Kuldeep Purohit, Praveen Kandula, Maitreya Suin, AN Rajagoapalan, Nam Hyung Joon, al. Aim 2019 challenge on real-world image super-resolution: Methods and results. In 2019 IEEF/CVF International Conference on Computer Vision Workshop (ICCVW), pages 3575-3583 IEEE, 2019.
[30] Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hehert. Cross-stitch networks for multi-task learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3994-4003, 2016.
[31] Chong Mou, Yanze Wu, Xintao Wang, Chao Dong, Jian Zhang, and Ying Shan. Metric learning based interactive modulation for real-world super-resolution. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVII, pages 723-740. Springer, 2022.
[32] Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05090 2017.
[33] Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, and Anders Søgaard. Latent multi- task architecture learning. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 33, pages 4822-4829, 2019.
[34] Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. Advances in neural information processing systems, 31, 2018
[35] Jiayi Shen, Xiantong Zhen, Marcel Worring, and Ling Shao. Variational multi-task learning with gumbel-softmax priors. Advances in Neural Information Processing Systems, 34:21031-21042, 2021.
[36] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR, 2015.
[37] Trevor Standley, Amir Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together multi-task learning? In International Conference on Machine Learning, pages 9120-9132. PMLR, 2020.
[38] Longguang Wang, Yingqian Wang. Xiaoyu Dong, Qingyu Xu, Jungang Yang. Wei An, and Yulan Guo. Unsupervised degradation representation learning for blind super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10581-10590, 2021.
[39] Xintao Wang, Liangiin Xie, Chao Dong, and Ying Shan. Real-esrgan: Training real-world blind super-resolution with pure synthetic data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1905-1914, 2021.
[40] Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy. Recovering realistic texture in image super-resolution by deep spatial feature transform. arXiv preprint arXiv: 1804.02815, 2018.
[41] Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Chen Change Loy, Yu Qiao, and Xiaoou Tang. Esrgan: Enhanced super-resolution generative adversarial networks. 2018.
[42] Zhang Wenlong, Liu Yihao, Chao Dong, and Yu Qiao. Ranksrgan: Generative adversarial networks with ranker for image super-resolution. IEEF Transactions on Pattern Analysis and Machine Intelligence, 2021.
[43] Yanze Wu, Xintao Wang, Gen Li, and Ying Shan. Animesr: Learning real-world super-resolution models for animation videos. Advances in Neural Information Processing Systems.
[44] Liangiin Xie, Xintao Wang, Chao Dong, Zhongang Qi, and Ying Shan. Finding discriminative filters for specific degradations in blind super-resolution. Advances in Neural Information Processing Systems, 34:51-61, 2021.
13

---

[45] Liangbin Xie, Xintao Wang, Shuwei Shi, Jinjin Gu, Chao Dong, and Ying Shan. Mitigating artifacts in real-world video super-resolution models. arXiv preprint arXiv:2212.07339, 2022.
[46] Fanghua Yu, Xintao Wang, Mingdeng Cao, Gen Li, Ying Shan, and Chao Dong. Osrt: Omnidirectional image super-resolution with distortion-aware transformer. arXiv preprint arXiv:2302.034553 2023.
[47] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33:5824-5836, 2020.
[48] Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3712-3722, 2018.
[49] Roman Zeyde, Michael Elad, and Matan Protter. On single image scale-up using sparse- representations. In International conference on curves and surfaces, pages 711-730. Springer, 2010.
[50] Kai Zhang, Jingyun Liang. Luc Van Gool, and Radu Timofte. Designing practical degradation model for deep blind image super-resolution. arXiv preprint arXiv:2103. 2021.
[51] Kai Zhang, Wangmeng Zuo, and Lei Zhang. Learning a single convolutional super-resolution network for multiple degradations. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 3262-3271, 2018.
[52] Wenlong Zhang, Xiaohui Li, Xiangyu Chen, Yu Qiao, Xiao-Ming Wu, and Chao Dong. Seal: framework for systematic evaluation of real-world super-resolution arXiv preprint arXiv:2309.0300 00000,,,0, 2023.
[53] Wenlong Zhang, Yihao Liu, Chao Dong, and Yu Qiao. Ranksrgan: Generative adversarial networks with ranker for image super-resolution. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 3096-31060 2019.
[54] Wenlong Zhang, Guangyuan Shi, Yihao Liu, Chao Dong, and Xiao-Ming Wu. A closer look at blind super-resolution: Degradation models, baselines, and performance upper bounds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 527-536, 2022.
[55] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image super- resolution using very deep residual channel attention networks. In ECCV, 2018
[56] Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. Residual dense network for image super-resolution. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018
14

---

7 Appendix
7.1 More Details of TGSR
In this section, we provide more training details of our TGSR. As mentioned in the main paper, we follow the RealESRGAN setting to train the real-SR network by standard BAN-based SR loss, including GAN loss LGAN, L1 loss L1, and perceptual loss per. The total loss can be defined as follows:
wn LL, + U2 Leer + wy LCAN;
The loss weights W1, W2, and W3 are set to 1. 1, and 0.1, respectively. Specifically, for Lt,, we calculate the pixel loss the Lı distance I y, where I and y denote the reconstructed HR image and the ground-truth HR image respectively. For the perceptual loss Leer. we extract the conv I, conv2, conv3, conv4, conv5 feature maps of and y using the pre-trained VGG19 network [36), and the loss is calculated as the weighted sum of the respective L1 distances between the feature maps of ÿ and with weights set [0.1. 0.1, 1, I, 1] for each layer. For the adversarial loss LGAN, we use U-Net discriminator with spectral normalization.
7.2
More Ablation Studies and Additional Experimental Results
7.2.1 Impact of the Increased Task Volume
In the main paper, conducted experiments with total of 4,000 (4k) degradation tasks. In this subsection, we have increased the number of tasks to 10,000 (10k) by sampling from the degradation space, in order to investigate the impact of the increased task volume on the results. In Fig. ? draw the histograms of the performance indicators computed with 4k (a) and 10k (b) degradation tasks respectively. It can be seen that their distributions are quite similar.
0.68
0.58
0.06
0.06
Podeistly
0.64
9.04
0.02
0.00
0.00
indicsior
(a)
(b)
Figure 9: The histograms of the performance indicators with 4k (a) and 10k (b) degradation tasks respectively.
Furthermore, use the same setup in the main paper to do task grouping with the 10k degradation tasks and generate the test set for each group. We then compare the performance of TGSR trained with the 10k degradation tasks (TGSR_ 10k) and the 4k degradation tasks (TGSR 4k). In Tab. 5 we observe that the performance of TGSR 4k on the dataset DIV2K5G _OK is comparable to that of TGSR 10k. This finding suggests that increasing the number of tasks from 4,000 to 10,000 does not lead to significant improvements performance for TGSR.
7.2.2 Necessity of Iterative Task Grouping
In Tab. 6. we provide the changes in performance improvement scores for various tasks across four training loops in the task grouping process in Alg. [ It can be observed that many tasks show variations in their performance improvement scores across different training loops. For example, degradation task 5 exhibits a performance improvement of 0.57dB in the first training loop and 0.7 0.72dB in the third loop. This suggests that certain tasks are given priority during training in each loop, which leads to limited performance improvement for other tasks due to task competition. The observation supports the necessity of iterative task grouping outlined in Alg.
15

---

Table 5: Quantitative results of TGSR trained with the 4k and 10k degradation tasks, respectively. GroupO denotes the validation set with satisfactory degradation tasks, and Groupl-4 represents the validation sets with unsatisfactory degradation tasks. The suffix 4k and 10k represent groups generated with the 4k and 10k degradation tasks, respectively.
Metrics
Group0_4k
Group1_4k
Group2_4k
Group3_44 4k
Group4_44 __
PSNR (1)
23.85
20.11
22.08
24.00
24.59
RealESRGAN
LPIPS (L)
0.4325
0.5355
0.4701
0.4147
0.3970
PSNR (1)
23.98
21.10
23.15
24.62
25.03
RealESRGAN-TG.44
I.IPS (L)
0.4286
0,5056
0.4494
0.3975
0.3851
PSNR (1)
23.91
21.03
22.98
24.50
24.97
RealESRGAN-TG.
LPIPS (L)
0.4299
0.5064
0.4525
0,4002
0,3859
Metrics
GroupO_ 10k
Groupl_ 10k
Group2_
Group3_ 10k
Group4_
PSNR (t)
23.81
19.85
20.95
23.77
24.38
RealESRGAN
LPIPS (L)
0.4324
0.5495
0.5073
0,4237
0.4019
PSNR (1)
24.12
20.90
21.85
24.38
25.03
RealESRGAN-TG
LPIPS (L)
0.4272
0.5187
0.4890
0.4046
0.3875
PSNR (1)
24.05
20.94
21.85
24.31
24.93
RedESRGAN-TG_LOK
LPIPS (1)
0.4285
0.5171
0.4907
0.4052
0.3886
Table 6: Performance improvement score of some degradation tasks across four training loops. NA means that the task is not available, since it has been grouped in the previous loop.
Training loop
Deg.1
Deg.2
Deg.3
Deg.4 4
Deg.5
Deg.6
Deg.7
Deg.8
Deg.9
Deg.10
Trainl
0.45
1.50
0.60
0.66
0.57
0.75
0.55
0.60
0.61
0.46
Train2
0.47
NA
0.63
0.67
0.41
0.76
0.49
0.59
0.64
0.48
Thain3
0.48
NA
NA
NA
0.72
NA
0.65
0.63
NA
0.48
Train4
NA
NA
NA
NA
NA
NA
NA
NA
NA
NA
Degradation
Deg.11
Deg.12
Deg.13
Deg.14
Deg.15
1eg.16
Deg.17
Deg.18
Deg.19
Deg.20
Trainl
0.43
0.60
0.47
0.27
0.66
0.53
0.22
0.31
0.63
0.20
Train2
0.42
0.59
0.47
0.33
0.51
0.52
0.18
0.31
0.67
0.21
Thain3
0.48
0.63
0.50
0.31
0.73
0.54
0.18
0.37
NA
0.25
Train4
NA
NA
NA
0.35
NA
NA
0.23
0.46
NA
0.26
7.2.3 Comparison with Single-task Network
is commonly believed that a single-task network fine-tuned specifically for a particular degradation task can achieve the upper bound of performance for that task. However, in Fig. [0] we compare the PSNR improvement over RealESRGAN between our TGSR and the single-task network on set of randomly selected tasks. Interestingly, we observe that the performance of the single-task network even decreases for tasks and terms of PSNR, while our TGSR, which utilizes task grouping for multi-task learning, consistently improves performance actoss all tasks. On average, TGSR achieves a PSNR improvement O. 74dB and an LPIPS improvement of 0.04, whereas the single-task network only achieves PSNR improvement of 0.30dB and an LPIPS improvement of 0.03. This demonstrates the effectiveness of the task grouping and multi-task learning strategy in TGSR, which leverages the shared information across tasks to improve the generalization ability of the model and achieve better overall performance.
7.2.4 Impact of the Increased Volume of Unsatisfactory Tasks
Tab. T presents the results of TGSR with additional unsatisfactory tasks. We categorize the unsatisfied tasks into groups based on thresholds [0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2, 05, 0. (0. 15 and 0.1 are used include more unsatisfactory tasks compared Tab. 4). with the corresponding number of degradation tasks as [14, 15, 14, 31, 116, 258, 190, 206). We can observe that TGSR consistently achieves performance improvement on the unsatisfactory tasks in Groups 1-9 while maintaining comparable results on the satisfactory tasks in Group 0. It is worth noting that as the threshold for dividing the groups decreases, the performance gain becomes gradually smaller.
16

---

Single-cat TGSR
0.10
Single-task network
TGSR
0.08
PSNR distanedddB)
1.PIPS distunce
0.06
0.0
0.02
0.5-
0.00
2
2
5
Degudation tack
Degradation
(a) Performance comparison PSNR.
(b) Performance comparison LPIPS.
Figure 10: Performance comparison of single-tast task network and TGSR in (a) PSNR and (b) LPIPS.
Table 7: Quantitative results of different methods on DIV2KI0G. Group0 denotes the validation set with satisfactory degradation tasks, and Group1-9 represents the validation with unsatisfactory degradation tasks.
Metrics
ESRGAN
RDSR
BSRGAN
RealSwinIR
DASR
MM RealSR
RealESRGAN
RealESRGAN TG (ours)
GroupO
PSNR I.PIPS (L)
22.97
23.28
0.6268
0.5255
0.4616
0.4503
0.4743
0.4403
0.4358
0.4344
Group1
PSNR
20.01
21,13
20.07
19.62
19.85
19.63
20,01
21.07
LPIPS (U)
0.6708
0.6267
0.6051
0.5834
0.5923
0.5552
0.5563
0.5188
Group2
PSNR
21.08
22.38
21.82
20.97
21.53
21,06
21.36
22.47
LPIPS (L)
0.6687
0.5821
0.5177
0.5092
0.5200
0.4910
0,4922
0.4735
Group3
PSNR (T)
20.94
23.58
23.06
22.40
22,82
22.30
22,72
23.58
LPIPS
0.4564
PSNR (1)
22.10
24.80
24.27
23.56
24.31
23.46
23.82
24.40
Group4
0.4496
0.4257
GroupS
PSNR (1)
22.84
25.58
24.89
24.27
24.91
23.85
24.56
25.07
LPIPS
0.4843
0,4208
0.4057
0.4187
0,4009
0.3974
0.3768
Group6
PSNR (1)
22.50
25.11
24.39
23.72
24.42
23 64
24.10
24.63
LPIPS
0.5895
0.4979
0.4400
0,4254
0.4410
0.4161
0.4130
0,3993
PSNR (t)
22.41
25.54
24.72
24.03
24:44
23.
24.50
24.90
Group?
LPIPS
0.5884
0.4892
0.4248
0.4146
0.4341
0.4048
0.4016
0.3897
Group8
PSNR (t)
20.94
24.19
23.57
23.77
23
LPIPS (1)
0.6223
0.5057
0.4492
0.4354
0.4568
0.4226
0.4211
0.4112
Group9
LPIPS (1)
0.6340
0.5140
0.4624
0.4485
0.4820
0.4333
0.4302
0.4209
7.3 More Visual Results
Fig. presents additional sample images from our validation datasets, including unsatisfactory degradation tasks from Groups 1-4 and some satisfactory degradation tasks from Group O. These images accompanied by the corresponding ground-trune -oound-truth images. Fig. 12 presents an additional qualitative comparison of competing methods on DIV2K5G. Fig. [3 an additional qualitative comparison of competing methods on unsatisfactory degradation tasks, our TGSR exhibits enhanced generalization ability when applied to real-world images.
images. Through the grouping and handling of
7.3.1 More Sample Images in DIV2K5G
17

---

STATE BANK OF
STATE BANK OF
STATE BANK OF
STATE BANK OF
STATE BANK OF
STATE BANK OF
MARKET BRA
MARKET BRA
MARKET BRA
MARKET BRA
MARKET BRA
MARKET BRA
HR
Group
Group
Group 2
Group 3
Group 4
Figure 11: Sample images from different degradation task groups in our DIV2K5G datasets.
18

---

7.3.2 More Qualitative Comparisons on DIV2K5G
ESRGAN
ROSK
MM-RcalSR
BSRGAN
Input
RealSwinIR
DASR
ReaEESGAAN
RealESRGAN-
ESRGAN
RDSR
MM-RcaiSR
BSRGAN
Input
RealSwinIR
ESRGAN
RDSR
MM-RealSR
BSRGAN
RealSwinIR
DASR
ReaIESRGAN
RealESRGAN-TG
ESRGAN
MM-RealSR
Input
RealSwinIR
DASR
RealESRGAN
ResIESRGAN-TG
ESRGAN
Input
RealSwinIR
DASR
RealESRGAN
RealESRGAN-TC
Figure 12: Qualitative comparison of competing methods on DIV2K5G.
19

---

7.3.3 More Qualitative Comparisons on Real-world Images
ESRGAN
ROSK
MM-RcaISR
BSRGAN
Imput
SwinIR
RealESRGAN
RealESRGAN-TO
PHJ
PDJ
A Pi
ESRGAN
RDSR
MM-RcaISR
BSRGAN
*
8
P
B P
Input
RealESRGAN-TG
ESRGAN
ROSR
MM-RealSR
BSRGAN
Input
SwinIR
DASR
ReIFSSGGAN
ReaIESRGANNTG
ESRGAN
MM-RealSR
BSRGAN
Input
SwinIR
DASR
RealESRGAN
RealESRGAN-TG
ESRGAN
Input
SwinIR
DASR
RealESRGAN
RealESRGAN-TG
Figure 13: Qualitative comparison of competing methods real-world images.
20


Reviews:
[{"review_id": "lvdRyjlqS0", "reviewer": "Reviewer_LkK8", "summary": "This paper casts real-world image SR as a multi-task learning task for complex image degradations. By comparing the performance gap between the real-SR model and the fine-tuned model for a specific degradation, the satisfactory task and unsatisfactory task are divided. And thus it proposes a task grouping method to address real-world SR.", "questions": "I have listed and explained my concerns on this paper in \"Weakness\". I expect the responses to those issues.", "limitations": "Authors just claim one limitation of the sampling from the degradation space in \"Conclusion\". This is actually an important issue for this work. Thus the more demonstration on real data, not from realESRGAN would be more convincing.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ The paper is well-written and easy to understand.\n+ It is interesting to analyze the real-world image SR from a multi-task learning view.\n+ The paper has considered many degradations for evaluation.", "weaknesses": "- The main concern is about the motivation. As claimed in Line 36-37, real SR faces a challenge of task competition or task conflict. But, \n1) it is still blind to explicitly represent or accurately model specific real degradations for the task of real SR. What is the meaning of \"task\" for real SR? What is the evidence of the \"task conflict\"? \n2) Although authors use the performance gap to define satisfactory task and unsatisfactory task, it seems to have a prerequisite that the results of pre-trained real-SR netwrok and single-task networks on each degradation task are reliable, especially based on their PSNR distance (Line 119).  This is rather unconvincing. It is hard for me to believe this demonstration for the motivation in Sec.3 and Fig.1. \n3) Line 126: \"Our analysis in Sec. 3 shows that when a real-SR network is tasked with many degradation tasks, they may compete for model capacity or interfere with each other, resulting in a significant decline in performance for certain tasks.\"  I could not understand what are the evidences for the claimed completition. Besides, the performance decline may be not cetainly resulted from the claimed completition and be influenced by other factors. \n\nOverall, the analyses on the motivation are rather unconvincing and it is hard for me to believe the rationality of the work.\n\n- Despite the analyses in Sec.3, it does not explain what cases the satisfactory task or unsatisfactory task actually indicate (it is too simple based on a PSNR threshold) and why they have these differences. I think this explain is important to well understand the real SR task and the motivation of taking it as a multi-task learning. Besides, how to undestand the derived degradation task groups in Fig.4?\n\n- Method (Sec.4). It could not ensure the reliability of using pre-trained real-sr network and single-task networks.\n\n- Evaluation:\n1. Line 204: \"we evaluate the model on Set14....\" But, Set14 has very limited number of images. Is this concincing?\n2. All the evaluations are based on RealESRGAN and there is no any evaluation on real data for real degration, not generated degradation by gan-based model. This is a very important demonstration extension for the work.\n\n- ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nDear Reviewer LkK8,\n\nThank you for your feedback. We’d like to provide some contextual information to help you better understand our method and address your concerns.\n\n## Task Grouping in Multi-task Learning\n\n(1) The task grouping approach (e.g., the CVPR2018 best paper: Taskonomy [2]) widely used in multi-task learning [1,2,3] focuses on finding the tasks that should be trained together based on the performance differences (e.g., PSNR distance) between the multi-task network and single-task networks.\n\n(2) As mentioned in [1], the term “task similarity” can easily be misunderstood to imply a strong attribute relationship between tasks. In fact, the task similarity in our paper represents the affinity relationship between the single-task network and the multi-task (real-SR) network, rather than the similarity between degradation cases. We will further clarify this in the final version.\n\n## Concerns \n\n### Q1. It is still blind to explicitly represent or accurately model specific real degradations for the task of real SR. What is the meaning of \"task\" for real SR? What is the evidence of the \"task conflict\"?\n\nWe've provided the definition of \"task\" in **Lines 93-110**. It is important to note that our task grouping method is not based on the similarity of degradation types but performance. The concept of \"task conflict/competition\" is explained in **Lines 111-124**.\n\n### Q2. Despite the analyses in Sec.3, it does not explain what cases the satisfactory task or unsatisfactory task actually indicate (it is too simple based on a PSNR threshold) and why they have these differences. Besides, how to understand the derived degradation task groups in Fig.4?\n\n(1) Basically, the unsatisfactory tasks are the degradation cases not well solved by the jointly-trained real-SR network. Please refer to **Lines 120-124** for the explanation of satisfactory and unsatisfactory tasks. \n\n(2) From Table 1, we can see that the performance of unsatisfactory tasks include both simple degradation cases (25dB in Group 4) and difficult degradation cases (21dB in Group 1), while the performance of satisfactory tasks (Group 0) is around 24dB. This observation indicates that the real-SR network tends to learn the degradations of medium difficulty. \n\n\n### Q3. Although authors use the performance gap to define satisfactory task and unsatisfactory task, it seems to have a prerequisite that the results of pre-trained real-SR netwrok and single-task networks on each degradation task are reliable, especially based on their PSNR distance (Line 119). This is rather unconvincing. It is hard for me to believe this demonstration for the motivation in Sec.3 and Fig.1. \n\nAs mentioned in **Line 210**, the pre-trained real-SR model we used is based on RealESRGAN, a highly influential and well-recognized work in the SR field with **over 22k stars on GitHub**. Using a single-task network as an empirical performance upper bound is a common practice, as demonstrated by works such as AdaFM[6], CResMD[7], and the accuracy-generalization trade-off[4]. \n\n### Q4. Method (Sec.4). It could not ensure the reliability of using pre-trained real-sr network and single-task networks.\n\nPlease see our response to Q3.\n\n### Q5. I could not understand what are the evidences for the claimed completition. Besides, the performance decline may be not cetainly resulted from the claimed completition and be influenced by other factors.\n\nAs described in **Lines 114-124**, the only variable we changed is the degradation task number. Therefore, the performance drop is only affected by the task number. This observation is consistent with multi-task learning, where this phenomenon is referred to as task competition, as also stated in **Lines 111-114**.\n\n### Q6. Evaluation: Line 204: \"we evaluate the model on Set14....\" But, Set14 has very limited number of images. Is this concincing? All the evaluations are based on RealESRGAN and there is no any evaluation on real data for real degration, not generated degradation by gan-based model. This is a very important demonstration extension for the work.\n\nAs stated in **Line 202**, we evaluate the model on Set14 for **each unsatisfactory degradation task**. Our main experiment employs 4000 Set14 test sets, each corresponding to a distinct degradation task.\n\nDue to space limitations, we have included the results on real data in the supplementary materials, as detailed in **Section 7.4.3**. \n\nThank you for the suggestion. Following the approach used by SwinIR and BSRGAN, we generate a new dataset DIV2K\\_random by randomly adding degradations sampled from the degradation model of RealESRGAN to the DIV2K\\_val dataset. Additionally, we conduct a comparison on the real benchmark AIM2019-val, the test set used for the real SR track in the AIM 2019 Challenge. On both datasets, our TGSR outperforms state-of-the-art methods consistently and significantly. We will include the results in final version.\n\n\n |          |        | ESRGAN|BSRGAN|RealESRGAN|SwinIR|DASR|TGSR|\n |  ----  | ----  |  ----  | ----  |  ----  | ----  |  ----  | ----  |\n | DIV2K\\_random| PSNR |20.63 |23.76|23.54|23.13|23.52|**23.84**|\n | DIV2K\\_random| LPIPS |0.6345 | 0.4622|0.4423|0.4432|0.4832|**0.4368**|\n | AIM2019-val | PSNR | 23.16 | 24.20 | 23.89 | 23.89 | 23.76 | **24.27**  | \n | AIM2019-val | LPIPS | 0.5500 | 0.4000 | 0.3960 | **0.3870** | 0.4210 | **0.3899** | \n\n[1] Efficiently Identifying Task Groupings for Multi-Task Learning\n\n[2] Taskonomy: Disentangling task transfer learning\n\n[3] Which Tasks Should Be Learned Together in Multi-task Learning?\n\n[4] Crafting Training Degradation Distribution for the Accuracy-Generalization Trade-off in Real-World Super-Resolution\n\n[5] Unsupervised degradation representation learning for blind super-resolution\n\n[6] Modulating image restoration with continual levels via adaptive feature modification layers\n\n[7] Interactive multi-dimension modulation with dynamic controllable residual learning for image restoration\n"}, {"review_id": "FtSA4thxvf", "reviewer": "Reviewer_tGSk", "summary": "This paper models a real-world image super-resolution (real-SR) from a multi-task learning perspective, that is treat real-SR as solving multiple distinct degradation tasks. To this end, the authors propose a task-grouping approach by grouping similar tasks together. Extensive experiments demonstrate the effectiveness of the proposed method.", "questions": "Please refer to the Weaknesses.\n\n", "limitations": "The authors do not address the limitations in the paper. It would be better to include the above suggestions for improvement.\n\n", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The authors regard learning a real-SR model as a multi-task learning task and highlight the task competition problem. The authors develop a task grouping-based real-SR network (TGSR).", "weaknesses": "1. The novelty of this paper should be highlighted. This paper refers to the tasks that are solved by the real-SR network well or not as unsatisfactory and satisfactory tasks. Such a way is similar to [1] which classifies the images as simple, medium and hard. It would be better to highlight the novelty of the method.     \n    [1] ClassSR: A General Framework to Accelerate Super-Resolution Networks by Data Characteristic\n\n2. The multi-task Real-SR method seems to be not smart. There are infinite number of degradations. The method randomly samples 100 degradation tasks and obtains 100 fine-tuned models.\n\n3. The task-grouping approach groups similar tasks together. The similarity is based on the performance. It means that the low noisy degradation and weak blurry degradation are similar. However, low noise and strong noise are the same degradations, which should be treated as the same task.\n\n4. In Figure 1 (a), this paper uses the degradation models which are based on Real-ESRGAN. Each degradation on an image is a composite function which different degradations. Could you investigate the effect of such a case?\n\n5. The authors identify the unsatisfactory tasks by comparing the performance between the jointly-trained multi-task real-SR network and the fine-tuned single-task networks. What if directly compare the PSRN (like [1]) with a threshold?\n\n6. Some experiment details are not clear. Is the real-SR model fine-tuned on a pre-trained Real-ESRGAN or trained from scratch?\n\n7. The performance is not state-of-the-art. In Table 1, the proposed method is worse than RDSR. In Table 2, RealESRGAN-G has no advantage over RealESRGAN for Group0 and Group1. How to demonstrate the superiority of the proposed method. In Figure 5, the input images have much noise, not low-resolution. It would be better to provide SR results. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nDear Reviewer tGSk,\n\nThank you for your feedback. We'd like to address some factual inaccuracies in your comments and clarify the misunderstandings.\n\n## Factual Errors & Misunderstandings \n\n### Q1. The multi-task Real-SR method seems to be not smart. There are infinite number of degradations. The method randomly samples 100 degradation tasks and obtains 100 fine-tuned models.\n\nPlease note that **the 100 degradation tasks are only used for illustration in Figure 1 (Line 115)**. We sampled **4,000** tasks for the main experiments (**Line 198**) and **10,000** tasks for experiments in the supplementary materials (**Line 460**). To efficiently identify the unsatisfactory tasks from such a large set of tasks, we do not train a single-task network for each task, but propose a gradient-based performance indicator as described in **Lines 138-148**. As also noticed by Reviewer yhLt, it is obviously much faster than directly fine-tuning a single-task network.\n\n### Q2. The performance is not state-of-the-art. (i) In Table 1, the proposed method is worse than RDSR. (ii)In Table 2, RealESRGAN-G has no advantage over RealESRGAN for Group0 and Group1. How to demonstrate the superiority of the proposed method. (iii)In Figure 5, the input images have much noise, not low-resolution. It would be better to provide SR results.\n\n(i) Please note that RDSR is trained with the MSE loss and hence has higher PSNR than GAN-based SR methods, which is normal.\n\n(ii) RealESRGAN-G is not our method but used to analyze the performance of RealESRGAN in different groups (**Lines 272-276**). It can be seen clearly that our TGSR outperforms RealESRGAN. \n\n(iii) Figure 5 shows real-SR results. Please note that real-SR encompasses multiple degradations, including noise. In the supplementary file (**Section 7.4.1 and 7.4.2**), we've provided additional real-SR results, which include degradations with slight noise. \n\n### Q3. The novelty of this paper should be highlighted. This paper is similar to ClassSR. What if directly compare the PSRN (like ClassSR) with a threshold?\n\nWe'd like to highligt the novelty of our method by clarifying two basic concepts.\n#### 1. **Efficient-SR vs. Real-SR** \n(1) We are very familiar with ClassSR, which solves the efficient SR problem under the classical SR setting and only has one degradation task, i.e., bicubic down-sampling. Hence, one GT image corresponds to one LR image. ClassSR divides the LR images into simple, middle, and hard texture groups by absolute PSNR value.\n\n(2) Real-SR (e.g., BSRGAN and Real-ESRGAN) introduces a complex degradation model to synthesize LR images, which include a series of degradation types, such as blur, noise, resize, and JPEG. Hence, one GT image corresponds to multiple LR images. It is clear that hard textures may contain slight and strong degradations. Therefore, there is no linear correlation between absolute PSNR values and unsatisfactory degradation tasks.\n\n#### 2. **Absolute PSNR vs. Relative PSNR**\n\n(1) The objective of Real-SR is to achieve improved performance across all degradation tasks. In Section 3, we observe a range of absolute PSNR values, from 19dB to 25dB, on the validation set. It is worth noting that unsatisfactory degradation tasks can encompass both slight (e.g., 24dB) and strong (e.g., 21dB) degradations. Therefore,**relying solely on absolute PSNR values is inadequate for identifying unsatisfactory tasks**.\n\n(2) To address this issue, we utilize the performance difference, specifically the PSNR distance between single-task networks and the multi-task Real-SR network, as a means to identify unsatisfactory tasks. This approach is rooted in the common understanding within the field of multi-task learning [1,2,3] and is exemplified in notable works such as the CVPR 2018 best paper Taskonomy [2]. By considering the performance difference, we can more effectively identify tasks that fall short of expectations and require further attention.\n\n### Q4. Low noise and strong noise are the same degradations, which should be treated as the same task.\n\n(1) The task grouping approach (e.g., the CVPR-2018 best paper: Taskonomy [2]) widely used in multi-task learning [1,2,3] focuses on finding the tasks that should be trained together based on the performance differences (e.g., PSNR distance) between the multi-task network and single-task networks. \n\n(2) As mentioned in [1], the term “task similarity” can easily be misunderstood to imply a strong attribute relationship between tasks. In fact, the task similarity in our paper represents the affinity relationship between the single-task network and the multi-task (real-SR) network, rather than the similarity between degradation types. We will further clarify this in the final version.\n\n(3) The consensus in the field recognizes that improved performance, such as a higher PSNR, contributes to superior visual outcomes. Therefore, if performance enhancements can be attained, grouping weak blur and noise together is deemed acceptable. Our experimental findings further validate the achievement of state-of-the-art results.\n\n### Q5. Each degradation of RealESRGAN is a composite function which different degradations. Could you investigate the effect of such a case?\n\nIn our response to Q4, we've clarified that our task grouping method is not based on degradation types but performance. In addition, it is difficult to analyze the composite function in a large degradation space, as pointed out in [4].\n\n## Other Comments\n\n### Q6. Is the real-SR model fine-tuned on a pre-trained Real-ESRGAN?\n\nIn our experiment, the real-SR model is fine-tuned on a pre-trained Real-ESRGAN. We will make this clear in the final version.\n\n[1] Efficiently Identifying Task Groupings for Multi-Task Learning\n\n[2] Taskonomy: Disentangling task transfer learning\n\n[3] Which Tasks Should Be Learned Together in Multi-task Learning?\n\n[4] Crafting Training Degradation Distribution for the Accuracy-Generalization Trade-off in Real-World Super-Resolution"}, {"review_id": "wIEKovY50f", "reviewer": "Reviewer_4Sx7", "summary": "This paper aims at the task conflict issue of real-world image super-resolution (SR) with multiple degradation tasks, and proposes a task grouping approach to group similar tasks together to mitigate task competition. In addition, this paper designs a real-SR network called TGSR (task grouping-based real-SR network), which leverages the identified task groups to train a task group classifier and use the predicted information to generate modulation signals for image restoration.", "questions": "1. AT lines 166-167, \"we make a trade-off to fine-tune the pre-trained real-SR network on all unsatisfactory tasks simultaneously through joint-training\". What is the joint learning in this paper? \n\n\n2. It is unfair to conduct the evaluation experiments on the grouped DIV2k (namely, DIV2K5G) with 5 different degradation groups. The authors should also conduct a fair comparison with Sota methods on the well-used benchmarks, such as SwinIR [A1], DASR [A2]. \n\n[A1] Liang J, Cao J, Sun G, et al. Swinir: Image restoration using swin transformer[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2021: 1833-1844.\n[A2] Wang L, Wang Y, Dong X, et al. Unsupervised degradation representation learning for blind super-resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 10581-10590.", "limitations": "none", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. a new look at real SR from a multi-task learning perspective.\n2. a performance indicator based on gradient updates to efficiently identify the degradation tasks.", "weaknesses": "1. The authors compare the PSNR distance between the real-SR model trained on entire degradation space introduced by Real-ESRGAN [34] and 100 types of single-task models fine-tuned on the specific-degradation, and highlight some degradation tasks not well solved by the real-SR model as unsatisfactory tasks. However, they only number 1-100 for different tasks, and do not present the detailed settings (blur type, kernel width, noise type, noise level, and jpeg compression) of each degradation task. Such inadequate expression makes readers including me confued as to whether there is a relationship between PSNR distance and degradation type. \n\n2. The authors claim that \"... divide them (unsatisfactory degradation tasks) into multiple task groups based on their similarity to reduce negative transfer.\" However, the task similarity computed by the PSNR distance is not convincing, and it is worth to further studying the relationship between the task grouping strategy and the degradation distribution. \n\n3. At lines 177-178, \"in this study, we use the found degradation task groups to train a task group classifier, which is then integrated into a standard real-SR framework\". I want to know how to select the number of task groups, and how do different numbers of groupings affect the final performance. If the number of groupings is set to infinity, is the task group classifier equivalent to the well-studied degradation estimator?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nDear Reviewer 4Sx7,\n\nWe appreciate your feedback and would like to address the concerns raised in your comments, which include some factual errors and misunderstandings. \n\n## Factual Errors & Misunderstandings \n\n### Q1. The authors compare the PSNR distance... However, they only number 1-100 for different tasks, and do not present the detailed settings (blur type, kernel width, noise type, noise level, and jpeg compression) of each degradation task. Such inadequate expression makes readers including me confued as to whether there is a relationship between PSNR distance and degradation type.\n\n(1) Please note that **the 100 degradation tasks are only used for illustration in Figure 1**, as stated in **Line 115**.\n\n(2) Please note that we sampled **4,000** tasks for the main experiments (**Line 198**) and **10,000** tasks for the experiments in the supplementary materials (**Line 460**), which can sufficiently represent the degradation space. To efficiently identify the unsatisfactory tasks from such a large number of tasks, we do not train a single-task network for each task, but propose a gradient-based performance indicator as described in **Lines 138-148**.\n\n(3) Please also note that due to the complex combinations of degradations in a large degradation space (e.g., RealESRGAN degradation model), it is very difficult to group degradation tasks by their types, as pointed out in [1].\n\n### Q2. The authors claim that \"... divide them (unsatisfactory degradation tasks) into multiple task groups based on their similarity to reduce negative transfer.\" However, the task similarity computed by the PSNR distance is not convincing, and it is worth to further studying the relationship between the task grouping strategy and the degradation distribution.\n\n(1) The task grouping approach (e.g., the CVPR-2018 best paper: Taskonomy [2]) widely used in multi-task learning [2,3,4] focuses on finding the tasks that should be trained together based on the performance differences (e.g., PSNR distance) between the multi-task network and single-task networks. \n\n(2) As mentioned in [4], the term “task similarity” can easily be misunderstood to imply a strong attribute relationship between tasks. In fact, the task similarity in our paper represents the affinity relationship between the single-task network and the multi-task (real-SR) network, rather than the similarity between degradation types. We will further clarify this in the final version.\n\n(3) The consensus in the field recognizes that improved performance, such as a higher PSNR, contributes to superior visual outcomes. Our experimental findings further validate the achievement of state-of-the-art results.\n\n## Other Comments\n\n### Q3. At lines 177-178, \"in this study, we use the found degradation task groups to train a task group classifier, which is then integrated into a standard real-SR framework\". I want to know how to select the number of task groups, and how do different numbers of groupings affect the final performance. If the number of groupings is set to infinity, is the task group classifier equivalent to the well-studied degradation estimator?\n\nPlease note that in **Section 7.3.4 (Line 490)** of the supplementary materials, we've provided additional results and a discussion about the effect of the number of task groups. The number of task groups is determined by the threshold (**Line 206**) of performance improvement. We find that as more and more groups are added, the rate of performance improvement for each new group becomes smaller. Our approach differs from the degradation estimation method, which estimates every degradation task, because we have a limited group number of unsatisfactory tasks.\n\n### Q4. AT lines 166-167, \"we make a trade-off to fine-tune the pre-trained real-SR network on all unsatisfactory tasks simultaneously through joint-training\". What is the joint learning in this paper?\n\nJoint-training refers to training a single real-SR network with multiple (e.g., thousands of) degradation tasks simultaneously, which is the strategy used by most existing real-SR methods such as RealESRGAN and BSRGAN, as stated in **Lines 139-140**.\n\n### Q5. It is unfair to conduct the evaluation experiments on the grouped DIV2k (namely, DIV2K5G) with 5 different degradation groups. The authors should also conduct a fair comparison with Sota methods on the well-used benchmarks, such as SwinIR [A1], DASR [A2].\n\nThank you for the suggestion. Following the approach used by SwinIR and BSRGAN, we generate a new dataset DIV2K\\_random by randomly adding degradations sampled from the degradation model of RealESRGAN to the DIV2K\\_val dataset. Additionally, we conduct a comparison on the real benchmark AIM2019-val, the test set used for the real SR track in the AIM 2019 Challenge [5]. On both datasets, our TGSR outperforms state-of-the-art methods consistently and significantly. We will include the results in the final version.\n\n|          |        | ESRGAN|BSRGAN|RealESRGAN|SwinIR|DASR|TGSR|\n|  ----  | ----  |  ----  | ----  |  ----  | ----  |  ----  | ----  |\n| DIV2K\\_random| PSNR |20.63 |23.76|23.54|23.13|23.52|**23.84**|\n| DIV2K\\_random| LPIPS |0.6345 | 0.4622|0.4423|0.4432|0.4832|**0.4368**|\n| AIM2019-val | PSNR | 23.16 | 24.20 | 23.89 | 23.89 | 23.76 | **24.27**  | \n| AIM2019-val | LPIPS | 0.5500 | 0.4000 | 0.3960 | **0.3870** | 0.4210 | **0.3899** | \n\n[1] Crafting Training Degradation Distribution for the Accuracy-Generalization Trade-off in Real-World Super-Resolution. ICML2023.\n\n[2] Taskonomy: Disentangling task transfer learning\n\n[3] Conflict-averse gradient descent for multi-task learning\n\n[4] Efficiently Identifying Task Groupings for Multi-Task Learning\n\n[5] AIM 2019 challenge on real-world image super-resolution: Methods and results"}, {"review_id": "vwUaLqajbU", "reviewer": "Reviewer_yhLt", "summary": "The authors rethink the real-world super-resolution problem from the perspective of multi-task learning. And point out the primary challenge: task competition problem. To address this issue, they propose a task grouping method to identify unsatisfactory tasks and introduce TGSR to handle them separately, thereby eliminating task competition. Extensive experiments demonstrate the effectiveness of the proposed method.", "questions": "See above weakness, authors need to address them well.", "limitations": "Yes. Besides, since this work still uses synthetic data. It would be better to consider low quality images from real scenes.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1.\tTaking real-SR as a multi-task learning problem offers a novel perspective that can yield further insights and considerations. For instance, how to effectively sample degradation in a large degradation space? What relationships exist between different tasks?\n\n2.\tThe authors present a clear motivation in the form of task competition, as current real-SR networks are unable to perfectly handle all cases of degradation.\n\n3.\tThe paper includes extensive ablation studies. Interestingly, the authors found that even when fine-tuning directly on a single degradation task, many cases still do not perform as well as a range of degradation. This suggests that traditional blind super-resolution networks based on kernel estimation may no longer be applicable within large degradation spaces. This is a highly insightful finding.\n", "weaknesses": "1.\tThe authors should provide the calculation time of the performance indicator, although the performance indicator is obviously faster than directly fine-tuning a single-task network. \n\n2.\tThe groups 1-4 in Figure 4 of the main text seem to be inconsistent with the samples in the supp file, and Table 1 also shows that group 1 should be a low-quality difficult case (with the lowest PSNR value).\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nDear Reviewer yhLt,\n\nWe'd like to thank you for your positive feedback and address the concerns raised in your comments.\n\n### Q1. The authors should provide the calculation time of the performance indicator, although the performance indicator is obviously faster than directly fine-tuning a single-task network.\n\nThe computational time of the performance indicator for one degradation task is around 100 seconds. In contrast, it takes about 2 hours to fine-tune a single-task network. Hence, our approach offers a nearly 70-fold speed increase when compared to the direct fine-tuning of single-task networks for identifying unsatisfactory tasks. We will include the comparison and discussion of this matter in the final version.\n\n### Q2. The groups 1-4 in Figure 4 of the main text seem to be inconsistent with the samples in the supp file, and Table 1 also shows that group 1 should be a low-quality difficult case (with the lowest PSNR value).\n\nThank you for your careful reading. The order of groups 1-4 in Figure 4 has been incorrectly reversed, but the order in the supplement is correct. We will correct this error in the final version.\n"}, {"review_id": "rSt0VvRGKt", "reviewer": "Reviewer_o9KK", "summary": "This paper revisits the real-world super-resolution task from the perspective of multi-task learning, considering each type of degradation as a separate task. However, there are countless types of degradation in the real world, which often results in severe task competition in previous methods. The authors propose a task grouping strategy to address this problem.\nFurthermore, they present a multi-task learning framework suitable for real SR tasks, guiding the target SR model by the task group label probability. Experimental results showcase an advanced performance improvement by the proposed TGSR across a wide range of degradation types.\n", "questions": "My concerns regarding the evaluation data DIV2K5G and its close connection with the proposed TGSR model are valid. The potential for an inherent bias exists since the dataset was constructed using the proposed task grouping algorithm, which is closely tied with the iteratively trained and fine-tuned SR model.\n\nIt would indeed be beneficial for the authors to evaluate the TGSR model on the standard benchmark dataset, which would provide a more independent measure of the effectiveness of the multi-task learning-based task grouping strategy.\n\nAdditionally, performing comparisons on the RealSR dataset or a randomly generated benchmark by BSRGAN or RealESRGAN can be beneficial. These additional evaluations can provide further evidence to the superiority of the proposed TGSR.\n", "limitations": "None.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper addresses real-world super-resolution tasks from the multi-task learning perspective, which I think is novel to this field. The paper effectively dissects the issues present in previous methods from a multi-task learning perspective and provide in-depth insight.\n\nI think that the degradation task grouping strategy proposed in the paper may sheds new light on the analysis and study of the blind SR tasks.", "weaknesses": "Detailed selection strategy for different degradation tasks in Section 3.1 and Figure 1 are recommended.\n\nHyper-parameters in the proposed task grouping strategy, such as the number of task group and division threshold, are given and fixed in this paper, which is not motivated to me. I think these could lead to potential impact for the proposed method, but detailed analysis or ablation is missing is this paper. The authors should provide more comprehensive analyses to address these concerns.\n\nIn the degradation type grouping strategy, the threshold for each group division is determined manually, and the paper does not mention ablation in the number of groups or threshold, which could lead to potential imprecision in the division strategy.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nDear Reviewer o9KK,\n\nWe appreciate your thoughtful comments. Please find below our detailed response addressing your concerns.\n\n### Q1. Detailed selection strategy for different degradation tasks in Section 3.1 and Figure 1 are recommended.\n\nPlease note that we've stated in **Line 115** that the 100 degradation tasks (used for the illustration in Figure 1) were randomly sampled from the degradation model of RealESRGAN. In our main experiment, we also randomly generated 4000 degradation tasks using the RealESRGAN degradation model. \n\n### Q2. Hyper-parameters in the proposed task grouping strategy, such as the number of task group and division threshold, are given and fixed in this paper, which is not motivated to me. I think these could lead to potential impact for the proposed method, but detailed analysis or ablation is missing is this paper. The authors should provide more comprehensive analyses to address these concerns.\n\nPlease note that in **Section 7.3.4** of the supplementary materials, we've provided additional results and a discussion about the effect of the number of task groups. We find that as more and more groups are added, the rate of performance improvement for each new group becomes smaller.\n\n### Q3. In the degradation type grouping strategy, the threshold for each group division is determined manually, and the paper does not mention ablation in the number of groups or threshold, which could lead to potential imprecision in the division strategy.\n\nPlease note that in **Section 7.3.5** of the supplementary materials, we've provided an ablation study of the division threshold. We find that as the division threshold decreases, the performance gain becomes smaller.\n \n### Q4. Performing comparisons on the RealSR dataset or a randomly generated benchmark by BSRGAN or RealESRGAN can be beneficial. These additional evaluations can provide further evidence to the superiority of the proposed TGSR.\n\nThank you for the suggestion. Following the approach used by BSRGAN, we generate a new dataset DIV2K\\_random by randomly adding degradations sampled from the degradation model of RealESRGAN to the DIV2K\\_val dataset. Additionally, we conduct a comparison on the real benchmark AIM2019-val, the test set used for the real SR track in the AIM 2019 Challenge [1]. On both datasets, our TGSR outperforms state-of-the-art methods consistently and significantly. We will include the results in the final version.\n\n\n|          |        | ESRGAN|BSRGAN|RealESRGAN|SwinIR|DASR|TGSR|\n|  ----  | ----  |  ----  | ----  |  ----  | ----  |  ----  | ----  |\n| DIV2K\\_random| PSNR |20.63 |23.76|23.54|23.13|23.52|**23.84**|\n| DIV2K\\_random| LPIPS |0.6345 | 0.4622|0.4423|0.4432|0.4832|**0.4368**|\n| AIM2019-val | PSNR | 23.16 | 24.20 | 23.89 | 23.89 | 23.76 | **24.27**  | \n| AIM2019-val | LPIPS | 0.5500 | 0.4000 | 0.3960 | **0.3870** | 0.4210 | **0.3899** | \n\n[1] AIM 2019 challenge on real-world image super-resolution: Methods and results.\n"}]

Author rebuttal:

Meta-review: The paper received mixed reviews from five reviewers and the authors provided responses to the expressed concerns.

Reviewer LkK8 is Borderline Reject and keeps his rating after rebuttal considering that his concerns are not well addressed.

Reviewer tGSk is Reject and keeps his rating after rebuttal.

Reviewer 4Sx7 upgrades to Borderline Accept after rebuttal.

Reviewer yhLt upgrades to Strong Accept after rebuttal.

Reviewer o9KK keeps his Borderline Accept after rebuttal.

After carefully reading the manuscript, the reviews, the authors' responses and the discussions, it is clear that the authors did a good job with their rebuttal and convinced two reviewers to improve their ratings. The negative ("Reject") is from Reviewer tGSk but, unfortunately, did not provide a final comment to the authors' response, while the other reviewers are borderline accept (4Sx7 and o9KK), borderline reject (LkL8) and strong accept (yhLT).

The ACs agree with the majority of the reviewers that the work has merits and is of interest for the community and considers that the authors should further refine their manuscript for camera ready by including (part of) contents/info provided in their responses and by benefiting from the reviewers' feedback.
Decision: Accept (poster)