Title: Real-World Image Super-Resolution as Multi-Task Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Image super-resolution
Abstract: In this paper, we take a new look at real-world image super-resolution (real-SR) from a multi-task learning perspective. We demonstrate that the conventional formulation of real-SR can be viewed as solving multiple distinct degradation tasks using a single shared model. This poses a challenge known as task competition or task conflict in multi-task learning, where certain tasks dominate the learning process, resulting in poor performance on other tasks. This problem is exacerbated in the case of real-SR, due to the involvement of numerous degradation tasks. To address the issue of task competition in real-SR, we propose a task grouping approach. Our approach efficiently identifies the degradation tasks where a real-SR model falls short and groups these unsatisfactory tasks into multiple task groups. We then utilize the task groups to fine-tune the real-SR model in a simple way, which effectively mitigates task competition and facilitates knowledge transfer. Extensive experiments demonstrate our method achieves significantly enhanced performance across a wide range of degradation scenarios.

Real-World Image Super-Resolution as Multi-Task Learning

Abstract

In this paper, we take a new look at real-world image super-resolution (real-SR) from a multi-task learning perspective. We demonstrate that the conventional formulation of real-SR can be viewed as solving multiple distinct degradation tasks using a single shared model. This poses a challenge known as task competition or task conflict in multi-task learning, where certain tasks dominate the learning process, resulting in poor performance on other tasks. This problem is exacerbated in the case of real-SR, due to the involvement of numerous degradation tasks. To address the issue of task competition in real-SR, we propose a task grouping approach. Our approach efficiently identifies the degradation tasks where a real-SR model falls short and groups these unsatisfactory tasks into multiple task groups. We then utilize the task groups to fine-tune the real-SR model in a simple way, which effectively mitigates task competition and facilitates knowledge transfer. Extensive experiments demonstrate our method achieves significantly enhanced performance across a wide range of degradation scenarios. The source code is available at https://github.com/XPixelGroup/TGSR .

1 Introduction

Real-world image super-resolution (real-SR) aims to enhance the resolution and quality of lowresolution images captured in real-world scenarios. Real-SR algorithms enable improved image quality and better visual understanding, making them valuable in a wide range of applications [43, 46, 45, 14, 17]. Unlike non-blind SR [9, 56, 55] and classical blind SR [13, 18, 38] that assumes a simple degradation process, real-SR deals with complex and unknown degradations present in real-world imaging conditions, such as noise, blur, compression artifacts, and sensor limitations. The diversity of real-world degradations and unknown degradation parameters make it challenging to reverse the specific degradation effects and accurately recover high-resolution details.

Prior studies tackle real-SR by designing various degradation models to simulate the degradation process in real-world scenarios. For instance, they generate synthetic paired training data with a shuffled degradation model (BSRGAN [50]), a high-order degradation model (RealESRGAN [39]), or a three-level degradation model (DASR [25]). Existing methods commonly train a single realSR network with training data generated by a sophisticated degradation model, aiming to cover as many degradation cases as possible during the training process. Under this framework, recent works have focused on enhancing various aspects of the real-SR network, including improving the backbone network architecture [26], optimizing inference efficiency [25], enhancing generalization capabilities [23], and improving modulation ability [31].

This study takes a new look at real-SR from a multi-task learning perspective. We show that the conventional formulation of real-SR essentially corresponds to a multi-task learning problem, which involves solving a large number of different degradation tasks simultaneously with a single shared model. Consequently, real-SR faces a well-known challenge in multi-task learning, namely task competition or task conflict [32]. It refers to the situation that tasks compete for model capacity, potentially resulting in certain tasks dominating the learning process and adversely affecting the performance of other tasks. This problem is severely aggravated in the case of real-SR, where a large number of degradation tasks (up to thousands) are typically involved. In our pilot experiment (Fig. 1), we have observed that a real-SR network trained with the multi-task objective fails to yield satisfactory results for a significant portion of degradation tasks.

We propose a task grouping approach (Fig. 2) to alleviate the negative impact of task competition for real-SR. Task grouping [48, 37, 35, 10] is an effective technique in multi-task learning that helps mitigate negative transfer by training similar tasks together. This approach typically involves learning the relationship or relevance between pairs of tasks through validation or fine-tuning, which becomes impractical in the context of real-SR where there is a large number of tasks to consider. Hence, we resort to design indirect measures to assess task affinity. Specifically, we introduce a performance indicator based on gradient updates to efficiently identify the degradation tasks that a real-SR model falls short of. Then, we propose an algorithm to group these unsatisfactory tasks into multiple groups based on a performance improvement score. Finally, we propose TGSR, a simple yet effective real-SR method, which leverages the identified task groups to fine-tune the pre-trained real-SR network. Our key contributions are summarized as follows:

We take a new look at real-SR from a multi-task learning perspective and highlight the task competition problem.

We propose a task grouping approach to effectively address the task competition problem for real-SR and develop a task grouping based real-SR method (TGSR).

We conduct extensive experiments to validate the effectiveness of TGSR, and the results demonstrate its superior performance compared to state-of-the-art real-SR methods.

2 Related Work

Image super-resolution. Since Dong et al. [9] first introduced Convolutional Neural Networks (CNNs) to the SR problem, a series of learning-based SR methods have made great progress, including deep networks [21], dense connections [56], channel attention of [55], residual-in-residual dense blocks [41], and transformer structure [26]. To reconstruct realistic textures, Generative Adversarial Networks (GANs) [24, 41, 53, 42] have been employed to generate visually pleasing results. However, these methods adopt a bicubic down-sampling degradation that is insufficient for real-world images.

Real-world super-resolution. To address the SR problem in real-world scenarios, classical blind SR methods primarily use Gaussian blur and noise to model the distribution of real-world images. Significant progress has been made through a variety of approaches, including the use of a single SR network with multiple degradations [51], kernel estimation [13, 18, 44, 2], and representation learning [38]. Additionally, BSRGAN [50] proposes a large degradation model that incorporates multiple degradations using a shuffled strategy, while RealESRGAN [39] employs a high-order strategy to construct a large degradation model. DASR [25] adopts a three-level degradation distribution (i.e., two one-order and one high-order degradation models) to simulate the distribution of real-world images. These works demonstrate the potential of large degradation models for real-world applications. In addition, recent works also improve real-SR in multiple dimensions, such as network backbone [26], efficiency [25], generalization [23, 54, 52] and modulation ability [31].

Multi-task learning . Multi-task learning methods can be roughly divided into three categories: task balancing, task grouping, and architecture design. Task balancing [16, 20, 47, 27, 34, 15, 8, 7, 19] methods address task/gradient conflicts by re-weighting the loss or manipulating the update gradient. Task grouping [48, 37, 35, 10] methods mainly focus on identifying which tasks should be learned together. Zamir et al. [48] provide a task taxonomy that captures the notion of task transferability. TAG [10] determines task groups by computing affinity scores that capture the effect between tasks. Architecture design methods can mainly be divided into hard parameter sharing methods [22, 28, 3] and soft parameter sharing methods [30, 33, 12, 11]. Hard parameter sharing methods require

0

10

20

30

40

50

60

70

80

90

Dgradation task

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

PSNR improvement(dB)

Unsatisfactory Tasks

Satisfactory Tasks

Image quality (lower

higher by PSNR)

Satisfactory threshold for degradation task

Real-SR network

Single-task network

Unsatisfactory task

Satisfactory task

different decoders for different tasks, while soft parameter sharing methods do cross-talk between different task networks. We focus on a task grouping strategy in our TGSR. The strategy identifies unsatisfactory degradation tasks in a large degradation space. Subsequently, we can further improve the overall performance of the real-SR network by fine-tuning it in the unsatisfactory tasks.

3 Real-SR as a Multi-task Learning Problem

Problem Formulation Real-SR aims to restore a high-resolution (HR) image x from its lowresolution (LR) counterpart y that has undergone an unknown and intricate degradation process:

where f i represents a degradation function such as Gaussian blurring with a kernel width [0.1, 2.4], adding Gaussian noise with a noise level [1, 25], a downsampling operation with a scale factor r , or applying the JPEG compression. Hence, D represents a vast continuous degradation space that encompasses an infinite number of degradations. Existing real-SR models such as BSRGAN [50], RealESRGAN [39], and DASR [25] make different assumptions in the generation of D , aiming to simulate the highly complex degradation process in real-world scenarios.

An SR task τ can be defined as a training pair ( x, y = d ( x )) formed by sampling a degradation d from the degradation space D and applying it on an HR image x to generate an LR image y . Due to the infinite size of D , it is impossible to consider every degradation case. Hence, a common approach is to sample a large number of degradations to sufficiently represent the degradation space. Given a set of high-resolution images X , with N different degradations sampled from D ( N ≫|X| ), we can generate a set of SR tasks: T = { τ i = ( x i , y i ) } N i =1 , where x i ∈ X and y i is obtained by applying a degradation on x i . A real-SR model is commonly trained by minimizing the empirical risk:

where L i is the loss on task τ i , and θ are the trainable model parameters shared by all N tasks. Therefore, real-SR is essentially a multi-task learning problem, aiming to solve N different SR tasks with a single shared model. In the following, we refer to τ i as a degradation task .

Task Competition From the view of multi-task learning, real-SR is a challenging problem as it intends to solve a large number of different degradation tasks (e.g., N = 10 3 ) altogether with a single model, inevitably suffering from task competition that some tasks may dominate the training process leading to poor performance on other tasks [37].

To illustrate the impact of task competition, we randomly sample 100 degradation tasks with the degradation model in [39] and train a real-SR network with the multi-task objective in Eq. 2. Next, we fine-tune the real-SR network on each degradation task independently and obtain 100 fine-tuned

Ⅰ: Jointly Train All Degradation

Tasks with a Single Network

Ⅲ: Group Unsatisfactory

Degradation Tasks

𝐺

!

𝐺

"

𝐺

#

𝐺

$

Ⅱ: Identify Unsatisfactory

Degradation Tasks

Ⅳ: Continue to Train the Real-

SR Network with Degradation

Task Groups

HR

Degradation space

Degradation tasks

Unsatisfactory degradation tasks

Satisfactory degradation tasks

LR in task 1

LR in task 2

A Pre-trained Real-SR

Network

A Task Grouping Based

Real-SR Network (TGSR)

𝐺

!

𝐺

"

𝐺

#

𝐺

$

𝐺

%

…

Compute the average

performance

indicator with Eq.4

Compute the

performance

improvement score

with Eq.5

Training

Fine-tuning

models, which we call single-task networks. We then compare the performance between the real-SR network and the single-task networks on each degradation task by computing their PSNR distance. The results in Fig. 1 show that for nearly half of the degradation tasks, the PSNR distance between the real-SR network and single-task network exceeds 0.4dB, indicating that these tasks are not well solved by the real-SR network, which we refer to as unsatisfactory tasks . From an optimization perspective, the other tasks (those with PSNR distance less than 0.4dB) dominate the learning process and are effectively solved by the real-SR network, which we refer to as satisfactory tasks .

4 Real-SR via Task Grouping

Our analysis in Sec. 3 shows that when a real-SR network is tasked with many degradation tasks, they may compete for model capacity or interfere with each other, resulting in a significant decline in performance for certain tasks. This phenomenon, commonly referred to as negative transfer [37], is a well-known challenge in the field of multi-task learning. An effective approach to mitigate negative transfer is task selection or task grouping [10, 37, 48]. By finding groups of tasks that may benefit from training together, the interference among tasks can be minimized.

4.1 Which Tasks Should Be Learned Together for Real-SR?

The results in Fig. 1 suggest that the satisfactory tasks should be grouped together. These dominant tasks may share certain characteristics or similarities that lead to minimal task conflicts, making them more prominent in the training process. More importantly, considering that the real-SR network has effectively tackled these tasks and reached a satisfactory performance level, we may focus on the unsatisfactory tasks that present greater potential for improvement.

Efficiently Identifying Unsatisfactory Tasks To identify the unsatisfactory tasks, we may use the approach described in Sec. 3 and Fig. 1, by comparing the performance between the jointly-trained multi-task real-SR network (referred to as the pre-trained real-SR network hereafter) and the finetuned single-task networks. However, the cost of adopting this approach is prohibitive due to the large number of degradation tasks (e.g., thousands or tens of thousands) involved in real-SR.

Therefore, to efficiently identify the unsatisfactory tasks, we propose a measure to assess whether a degradation task has been well solved by the real-SR network, which is done through fine-tuning the pre-trained real-SR network for a small number of iterations (e.g., 100) on the degradation task. Specifically, we assess the impact of the gradient update of task τ i on the shared parameters θ (i.e., the pre-trained real-SR network), by comparing the loss of τ i before and after updating θ . We develop a performance indicator defined as:

where θ t i are the parameters updated on task τ i at the time step t . Notice that a high value of z t i indicates the loss of task τ i is not significantly reduced after fine-tuning, suggesting that the pre-trained real-SR network has achieved good performance on this task.

Algorithm 1: Degradation Task Grouping for Real-SR

For stability, we consider the average performance indicator during the fine-tuning process on task τ i :

where t s and t e represent the starting and end points of a time span, respectively. With the computed average performance indicator of each task, we can set a threshold to select the unsatisfactory tasks. The effectiveness of ˆ z i is validated in Sec. 5.2 and Sec. 5.3.

Grouping Unsatisfactory Tasks into Multiple Task Groups The number of identified unsatisfactory degradation tasks can still be significant, typically in the range of hundreds in our experiments. Given the potential variance among these tasks, it would be advantageous to further divide them into multiple task groups based on their similarity to reduce negative transfer. A common approach for task grouping in multi-task learning is to learn pairwise performance indicator between tasks [48, 10, 37], which is impractical for real-SR due to the large number of degradation tasks involved.

A feasible way is to learn an indirect measure of task similarity by fine-tuning the pre-trained real-SR network on each degradation task independently and computing the PSNR distance as described in Sec. 3 and Fig. 1. However, this approach is still time-consuming given the large number of unsatisfactory tasks. Another alternative of indirect measure is the average performance indicator in Eq. 4, which, although efficient, may not be accurate enough. Hence, we make a trade-off to fine-tune the pre-trained real-SR network θ on all unsatisfactory tasks simultaneously through joint-training to obtain a new network ˆ θ . Then, we test the fine-tuned network ˆ θ on each degradation task with an available validation set and compute a performance improvement score (PIS) defined as:

where D val i represents the validation set of an unsatisfactory task τ i , I is an IQA (image quality assessment) metric such as PSNR, and s i is the PIS of task τ i . We then select the tasks with PIS larger than some threshold to form a task group, which should have small conflicts as they dominate the fine-tuning process. We repeat this process to find the rest of task groups as described in Alg. 1.

4.2 TGSR: A Task Grouping Based Real-SR Method

With the identified degradation task groups G = { G 1 , G 2 , .., G c } , we adopt a straightforward method to fine-tune the pre-trained real-SR network, referred to as TGSR. We first form ˆ G = { G 0 , G} = { G 0 , G 1 , G 2 , .., G c } , where G 0 represents the entire degradation space and serves for preventing

GT

Group 0

Group 1

Group 2

Group 3

Group 4

catastrophic forgetting during the fine-tuning process. Then, we randomly select a group from ˆ G and randomly sample a task from the chosen group for fine-tuning. This approach effectively increases the inclusion of the unsatisfactory tasks during the fine-tuning process and weights their likelihood of being chosen based on their respective group size (i.e., tasks in smaller groups have a higher probability to be selected). In essence, it is similar to the task re-weighting approach commonly used in multi-task learning.

5 Experiments

5.1 Experimental Setup

Datasets and evaluation. We employ DIV2K [1], Flickr2K [1] and OutdoorSceneTraining [40] datasets to implement our task grouping algorithm and train the TGSR network. For evaluation, we use DIV2K validation set to construct a DIV2K5G dataset consisting of 5 validation sets according to the divided 5 different degradation groups by the task grouping algorithm, as shown in Fig. 3. Each validation set contains 100 image pairs. In addition to the task group based test set, we employ the synthetic test set AIM2019 [29] and DIV2K_random for evaluation. The DIV2K_random is generated using the RealESRGAN degradation model on the DIV2K validation set. Furthermore, we incorporate the real-world test set, RealSR set [4], into our evaluation process. All evaluations are conducted on × 4 SR and PSNR is computed on the Y channel of YCbCr color space.

Implementation details . Weadopt the high-order degradation model proposed by RealESRGAN [39] as the real-SR degradation model in our experiments. To compute the performance indicator, N ( 4 × 10 3 ) degradation tasks are sampled from the whole degradation space. The single-task network is fine-tuned from the pre-trained RealESRNet model for 100 iterations. The performance indicator is computed based on the average of the last 10 iterations considering the instability of the training procedure. To implement the degradation grouping, the degradation tasks with the top 40% higher performance indicators ( 1 . 6 × 10 3 degradation tasks) are selected as unsatisfactory tasks. Then, we fine-tune the pre-trained RealESRNet for 1 × 10 4 iterations based on all unsatisfactory tasks. After that, we evaluate the model on Set14 [49] for each unsatisfactory degradation task. By using the proposed performance improvement score, we divide the degradation tasks into five groups based on the thresholds of [0.8, 0.6, 0.4, 0.2]. According to this set of thresholds, we obtain four groups with the number of tasks in each group being [14, 29, 84, 200] as groups1-4. Then, we label the entire degradation space beyond the four degradation groups as Group0. To avoid forgetting the satisfactory tasks, we uniformly sample degradation tasks from each group to fine-tune the pre-trained real-SR network. Other basic training settings follow RealESRGAN.

5.2 Comparison with State-of-the-Art

We compare our TGSR with the state-of-the-art methods, including SRGAN [24], ESRGAN [41], RDSR [23], MM-RealSR [31] BSRGAN [50], SwinIR [26], RealESRGAN [39], DASR [25] and HAT [6, 5]. Officially released pre-trained models are used for the compared methods.

Our TGSR improves the overall performance. The quantitative results of different methods are presented in Tab. 1. ESRGAN is based on a single-degradation model (i.e., Bicubic setting), so it performs worst under the multi-degradation evaluation system. RDSR is an MSE-based method, so it achieves the highest PSNR but performs rather poorly (the second worst) performance on LPIPS. BSRGAN, RealSwinIR, and DASR employ a one-order degradation model. They sacrifice performance on PSNR for better perceptual quality (reflected in low LPIPS). MM-RealSR and

STATE BANK

Input

ESRGAN

RDSR

MM-RealSR

BSRGAN

RealSwinIR

DASR

STATE BANK

STATE BANK

STATE BANK

STATE BANK

STATE BANK

STATE BANK

STATE BANK

MARKET

MARKET

MARKET

MARKET

MARKET

MARKET

MARKET

MARKETE

(ours)

RealESRGAN utilize a more complex high-order degradation. As a result, they achieve better LPIPS for real-SR evaluation. However, the two approaches obtain low PSNR performance due to the great difficulty of optimizing. Notably, we can enhance the performance of the pre-trained real-SR network by fine-tuning it on the identified Task Groups (TG). This improvement is particularly significant in Groups 1-4. For instance, on RealESRGAN-TG, we can observe a maximum boost of 1 dB in PSNR and 0.03 in LPIPS.

Our TGSR demonstrates significant superiority when applied to a large degradation space.

It is noticeable that RealESRGAN-TG outperforms RealESRGAN even in Group0. This indicates that our approach enhances real-SR performance across nearly the entire degradation space, not just the specifically addressed Groups 1-4. In addition to the identified task groups, our TGSR can also achieve performance gains on the randomized synthetic test sets DIV2K_random and AIM2019 test set as shown in Tab. 2. Additionally, our method achieves a gain on the real scene test set RealSRset. These results clearly demonstrate that our approach does not compromise the performance of certain degradation tasks to enhance the performance of others.

RealHAT

RealHAT-TG

Input

GT

RealSwinIR

RealSwinIR-TG

Input

GT

Our TGSR obtains better visual results than other methods. The visual results of different methods are shown in Fig. 4 and Fig. 5. TGSR significantly improves existing methods by removing various degradation artifacts and noticeable unpleasant artifacts. For the first four rows of images with complex degradation, the other methods cannot remove the degradation or generate unacceptable artifacts. In contrast, our TGSR handles the degradation well and produces visually pleasant results. For the image on the last row, our method can generate more realistic results than other methods with clear textures. In the RealSwinIR results of Fig. 5, we can observe the unpleasant artifacts in the flat area of Flower image and the fur of cat image. In the RealHAT results, we also find semantic textures appearing where they shouldn't be. For example, a tree texture appears on the train , and fur-like textures are present in the background of the Wolf image. However, our method can remove these artifacts by fine-tuning the real-SR network on identified task groups.

5.3 Ablation and Analysis

Effectiveness of the performance indicator. 1) In Fig. 6 (a) and (b), we present the PSNR improvement achieved by fine-tuning the real-SR network on individual tasks with lower and higher performance indicators. It can be observed that tasks with higher performance indicators exhibit a significant PSNR improvement (about 0.4-1.4dB), while tasks with lower performance indicators

1

2

3

4

5

6

7

8

9

10

Degradation task

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

PSNR distance(dB)

1

2

3

4

5

6

7

8

9

10

Degradation task

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

PSNR distance(dB)

Real-SR network

Single-task network

Lower performance indicator:

Unsatisfactory task

Higher performance indicator:

Satisfactory task

(a) Lower performance indicator (b) Higher performance indicator

(c) Visual comparison

G0

G2

G3

G4

R1

R2

R3

R4

Validation sets

0.0

0.2

0.4

0.6

0.8

1.0

PSNR distance(dB)

TGSR

TGSR-Random

(a) Comparison on PSNR.

G0

G2

G3

G4

R1

R2

R3

R4

Validation sets

0.000

0.005

0.010

0.015

0.020

0.025

0.030

LPIPS distance

TGSR

TGSR-Random

(b) Comparison on LPIPS.

TGSR

Random grouping: Unsatisfactory task

Task grouping: Unsatisfactory task

(c) Visual comparison

show only a small PSNR improvement (about 0.2 dB). This finding indicates that our proposed performance indicator can effectively distinguish between satisfactory and unsatisfactory tasks for a real-SR model in a large degradation space. Additionally, Fig. 6 (c) illustrates that significant visual improvements can be achieved with the unsatisfactory degradation task. 2) Directly fine-tuning a model for each task incurs a high computational cost (empirically requiring at least 10,000 iterations). However, our performance indicator requires only 100 iterations of fine-tuning. It suggests that utilizing the performance indicator can be 100 times faster than direct fine-tuning for distinguishing the unsatisfactory and satisfactory degradation tasks. This further illustrates the superiority of the proposed performance indicator.

Effectiveness of our task grouping algorithm. We compare the performance of RealESRGAN-TG with our task grouping and random grouping to demonstrate the effectiveness of our method. As shown in Fig. 7 (a) and (b), random grouping can only bring very limited performance gains of about 0.2dB on PSNR and about 0.01 on LPIPS. In contrast, the groups generated by our task grouping algorithm show a significant improvement up to 1dB on PSNR and 0.03 on LPIPS. These results demonstrate the significance of a well-designed task grouping approach. Furthermore, Fig. 7 (c) shows that random grouping can not select the unsatisfactory task, while our task grouping algorithm can find it.

Study of the performance upper bound. (1) We randomly select 100 single tasks and then add the corresponding small range of similar degradation parameters to each task, named single-task network with range (see details in appendix). Fig. 8 shows that the existence of related tasks can improve the upper bound of some tasks (about 40%). However, there has been a performance drop for other tasks due to similar degradation parameters that may not strictly represent similar tasks. (2) To further show the superiority of our TGSR, we fine-tune the RealESRGAN model on each group to obtain

0

20

40

60

80

100

Validation sets

0.0

0.5

1.0

1.5

2.0

PSNR distance(dB)

Single-task network

Single-task network with range

their empirical performance upper bound for these degradation groups, and the results are denoted as RealESRGAN-SG. As presented in Tab. 3, the fine-tuned models exceed the baseline models and obtain significant performance improvement on LPIPS. Although RealESRGAN-TG cannot surpass the empirical upper bound performance on LPIPS, it still achieves comparable performance. Moreover, our RealESRGAN-TG obtains higher PSNR compared to RealESRGAN-SG. This further shows the superiority of our method and suggests that multi-task learning enhances the performance of the specific task.

Impact of the number of task groups. In Tab. 4, we group the unsatisfactory tasks into more compact new groups using new thresholds of [0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2], with the corresponding number of degradation tasks being [14, 15, 14, 31, 53, 116, 258]. The results demonstrate that our TGSR method exhibits steady improvement on the unsatisfactory tasks in Groups 1-7, while achieving comparable performance on the satisfactory tasks in Group 0. This indicates that our approach can be applied with a more fine-grained division of unsatisfactory tasks based on user requirements.

6 Conclusion

We have re-examined the real-SR problem through the lens of multi-task learning and introduced a novel approach, TGSR, to address task competition in real-SR. TGSR aims to identify and enhance the degradation tasks where a real-SR model underperforms. It involves a task grouping method and a simple fine-tuning approach using the identified task groups. Comprehensive evaluation confirms the effectiveness of our approach in producing high-quality super-resolution results in real-world scenarios. One potential limitation of our method is the need for adequate sampling from the degradation space.

Acknowledgments and Disclosure of Funding

This work was supported in part by the National Natural Science Foundation of China under Grant (62276251,62272450), the Joint Lab of CAS-HK, the National Key RD Program of China (NO. 2022ZD0160100), and in part by the Youth Innovation Promotion Association of Chinese Academy of Sciences (No. 2020356).

References

Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , volume 3, page 2, 2017.

Sefi Bell-Kligler, Assaf Shocher, and Michal Irani. Blind super-resolution kernel estimation using an internal-gan. Advances in Neural Information Processing Systems , 32, 2019.

Felix JS Bragman, Ryutaro Tanno, Sebastien Ourselin, Daniel C Alexander, and Jorge Cardoso. Stochastic filter groups for multi-task cnns: Learning specialist and generalist convolution kernels. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1385-1394, 2019.

Jianrui Cai, Hui Zeng, Hongwei Yong, Zisheng Cao, and Lei Zhang. Toward real-world single image super-resolution: A new benchmark and a new model. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3086-3095, 2019.

Xiangyu Chen, Xintao Wang, Wenlong Zhang, Xiangtao Kong, Yu Qiao, Jiantao Zhou, and Chao Dong. Hat: Hybrid attention transformer for image restoration. arXiv preprint arXiv:2309.05239 , 2023.

Xiangyu Chen, Xintao Wang, Jiantao Zhou, Yu Qiao, and Chao Dong. Activating more pixels in image super-resolution transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 22367-22377, 2023.

Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In International conference on machine learning , pages 794-803. PMLR, 2018.

Zhao Chen, Jiquan Ngiam, Yanping Huang, Thang Luong, Henrik Kretzschmar, Yuning Chai, and Dragomir Anguelov. Just pick a sign: Optimizing deep multitask models with gradient sign dropout. Advances in Neural Information Processing Systems , 33:2039-2050, 2020.

Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Learning a deep convolutional network for image super-resolution. In European conference on computer vision , pages 184-199. Springer, 2014.

Chris Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn. Efficiently identifying task groupings for multi-task learning. Advances in Neural Information Processing Systems , 34:27503-27516, 2021.

Yuan Gao, Haoping Bai, Zequn Jie, Jiayi Ma, Kui Jia, and Wei Liu. Mtl-nas: Task-agnostic neural architecture search towards general-purpose multi-task learning. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition , pages 11543-11552, 2020.

Yuan Gao, Jiayi Ma, Mingbo Zhao, Wei Liu, and Alan L Yuille. Nddr-cnn: Layerwise feature fusing in multi-task cnns by neural discriminative dimensionality reduction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 3205-3214, 2019.

Jinjin Gu, Hannan Lu, Wangmeng Zuo, and Chao Dong. Blind super-resolution with iterative kernel correction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1604-1613, 2019.

Yuchao Gu, Xintao Wang, Liangbin Xie, Chao Dong, Gen Li, Ying Shan, and Ming-Ming Cheng. Vqfr: Blind face restoration with vector-quantized dictionary and parallel decoder. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVIII , pages 126-143. Springer, 2022.

SHI Guangyuan, Qimai Li, Wenlong Zhang, Jiaxin Chen, and Xiao-Ming Wu. Recon: Reducing conflicting gradients from the root for multi-task learning. In The Eleventh International Conference on Learning Representations .

Michelle Guo, Albert Haque, De-An Huang, Serena Yeung, and Li Fei-Fei. Dynamic task prioritization for multitask learning. In Proceedings of the European conference on computer vision (ECCV) , pages 270-287, 2018.

Weidong Hu, Wenlong Zhang, Shi Chen, Xin Lv, Dawei An, and Leo Ligthart. A deconvolution technology of microwave radiometer data using convolutional neural networks. Remote Sensing , 10(2):275, 2018.

Yan Huang, Shang Li, Liang Wang, Tieniu Tan, et al. Unfolding the alternating optimization for blind super resolution. Advances in Neural Information Processing Systems , 33:5632-5643, 2020.

Adrián Javaloy and Isabel Valera. Rotograd: Gradient homogenization in multitask learning. In ICLR , 2022.

Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 7482-7491, 2018.

Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1646-1654, 2016.

Iasonas Kokkinos. Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 6129-6138, 2017.

Xiangtao Kong, Xina Liu, Jinjin Gu, Yu Qiao, and Chao Dong. Reflash dropout in image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6002-6012, 2022.

Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew P Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic single image super-resolution using a generative adversarial network. In CVPR , volume 2, page 4, 2017.

Jie Liang, Hui Zeng, and Lei Zhang. Efficient and degradation-adaptive network for real-world image super-resolution. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVIII , pages 574-591. Springer, 2022.

Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In IEEE International Conference on Computer Vision Workshops , 2021.

Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. Advances in Neural Information Processing Systems , 34:18878-18890, 2021.

Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Philip S Yu. Learning multiple tasks with multilinear relationship networks. Advances in neural information processing systems , 30, 2017.

Andreas Lugmayr, Martin Danelljan, Radu Timofte, Manuel Fritsche, Shuhang Gu, Kuldeep Purohit, Praveen Kandula, Maitreya Suin, AN Rajagoapalan, Nam Hyung Joon, et al. Aim 2019 challenge on real-world image super-resolution: Methods and results. In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW) , pages 3575-3583. IEEE, 2019.

Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3994-4003, 2016.

Chong Mou, Yanze Wu, Xintao Wang, Chao Dong, Jian Zhang, and Ying Shan. Metric learning based interactive modulation for real-world super-resolution. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVII , pages 723-740. Springer, 2022.

Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098 , 2017.

Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, and Anders Søgaard. Latent multitask architecture learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 4822-4829, 2019.

Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. Advances in neural information processing systems , 31, 2018.

Jiayi Shen, Xiantong Zhen, Marcel Worring, and Ling Shao. Variational multi-task learning with gumbel-softmax priors. Advances in Neural Information Processing Systems , 34:21031-21042, 2021.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR , 2015.

Trevor Standley, Amir Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? In International Conference on Machine Learning , pages 9120-9132. PMLR, 2020.

Longguang Wang, Yingqian Wang, Xiaoyu Dong, Qingyu Xu, Jungang Yang, Wei An, and Yulan Guo. Unsupervised degradation representation learning for blind super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10581-10590, 2021.

Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan. Real-esrgan: Training real-world blind super-resolution with pure synthetic data. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1905-1914, 2021.

Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy. Recovering realistic texture in image super-resolution by deep spatial feature transform. arXiv preprint arXiv:1804.02815 , 2018.

Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Chen Change Loy, Yu Qiao, and Xiaoou Tang. Esrgan: Enhanced super-resolution generative adversarial networks. 2018.

Zhang Wenlong, Liu Yihao, Chao Dong, and Yu Qiao. Ranksrgan: Generative adversarial networks with ranker for image super-resolution. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2021.

Yanze Wu, Xintao Wang, Gen Li, and Ying Shan. Animesr: Learning real-world super-resolution models for animation videos. In Advances in Neural Information Processing Systems .

Liangbin Xie, Xintao Wang, Chao Dong, Zhongang Qi, and Ying Shan. Finding discriminative filters for specific degradations in blind super-resolution. Advances in Neural Information Processing Systems , 34:51-61, 2021.

Liangbin Xie, Xintao Wang, Shuwei Shi, Jinjin Gu, Chao Dong, and Ying Shan. Mitigating artifacts in real-world video super-resolution models. arXiv preprint arXiv:2212.07339 , 2022.

Fanghua Yu, Xintao Wang, Mingdeng Cao, Gen Li, Ying Shan, and Chao Dong. Osrt: Omnidirectional image super-resolution with distortion-aware transformer. arXiv preprint arXiv:2302.03453 , 2023.

Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems , 33:5824-5836, 2020.

Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3712-3722, 2018.

Roman Zeyde, Michael Elad, and Matan Protter. On single image scale-up using sparserepresentations. In International conference on curves and surfaces , pages 711-730. Springer, 2010.

Kai Zhang, Jingyun Liang, Luc Van Gool, and Radu Timofte. Designing a practical degradation model for deep blind image super-resolution. arXiv preprint arXiv:2103.14006 , 2021.

Kai Zhang, Wangmeng Zuo, and Lei Zhang. Learning a single convolutional super-resolution network for multiple degradations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 3262-3271, 2018.

Wenlong Zhang, Xiaohui Li, Xiangyu Chen, Yu Qiao, Xiao-Ming Wu, and Chao Dong. Seal: A framework for systematic evaluation of real-world super-resolution. arXiv preprint arXiv:2309.03020 , 2023.

Wenlong Zhang, Yihao Liu, Chao Dong, and Yu Qiao. Ranksrgan: Generative adversarial networks with ranker for image super-resolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3096-3105, 2019.

Wenlong Zhang, Guangyuan Shi, Yihao Liu, Chao Dong, and Xiao-Ming Wu. A closer look at blind super-resolution: Degradation models, baselines, and performance upper bounds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 527-536, 2022.

Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image superresolution using very deep residual channel attention networks. In ECCV , 2018.

Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. Residual dense network for image super-resolution. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2018.

7 Appendix

7.1 More Details of TGSR

In this section, we provide more training details of our TGSR. As mentioned in the main paper, we follow the RealESRGAN setting to train the real-SR network by standard GAN-based SR loss, including GAN loss L GAN , L1 loss L 1 , and perceptual loss L per . The total loss can be defined as follows:

The loss weights w 1 , w 2 , and w 3 are set to 1, 1, and 0.1, respectively. Specifically, for L L 1 , we calculate the pixel loss as the L 1 distance | ˆ y -y | , where ˆ y and y denote the reconstructed HR image and the ground-truth HR image respectively. For the perceptual loss L per, we extract the conv1, conv2, conv3, conv4, conv5 feature maps of ˆ y and y using the pre-trained VGG19 network [36], and the loss is calculated as the weighted sum of the respective L 1 distances between the feature maps of ˆ y and y , with weights set to [0.1, 0.1, 1, 1, 1] for each layer. For the adversarial loss L GAN, we use a U-Net discriminator with spectral normalization.

7.2 More Ablation Studies and Additional Experimental Results

7.2.1 Impact of the Increased Task Volume

In the main paper, we conducted experiments with a total of 4,000 (4k) degradation tasks. In this subsection, we have increased the number of tasks to 10,000 (10k) by sampling from the degradation space, in order to investigate the impact of the increased task volume on the results. In Fig. 9, we draw the histograms of the performance indicators computed with 4k (a) and 10k (b) degradation tasks respectively. It can be seen that their distributions are quite similar.

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

Performance indicator

0.00

0.02

0.04

0.06

0.08

Probability

(a)

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

Performance indicator

0.00

0.02

0.04

0.06

0.08

Probability

(b)

Furthermore, we use the same setup as in the main paper to do task grouping with the 10k degradation tasks and generate the test set for each group. We then compare the performance of TGSR trained with the 10k degradation tasks (TGSR_10k) and the 4k degradation tasks (TGSR_4k). In Tab. 5, we observe that the performance of TGSR_4k on the dataset DIV2K5G_10K is comparable to that of TGSR_10k. This finding suggests that increasing the number of tasks from 4,000 to 10,000 does not lead to significant improvements in performance for TGSR.

7.2.2 Necessity of Iterative Task Grouping

In Tab. 6, we provide the changes in performance improvement scores for various tasks across four training loops in the task grouping process in Alg. 1. It can be observed that many tasks show variations in their performance improvement scores across different training loops. For example, degradation task 5 exhibits a performance improvement of 0.57dB in the first training loop and 0.72dB in the third loop. This suggests that certain tasks are given priority during training in each loop, which leads to limited performance improvement for other tasks due to task competition. The observation supports the necessity of iterative task grouping as outlined in Alg. 1.

7.2.3 Comparison with Single-task Network

It is commonly believed that a single-task network fine-tuned specifically for a particular degradation task can achieve the upper bound of performance for that task. However, in Fig. 10, we compare the PSNR improvement over RealESRGAN between our TGSR and the single-task network on a set of randomly selected tasks. Interestingly, we observe that the performance of the single-task network even decreases for tasks 1 and 4 in terms of PSNR, while our TGSR, which utilizes task grouping for multi-task learning, consistently improves performance across all tasks. On average, TGSR achieves a PSNR improvement of 0.74dB and an LPIPS improvement of 0.04, whereas the single-task network only achieves a PSNR improvement of 0.30dB and an LPIPS improvement of 0.03. This demonstrates the effectiveness of the task grouping and multi-task learning strategy in TGSR, which leverages the shared information across tasks to improve the generalization ability of the model and achieve better overall performance.

7.2.4 Impact of the Increased Volume of Unsatisfactory Tasks

Tab. 7 presents the results of TGSR with additional unsatisfactory tasks. We categorize the unsatisfied tasks into 9 groups based on thresholds [0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2, 0.15, 0.1] (0.15 and 0.1 are used to include more unsatisfactory tasks compared to Tab. 4), with the corresponding number of degradation tasks as [14, 15, 14, 31, 53, 116, 258, 190, 206]. We can observe that TGSR consistently achieves performance improvement on the unsatisfactory tasks in Groups 1-9 while maintaining comparable results on the satisfactory tasks in Group 0. It is worth noting that as the threshold for dividing the groups decreases, the performance gain becomes gradually smaller.

0

1

2

3

4

5

Degradation task

0.5

0.0

0.5

1.0

1.5

PSNR distance(dB)

Single-task network

TGSR

(a) Performance comparison in PSNR.

0

1

2

3

4

5

Degradation task

0.00

0.02

0.04

0.06

0.08

0.10

LPIPS distance

Single-task network

TGSR

(b) Performance comparison in LPIPS.

7.3 More Visual Results

Fig. 11 presents additional sample images from our validation datasets, including some unsatisfactory degradation tasks from Groups 1-4 and some satisfactory degradation tasks from Group 0. These images are accompanied by the corresponding ground-truth images. Fig. 12 presents an additional qualitative comparison of competing methods on DIV2K5G. Fig. 13 presents an additional qualitative comparison of competing methods on real-world images. Through the grouping and handling of unsatisfactory degradation tasks, our TGSR exhibits enhanced generalization ability when applied to real-world images.

7.3.1 More Sample Images in DIV2K5G

HR

Group 0

Group 1

Group 2

Group 3

Group 4

STATE BANK OF

STATEBANKOF

STATE BANKOF

STATE BANK OF

STATE BANK OF

MARKETBRA

MARKETBRA

RSETES

MARKETBRA

MARKETBRA

MARKET BRA

7.3.2 More Qualitative Comparisons on DIV2K5G

ESRGAN

RDSR

MM-RealSR

BSRGAN

Input

RealSwinIR

DASR

ESRGAN

RDSR

MM-RealSR

BSRGAN

Input

RealSwinIR

DASR

ESRGAN

RDSR

MM-RealSR

BSRGAN

Input

RealSwinIR

DASR

ESRGAN

RDSR

MM-RealSR

BSRGAN

Input

RealSwinIR

DASR

ESRGAN

RDSR

MM-RealSR

BSRGAN

Input

RealSwinIR

DASR

7.3.3 More Qualitative Comparisons on Real-world Images

ESRGAN

RDSR

MM-RealSR

BSRGAN

Input

SwinIR

DASR

ESRGAN

RDSR

MM-RealSR

BSRGAN

Input

SwinIR

DASR

ESRGAN

RDSR

MM-RealSR

BSRGAN

Input

SwinIR

DASR

ESRGAN

RDSR

MM-RealSR

BSRGAN

Input

SwinIR

DASR

ESRGAN

RDSR

MM-RealSR

BSRGAN

Input

SwinIR

DASR

時间

时間

時間

時間

時间

效率

時間间

時间

時間

時间

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper revisits the real-world super-resolution task from the perspective of multi-task learning, considering each type of degradation as a separate task. However, there are countless types of degradation in the real world, which often results in severe task competition in previous methods. The authors propose a task grouping strategy to address this problem.\nFurthermore, they present a multi-task learning framework suitable for real SR tasks, guiding the target SR model by the task group label probability. Experimental results showcase an advanced performance improvement by the proposed TGSR across a wide range of degradation types.\n\nSTRENGTHS:\nThis paper addresses real-world super-resolution tasks from the multi-task learning perspective, which I think is novel to this field. The paper effectively dissects the issues present in previous methods from a multi-task learning perspective and provide in-depth insight.\n\nI think that the degradation task grouping strategy proposed in the paper may sheds new light on the analysis and study of the blind SR tasks.\n\nWEAKNESSES:\nDetailed selection strategy for different degradation tasks in Section 3.1 and Figure 1 are recommended.\n\nHyper-parameters in the proposed task grouping strategy, such as the number of task group and division threshold, are given and fixed in this paper, which is not motivated to me. I think these could lead to potential impact for the proposed method, but detailed analysis or ablation is missing is this paper. The authors should provide more comprehensive analyses to address these concerns.\n\nIn the degradation type grouping strategy, the threshold for each group division is determined manually, and the paper does not mention ablation in the number of groups or threshold, which could lead to potential imprecision in the division strategy.\n\nQUESTIONS:\nMy concerns regarding the evaluation data DIV2K5G and its close connection with the proposed TGSR model are valid. The potential for an inherent bias exists since the dataset was constructed using the proposed task grouping algorithm, which is closely tied with the iteratively trained and fine-tuned SR model.\n\nIt would indeed be beneficial for the authors to evaluate the TGSR model on the standard benchmark dataset, which would provide a more independent measure of the effectiveness of the multi-task learning-based task grouping strategy.\n\nAdditionally, performing comparisons on the RealSR dataset or a randomly generated benchmark by BSRGAN or RealESRGAN can be beneficial. These additional evaluations can provide further evidence to the superiority of the proposed TGSR.\n\nLIMITATIONS:\nNone.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors rethink the real-world super-resolution problem from the perspective of multi-task learning. And point out the primary challenge: task competition problem. To address this issue, they propose a task grouping method to identify unsatisfactory tasks and introduce TGSR to handle them separately, thereby eliminating task competition. Extensive experiments demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS:\n1.\tTaking real-SR as a multi-task learning problem offers a novel perspective that can yield further insights and considerations. For instance, how to effectively sample degradation in a large degradation space? What relationships exist between different tasks?\n\n2.\tThe authors present a clear motivation in the form of task competition, as current real-SR networks are unable to perfectly handle all cases of degradation.\n\n3.\tThe paper includes extensive ablation studies. Interestingly, the authors found that even when fine-tuning directly on a single degradation task, many cases still do not perform as well as a range of degradation. This suggests that traditional blind super-resolution networks based on kernel estimation may no longer be applicable within large degradation spaces. This is a highly insightful finding.\n\nWEAKNESSES:\n1.\tThe authors should provide the calculation time of the performance indicator, although the performance indicator is obviously faster than directly fine-tuning a single-task network. \n\n2.\tThe groups 1-4 in Figure 4 of the main text seem to be inconsistent with the samples in the supp file, and Table 1 also shows that group 1 should be a low-quality difficult case (with the lowest PSNR value).\n\nQUESTIONS:\nSee above weakness, authors need to address them well.\n\nLIMITATIONS:\nYes. Besides, since this work still uses synthetic data. It would be better to consider low quality images from real scenes.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims at the task conflict issue of real-world image super-resolution (SR) with multiple degradation tasks, and proposes a task grouping approach to group similar tasks together to mitigate task competition. In addition, this paper designs a real-SR network called TGSR (task grouping-based real-SR network), which leverages the identified task groups to train a task group classifier and use the predicted information to generate modulation signals for image restoration.\n\nSTRENGTHS:\n1. a new look at real SR from a multi-task learning perspective.\n2. a performance indicator based on gradient updates to efficiently identify the degradation tasks.\n\nWEAKNESSES:\n1. The authors compare the PSNR distance between the real-SR model trained on entire degradation space introduced by Real-ESRGAN [34] and 100 types of single-task models fine-tuned on the specific-degradation, and highlight some degradation tasks not well solved by the real-SR model as unsatisfactory tasks. However, they only number 1-100 for different tasks, and do not present the detailed settings (blur type, kernel width, noise type, noise level, and jpeg compression) of each degradation task. Such inadequate expression makes readers including me confued as to whether there is a relationship between PSNR distance and degradation type. \n\n2. The authors claim that \"... divide them (unsatisfactory degradation tasks) into multiple task groups based on their similarity to reduce negative transfer.\" However, the task similarity computed by the PSNR distance is not convincing, and it is worth to further studying the relationship between the task grouping strategy and the degradation distribution. \n\n3. At lines 177-178, \"in this study, we use the found degradation task groups to train a task group classifier, which is then integrated into a standard real-SR framework\". I want to know how to select the number of task groups, and how do different numbers of groupings affect the final performance. If the number of groupings is set to infinity, is the task group classifier equivalent to the well-studied degradation estimator?\n\nQUESTIONS:\n1. AT lines 166-167, \"we make a trade-off to fine-tune the pre-trained real-SR network on all unsatisfactory tasks simultaneously through joint-training\". What is the joint learning in this paper? \n\n\n2. It is unfair to conduct the evaluation experiments on the grouped DIV2k (namely, DIV2K5G) with 5 different degradation groups. The authors should also conduct a fair comparison with Sota methods on the well-used benchmarks, such as SwinIR [A1], DASR [A2]. \n\n[A1] Liang J, Cao J, Sun G, et al. Swinir: Image restoration using swin transformer[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2021: 1833-1844.\n[A2] Wang L, Wang Y, Dong X, et al. Unsupervised degradation representation learning for blind super-resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 10581-10590.\n\nLIMITATIONS:\nnone", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper models a real-world image super-resolution (real-SR) from a multi-task learning perspective, that is treat real-SR as solving multiple distinct degradation tasks. To this end, the authors propose a task-grouping approach by grouping similar tasks together. Extensive experiments demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS:\nThe authors regard learning a real-SR model as a multi-task learning task and highlight the task competition problem. The authors develop a task grouping-based real-SR network (TGSR).\n\nWEAKNESSES:\n1. The novelty of this paper should be highlighted. This paper refers to the tasks that are solved by the real-SR network well or not as unsatisfactory and satisfactory tasks. Such a way is similar to [1] which classifies the images as simple, medium and hard. It would be better to highlight the novelty of the method.     \n    [1] ClassSR: A General Framework to Accelerate Super-Resolution Networks by Data Characteristic\n\n2. The multi-task Real-SR method seems to be not smart. There are infinite number of degradations. The method randomly samples 100 degradation tasks and obtains 100 fine-tuned models.\n\n3. The task-grouping approach groups similar tasks together. The similarity is based on the performance. It means that the low noisy degradation and weak blurry degradation are similar. However, low noise and strong noise are the same degradations, which should be treated as the same task.\n\n4. In Figure 1 (a), this paper uses the degradation models which are based on Real-ESRGAN. Each degradation on an image is a composite function which different degradations. Could you investigate the effect of such a case?\n\n5. The authors identify the unsatisfactory tasks by comparing the performance between the jointly-trained multi-task real-SR network and the fine-tuned single-task networks. What if directly compare the PSRN (like [1]) with a threshold?\n\n6. Some experiment details are not clear. Is the real-SR model fine-tuned on a pre-trained Real-ESRGAN or trained from scratch?\n\n7. The performance is not state-of-the-art. In Table 1, the proposed method is worse than RDSR. In Table 2, RealESRGAN-G has no advantage over RealESRGAN for Group0 and Group1. How to demonstrate the superiority of the proposed method. In Figure 5, the input images have much noise, not low-resolution. It would be better to provide SR results.\n\nQUESTIONS:\nPlease refer to the Weaknesses.\n\nLIMITATIONS:\nThe authors do not address the limitations in the paper. It would be better to include the above suggestions for improvement.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper casts real-world image SR as a multi-task learning task for complex image degradations. By comparing the performance gap between the real-SR model and the fine-tuned model for a specific degradation, the satisfactory task and unsatisfactory task are divided. And thus it proposes a task grouping method to address real-world SR.\n\nSTRENGTHS:\n+ The paper is well-written and easy to understand.\n+ It is interesting to analyze the real-world image SR from a multi-task learning view.\n+ The paper has considered many degradations for evaluation.\n\nWEAKNESSES:\n- The main concern is about the motivation. As claimed in Line 36-37, real SR faces a challenge of task competition or task conflict. But, \n1) it is still blind to explicitly represent or accurately model specific real degradations for the task of real SR. What is the meaning of \"task\" for real SR? What is the evidence of the \"task conflict\"? \n2) Although authors use the performance gap to define satisfactory task and unsatisfactory task, it seems to have a prerequisite that the results of pre-trained real-SR netwrok and single-task networks on each degradation task are reliable, especially based on their PSNR distance (Line 119).  This is rather unconvincing. It is hard for me to believe this demonstration for the motivation in Sec.3 and Fig.1. \n3) Line 126: \"Our analysis in Sec. 3 shows that when a real-SR network is tasked with many degradation tasks, they may compete for model capacity or interfere with each other, resulting in a significant decline in performance for certain tasks.\"  I could not understand what are the evidences for the claimed completition. Besides, the performance decline may be not cetainly resulted from the claimed completition and be influenced by other factors. \n\nOverall, the analyses on the motivation are rather unconvincing and it is hard for me to believe the rationality of the work.\n\n- Despite the analyses in Sec.3, it does not explain what cases the satisfactory task or unsatisfactory task actually indicate (it is too simple based on a PSNR threshold) and why they have these differences. I think this explain is important to well understand the real SR task and the motivation of taking it as a multi-task learning. Besides, how to undestand the derived degradation task groups in Fig.4?\n\n- Method (Sec.4). It could not ensure the reliability of using pre-trained real-sr network and single-task networks.\n\n- Evaluation:\n1. Line 204: \"we evaluate the model on Set14....\" But, Set14 has very limited number of images. Is this concincing?\n2. All the evaluations are based on RealESRGAN and there is no any evaluation on real data for real degration, not generated degradation by gan-based model. This is a very important demonstration extension for the work.\n\n-\n\nQUESTIONS:\nI have listed and explained my concerns on this paper in \"Weakness\". I expect the responses to those issues.\n\nLIMITATIONS:\nAuthors just claim one limitation of the sampling from the degradation space in \"Conclusion\". This is actually an important issue for this work. Thus the more demonstration on real data, not from realESRGAN would be more convincing.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-11"}