Harmony in Diversity: Improving All-in-One Image Restoration via Multi-Task Collaboration

Authors: gang  wu
junjun  jiang
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-06 07:51:28

Abstract: Deep learning-based all-in-one image restoration methods have garnered significant attention in recent years due to capable of addressing multiple degradation tasks. These methods focus on extracting task-oriented information to guide the unified model and have achieved promising results through elaborate architecture design. They commonly adopt a simple mix training paradigm, and the proper optimization strategy for all-in-one tasks has been scarcely investigated. This oversight neglects the intricate relationships and potential conflicts among various restoration tasks, consequently leading to inconsistent optimization rhythms.
In this paper, we extend and redefine the conventional all-in-one image restoration task as a multi-task learning problem and propose a straightforward yet effective active-reweighting strategy, dubbed $\textbf{Art}$, to harmonize the optimization of multiple degradation tasks. Art is a plug-and-play optimization strategy designed to mitigate hidden conflicts among multi-task optimization processes.
Through extensive experiments on a diverse range of all-in-one image restoration settings, Art has been demonstrated to substantially enhance the performance of existing methods. When incorporated into the AirNet and TransWeather models, it achieves average improvements of $\textbf{1.16}$ dB and $\textbf{1.21}$ dB on PSNR, respectively. We hope this work will provide a principled framework for collaborating multiple tasks in all-in-one image restoration and pave the way for more efficient and effective restoration models, ultimately advancing the state-of-the-art in this critical research domain. Code and pre-trained models are available at our project page https://github.com/Aitical/Art.

Main paper: 
 introduction: 
 In real-world scenarios, the observed images inevitably suffer from various types of degradation, such as noise, blur, rain, and haze [Karavarsamis et al. 2022 ]. These degradations deteriorate the visual quality of images and pose significant challenges to downstream tasks. Consequently, developing effective image restoration techniques [Ali et al. 2023 ] has become increasingly important for a wide range of applications, and been a fundamental research topic in recent decades [Banham and Katsaggelos 1997; Su et al. 2022] .Recently, deep-learning based single-task models have demonstrated exceptional performance in the respective domains, such as denoising [Liang et al. 2021; Zamir et al. 2022; Zhang et al. 2023b] , deblurring [Chen et al. 2022a; Pham et al. 2024; Wang et al. 2022 ], deraining [Chen et al. 2023 [Chen et al. , 2024;; Jiang et al. 2020; Zamir et al. 2021] , and dehazing [Cui et al. 2023; Qin et al. 2020 ]. However, these task-specific methods face a significant challenge in real-world scenarios: when they excel in a certain task, they may completely fail in other tasks. In this paper, we focus on the all-in-one image restoration, which can integrate knowledge across a collection of related tasks, and propose to refresh the performance of existing all-in-one methods by the propose Art, as shown in Figure 1 .Beyond the single-task model illustrated in Figure 2 (a), there has been growing research interest in developing unified models to handle multiple degradation tasks within a single model, thereby providing more comprehensive and practical solutions to real-world image restoration challenges [Chen et al. 2021; Li et al. 2022; Liu et al. 2022; Potlapalli et al. 2023; Valanarasu et al. 2022; Zhang et al. 2023a] . In [Chen et al. 2021] , an early attempt to address multiple tasks within a single training process introduces a multi-head and multi-output architecture with a shared backbone, as illustrated in Figure 2 (b) . Recent research has shifted attention to unified models, which is achieved by harnessing task-oriented information against SOTA all-in-one image restoration methods. When integrating Art to retrain existing models, it consistently enhances the performance of the previous ones across diverse all-in-one image restoration tasks, underscoring its efficacy and versatility in handling multiple degradation tasks.to inform and guide the learning process of the unified model [Li et al. 2022; Liu et al. 2022; Zhang et al. 2023a] . In [Li et al. 2022] , the authors introduce a self-supervised approach [He et al. 2020] to pretrain a degradation encoder, which is fixed and utilized to extract the degradation representations of the input for the training of the all-in-one model. The recent PromptIR [Potlapalli et al. 2023 ] and Prompt-In-Prompt [Li et al. 2023 ] further simplify the task-oriented guidance by an adaptive prompt learning approach, enabling endto-end multi-task training and achieving robust performance. With the development of ingenious architectures, significant strides have been made in these methods. Most recently, MioIR [Kong et al. 2024 ] adopts a novel training scheme to improve the all-in-one image restoration problem by a sequential and prompt learning strategy.However, we observe that these methods typically adopt mixed training, which involves combining the optimization of individual tasks together. Less attention is paid to the optimization process of all-in-one tasks, thereby overlooking the intricate relationships and potential conflicts among multiple degradations in the mixed training paradigm. Although MioIR [Kong et al. 2024 ] has improved the performance of all-in-one image restoration by elaborately changing the task settings, it fundamentally does not solve the conflict between tasks. To better illustrate this insight, we present the loss curves of a 7-task all-in-one training process in Figure 2 (c). It is evident that large oscillations exist during training, and inconsistent convergence occurs for different tasks. We argue that this uncertain dynamics in the training process is due to hidden conflicts and inconsistent optimization objectives between multiple degraded tasks. Therefore, failing to account for these drawbacks through simply mix training can lead to performance discrepancies of different degradation tasks and hinder the overall convergence for the all-in-one restoration model. This underscores the need for a more sophisticated approach to handle the intricate relationships among different restoration tasks and improve all-in-one image restoration models through multi-task collaboration.To address this challenge, we propose an enhanced all-in-one image restoration approach via multi-task reweighting, denoted as Art. As illustrated in Figure 1 (d), we extend the loss function of all-in-one tasks with active-reweighting for each individual task, and one can observe that retrained all-in-one model with the proposed Art achieves a more stable learning process. Specifically, Art explicitly formulates the restoration process as a multi-task learning problem, aiming to harmonize the optimization of multiple restoration tasks. To achieve this goal, we explore the convergence status of each individual task from its local and global optimization process and propose an active-reweighting strategy, which can equilibrate the optimization among different restoration tasks. In detail, this reweighting approach comprises two key components: a local convergence rate (LCR) and a global correction weight (GCW). The former addresses the challenge of inconsistent convergence within a single iteration, while the latter dynamically adjusts the optimization objective when a task deviates from its best performance. The proposed Art is straightforward yet effective in mitigating the detrimental effects of hidden conflicts and ensures a delicate balance among individual tasks. Notably, Art is a generalized optimization strategy that can be applied as a plug-and-play method to existing all-in-one approaches.The current landscape of all-in-one image restoration research is characterized by a diverse range of task settings and model architectures. To validate the efficacy of our approach, we have summarized several prevalent all-in-one experimental settings. For each experimental setting, we retrained the previous models with our Art approach. When our approach is incorporated into the AirNet and TransWeather methods, it achieves average improvements of 1.16 dB and 1.24 dB on PSNR, respectively. As depicted in Figure 1 , the results consistently demonstrate that our proposed Art yields significant improvements in various task settings and model architectures, highlighting its robustness and generalizability.The main contributions of our work can be summarized as follows:• We revisit all-in-one image restoration from the perspective of multi-task learning and propose a novel active approach, Art, that explicitly addresses the challenges of inconsistency and potential conflicts among multiple restoration tasks.  
the proposed method 2.1 all-in-one image restoration: 
 All-in-one image restoration aims to recover clean images from corrupted observations that suffer from multiple degradations simultaneously. Given a set of degraded images x and their corresponding high-quality ground truths y, the goal is to learn a restoration model 𝑓 𝜃 parameterized by 𝜃 that can effectively map the degraded inputs to the desired clean outputs. Mathematically, the optimization process of all-in-one image restoration can be formulated as: [Chen et al. 2021 ]. The model is trained separately for each task using a shared backbone and task-specific input and output heads, limiting its applicability. (c) Top: Architecture of current all-in-one methods, aiming to obtain a unified model through mixed training [Li et al. 2022; Potlapalli et al. 2023; Valanarasu et al. 2022 ]. Bottom: Loss curves during the training process, where collisions are observed in multi-task optimization. (d) Top: Our proposed Art, explicitly addressing all-in-one image restoration as a multi-task problem. Bottom: Loss curves of our approach, achieving a more robust learning process.EQUATIONwhere 𝐿 is a reconstruction loss, such as the Mean Absolute Error (MAE), which measures the difference between the restored images 𝑓 𝜃 (𝑥) and the ground truths y. The model parameters 𝜃 are optimized through an end-to-end training process to minimize the reconstruction loss.To address the challenges of hidden conflict in multiple degradation types, we extend the loss function of all-in-one image restoration tasks with explicit active-reweighting. In detail, we assume that the degradation type of each training sample is known, and we denote the input as 𝑥 𝑡 𝑖 , where 𝑖 ∈ 1, . . . , 𝑁 represents the sample index, 𝑡 ∈ 1, . . . , 𝐾 indicates the degradation type, 𝑁 is the total number of samples, and 𝐾 is the number of degradation types.We introduce a simple but effective modification of conventional loss function, which brings significant improvements to the optimization process. Specifically, we reformulate the objective function as:EQUATIONwhere 𝛼 𝑡 is an adaptive rebalancing coefficient for each degradation type, and 𝐿 𝑡 = 𝐿(X 𝑡 , Y 𝑡 ) represents the loss of individual task 𝑡. X 𝑡 and Y 𝑡 are the subsets of degraded images and their corresponding ground truths with degradation type 𝑡, respectively. When the parameter weights 𝜃 are updated by the chain rule of backpropagation, we obtain:EQUATIONThe adaptive reweighting coefficient 𝛼 𝑡 offers a straightforward approach to control the optimization of each task within the explicitly formulated multi-task loss function. As observed in Section 1, when a biased task is identified, increasing its corresponding task weight 𝛼 𝑡 can compel the parameter weights 𝜃 to rectify the solution. The primary challenge lies in determining the appropriate task weight 𝛼 𝑡 to adaptively balance the tasks. To address this, we propose a novel approach that examines the loss value of each individual task from two perspectives: its current convergence status and its global optimization direction.The local convergence rate provides valuable insights into the optimization progress of each task within a single iteration. By analyzing the rate of change in the loss value, we can identify tasks that are converging slower than others and adjust their corresponding task weights accordingly. This local perspective ensures that all tasks progress at a more consistent pace, preventing any single task from dominating the optimization process.Furthermore, we introduce a global optimization direction component that considers the overall trajectory of each task's loss value across historical iterations. This global perspective allows us to detect tasks that deviate significantly from their optimal path and dynamically adjust their task weights to steer them back towards their desired objectives. By incorporating this global correction mechanism, we can effectively mitigate the detrimental effects of conflicting gradients and promote a more harmonious optimization process.The final task weight 𝛼 𝑡 is obtained by combining the insights gained from both the current convergence status and the global optimization direction. This adaptive reweighting scheme enables our proposed method to dynamically adjust the influence of each task on the overall optimization process, ensuring a delicate balance between individual task progress and the collective performance of the multi-task learning model.In the following subsections, we delve into the details of computing the current convergence rate and the global optimization direction, and how they are seamlessly integrated to derive the adaptive rebalancing coefficient 𝛼 𝑡 . Through this meticulous approach, our method effectively addresses the challenges of multi-task learning in all-in-one image restoration, paving the way for more efficient and balanced optimization processes.local convergence rate. The local convergence rate factor 𝑠 𝑡 is introduced to balance the convergence rates of different restoration tasks. It is computed as the ratio between the current iteration loss 𝐿 𝑖𝑡𝑒𝑟 𝑡 and the previous iteration loss 𝐿 𝑖𝑡𝑒𝑟 -1 𝑡 for each degradation type 𝑡:EQUATION)The convergence rate factor 𝑠 𝑡 is instrumental in addressing biased tasks by dynamically adjusting the optimization speed of each task based on its convergence behavior. When a task's loss begins to increase, signaling a deviation from the desired convergence trajectory, 𝑠 𝑡 assumes a value greater than 1. This elevated value effectively amplifies the gradient weight of the corresponding task, compelling the optimization process to prioritize the correction of the suboptimal convergence. Moreover, there is potential influence of sample quality in mini-batch training, which can give rise to unreliable coefficients, and we propose a more robust convergence rate estimate with an exponential moving average (EMA) strategy. This approach involves smoothing the historical loss values to attenuate the impact of local fluctuations and anomalies. The smoothed convergence rate, denoted as s𝑡 , is formulated as follows:EQUATIONwhere 𝐿 𝑖𝑡𝑒𝑟 𝑡 and L𝑡 are loss values for the current and the smoothed one, respectively. The smoothed loss values are obtained using the EMA:EQUATIONwhere 𝛽 ∈ [0, 1] is a hyperparameter that controls the degree of smoothing. A larger 𝛽 value gives more weight to the historical losses, resulting in a smoother convergence rate factor. The smoothed loss value reflects the overall convergence state of the target task, allowing the local convergence rate to steadily capture the current convergence status.global correction weight. The global correction factor 𝑟 𝑡 is introduced to mitigate inter-task conflicts and prevent the performance degradation of individual tasks. It is computed as the ratio between the historical minimum loss 𝐿 𝑚𝑖𝑛 𝑡 and the current iteration loss 𝐿 𝑖𝑡𝑒𝑟 𝑡 for each degradation type 𝑡: The effectiveness of the global correction factor lies in its ability to dynamically adjust the task weights based on their relative performance. When a task's current loss is much higher than its historical minimum, 𝑟 𝑡 becomes greater than 1, indicating the necessity for increased attention to that task. By allocating more weight to the biased task, the global correction factor facilitates the steering of the optimization process towards a more robust solution.EQUATIONMoreover, it is crucial to recognize that directly utilizing the global correction factor can potentially introduce instability issues, akin to the local oscillations observed in the loss values. To address this concern, we propose the application of a logarithmic transformation to the current and minimal loss values, yielding a more stable result denoted as r𝑡 .Active Reweighting. To effectively manage the optimization of multiple degradation types within our multi-task framework, we have developed a novel strategy for adaptive task weighting 𝛼 𝑡 . This approach strategically integrates two key components: the local convergence rate, s𝑡 , and the global correction weight, r𝑡 . The combination of these factors enables a balanced and stable optimization process across various tasks.The adaptive task weight 𝛼 𝑡 is mathematically formulated as follows:EQUATIONwhere 𝜏 represents the temperature coefficient. By leveraging these adaptive weights, our framework can more effectively prioritize tasks that require attention, thereby enhancing the overall efficacy of the multi-task learning process.Our proposed active-reweighting strategy addresses a critical challenge in all-in-one image restoration: the hidden conflicts in multitask learning. The core of our approach is the adaptive coefficient 𝛼, composed of two key elements: the local convergence rate (LCR), s, and the global correction weight (GCW), r . These components work in tandem to dynamically balance the optimization process across multiple tasks.Table 1 presents our ablation study, demonstrating the effectiveness of each component. The baseline mixed training achieves a PSNR of 29.52 dB. Introducing LCR without smoothing improves PSNR to 29.66 dB, while smoothed LCR further enhances performance to 29.81 dB. This improvement underscores the importance of stabilizing convergence rates across tasks. The GCW proves  
experiment 3.1 experiment setting: 
 In this paper, we conduct extensive experiments to evaluate the effectiveness and generalization of our proposed Art framework.Considering core contribution of our Art is the rebalanced loss function, which is a plug-and-play component to existing models, we take four different experimental settings, covering a wide range of image restoration tasks and datasets, as presented in Table 2 . Detailed experimental settings are as follows:i. 'multiple degradation' setting: Following [Kong et al. 2024] , there are 7 different image restoration tasks. The multidegradation dataset is synthesized based on DF2K dataset, which provides the unbiased analysis of influence between tasks. ii. 'rain-haze-noise' setting: Following [Li et al. 2022 For all experimental settings described above, we take the corresponding state-of-the-art methods as our baselines. We strictly follow their original training configurations to ensure a fair comparison. Additionally, we retrain these baseline models using their official code to adapt to our experimental settings and provide a through evaluation of the effectiveness of our proposed Art.Center of the our Art framework is the proposed rebalanced loss function, which is straightforward yet effective. There are only an adaptive rebalancing weight is involved beyond existing methods.Here we want to discuss some other components and provide more detailed insights of our Art. Detailed ablation studies are provides as follows.Impact of EMA weight. To evaluate the impact of the weight 𝛽 in our proposed method, we conduct an ablation study with different values of 𝛽 ranging from 0 to 0.1. The results are presented in Table 4 . Notably, to better understand the influence of 𝛽, this study is conducted with only s instead of 𝛼, because the 𝛽 only works with s. When 𝛽 is set to 0, representing no EMA component, the PSNR reaches 29.67 dB. This serves as a baseline for evaluating the contribution of the EMA in our method. Setting 𝛽 to 0.01 and 0.001 yields a PSNR of 29.81 dB in both cases. This suggests that EMA strategy works effectively to capture the historical information and stabilize the training process, highlighting the contribution of our proposed local convergence rate. Finally, as we increase 𝛽 to 0.1, the PSNR induces to 29.79 dB.Impact of Temperature coefficient. To investigate the influence of the temperature parameter 𝜏 on our proposed method's performance, we conduct an ablation study by varying the values of 𝜏 from 0.1 to 10. The results are presented in Table 5 . When 𝜏 is set to a small value of 0.1, the PSNR reaches 29.59 dB. However, we observe that this value is too small to obtain a robust 𝛼, leading to oscillations during the training process. As we increase 𝜏 to 1, the PSNR improves to 29.87 dB, indicating that a higher temperature facilitates better optimization and convergence. The best performance is achieved when 𝜏 is set to 3, with a PSNR of 30.05 dB. This suggests that an appropriate balance is struck among the multiple tasks at this temperature value, enabling the model to effectively learn and adapt to the different task requirements. Further increasing 𝜏 beyond 3 leads to a slight decline in performance. This behavior can be attributed to the fact that higher values of 𝜏 provide a more uniform weight distribution among the multiple tasks, potentially impeding the flexibility of our adaptive rebalancing weight. Consequently, the model's ability to dynamically adjust the task weights based on their individual performance is hindered. It is worth noting that even with suboptimal hyper-parameter settings, our proposed method still outperforms the baseline models presented in [Kong et al. 2024] , which achieve a PSNR of 29.51 dB. This demonstrates the robustness and effectiveness of our approach in handling multi-task learning scenarios, even when the temperature parameter is not optimally tuned.'multiple degradation' setting. Table 6 demonstrates the superior performance of our proposed multi-task balanced learning approach, Art, compared to the vanilla mixed Training and the state-of-the-art sequential training strategy introduced by MiOIR [ Kong et al. 2024 ]. The sequential training strategy, denoted as -S in the table, aims to mitigate the interference between different tasks by gradually increasing the number of tasks in the training set. While this approach has been shown to improve upon the mixed training paradigm in all-in-one image restoration, it has limitations in terms of scalability to multiple tasks due to its sensitivity to the training order. In contrast, our Art method achieves significant improvements over the sequential training strategy using the most straightforward mixed training approach. By incorporating task-specific rebalancing coefficients, Art effectively equilibrates dynamics optimization and mitigates inter-task conflicts and surpasses the performance of the carefully designed sequential training strategy without the need for controlling the task order.Furthermore, we observe that the sequential training strategy introduced in MiOIR ultimately reduces to mixed training when all tasks are included in the final stage. Building upon this insight, we apply our multi-task reweighting strategy to the mixed training portion of the sequential training pipeline. The results demonstrate that our approach can further enhance the performance of sequential training, achieving notable improvements.Figure 3 presents a subjective comparison of the restored images, showcasing the superior image quality achieved by our retrained SRResNet model. The visual improvements are evident not only when compared to the baseline models but also when contrasted with the sequential training approach. This further highlights the effectiveness of our multi-task balanced learning method in enhancing the perceptual quality of the restored images."dhaze-derain-denoise" setting. Table 7 presents the experimental results for three distinct degradation tasks: rain, haze, and noise, which are widely investigated in the current research landscape of all-in-one image restoration. By employing our proposed method, Art, to retrain the AirNet and PromptIR models, we observe significant improvements across all tasks, highlighting the effectiveness of our approach in addressing the challenges of multi-task imbalance.Notably, the retrained AirNet model using Art achieves significant improvement of performance that are comparable to, or even surpass, results of the original PromptIR model. This finding directly corroborates the severe impact of multi-task imbalance on the current state of all-in-one image restoration. The original Air-Net model struggles to attain optimal results due to the inherent difficulties in balancing multiple tasks during training. However, by applying our Art approach, we effectively mitigate these issues and [Li et al. 2020] 24.71 0.8980 WeatherDiff128 [Özdenizci and Legenstein 2023] 29.72 0.9216 TransWeather [Valanarasu et al. 2022] 28.83 0.9000 TransWeather + Art (Ours) 29.81 0.9088DDMSNet [Zhang et al. 2021] 28.85 0.8772 All-in-One [Li et al. 2020] 28.33 0.8820 WeatherDiff128 [Özdenizci and Legenstein 2023] 29.58 0.8941 TransWeather [Valanarasu et al. 2022] 29.31 0.8879 TransWeather + Art (Ours) 30.61 0.9083RainDrop IDT [Xiao et al. 2023] 31.87 0.9313 All-in-One [Li et al. 2020] 31.12 0.9268 WeatherDiff128 [Özdenizci and Legenstein 2023] 29.66 0.9225 TransWeather [Valanarasu et al. 2022 unlock the true potential of the AirNet architecture, demonstrating its capacity to achieve state-of-the-art performance when properly optimized.Conversely, the PromptIR model, with its large model capacity (near 36M parameters) and adaptive prompt mechanism, exhibits strong generalization capabilities across multiple tasks. The adaptive prompt enables the model to dynamically adjust its behavior based on the specific characteristics of each task, leading to improved performance. However, despite its advanced architecture, the PromptIR model is not impervious to the inherent conflicts and imbalances that arise in multi-task learning when employing vanilla mixed training approaches. By retraining the PromptIR with our Art method, we observe substantial enhancements in restoration quality across all three degradation tasks as well. The application of the proposed task-specific rebalancing coefficients effectively equilibrates the dynamics optimization and mitigates inter-task conflicts. This enables the retrained PromptIR model to surpass its original counterpart and provides a new benchmark in all-in-one image restoration."dhaze-derain-desnow" setting. Table 8 showcases the performance of our Art method applied to the TransWeather. Compared to the original model, the retrained version exhibits average PSNR gains of 1.21 dB. These improvements highlight the enhanced ability of the retrained model to restore weather-degraded images. Figure 5 provides visual comparisons, corroborating the quantitative results.Moreover, Table 9 presents results on real-world deweathering tasks. Notably, the proposed Art method significantly improves the TransWeather model's performance, elevating it to a level comparable with state-of-the-art methods in terms of average performance across various real-world scenarios.The alignment between quantitative PSNR improvements and qualitative visual enhancements offers compelling evidence of our approach's efficacy in addressing various weather-related image degradations."rain-haze-snow-blur-dark" setting. Table 10 presents a comprehensive comparison of our proposed Art method applied to the AirNet and Transweather models for all-in-one image restoration across five distinct tasks. We evaluate the performance of the retrained models against their original counterparts and the results demonstrate the effectiveness of our Art method in enhancing the performance of both AirNet and Transweather models across all five tasks. It is worth noting that while the retrained AirNet model outperforms the retrained Transweather model on average, the performance gap between the two models is reduced compared to their original versions. This suggests that our Art method is effective in improving the performance of different architectures, making them more competitive in all-in-one image restoration tasks.Discussion. Our experiments demonstrate the effectiveness of the proposed Art optimizing strategy for all-in-one image restoration across various degradation scenarios. Art consistently enhances the performance of state-of-the-art models, including Air-Net, PromptIR, TransWeather, and MioIR, outperforming both vanilla mixed training and sequential training strategies. Notably, the retrained AirNet's ability to match or surpass the original PromptIR (Table 7 ) highlights a critical insight: current limitations in allin-one image restoration are not solely due to model capacity or architecture, but also stem from suboptimal training strategies. This finding emphasizes the importance of addressing the implicit effects of multiple tasks in the field. 
limitations: 
 While our proposed Art approach marks a significant advancement in all-in-one image restoration, it is not without limitations. Our findings reveal uneven performance improvements across different tasks, as evidenced in Tables 6 and 7 . Although Art substantially enhances the performance of models like Uformer and AirNet, the degree of improvement varies among tasks. This variability likely stems from the inherent complexities and diverse characteristics of the tasks involved, highlighting the need for further research. 
conclusion: 
 This paper introduces a novel Multi-Task Balanced Learning (Art) approach for all-in-one image restoration, addressing the challenges of imbalanced learning status, and potential conflicts among multiple restoration tasks. Art explicitly formulates the restoration process as a multi-task learning problem, introducing task-specific rebalancing coefficients, a local convergence rate, and a novel global correction weight to equilibrate dynamics optimization and mitigate inter-task conflicts. Extensive experiments demonstrate the versatility and generalizability of Art across various task settings and model architectures, highlighting its potential to become a standard optimization strategy in the field of all-in-one image restoration. The insights and techniques introduced in this work lay the foundation for future research in multi-task learning and have the potential to revolutionize the approach to all-in-one image restoration and related problems in computer vision. 
acknowledgement: 
 The research was supported by the National Natural Science Foundation of China (U23B2009). 
 
 Appendix:  


Decision: This paper proposes a new multi-task balance learning method to improve the performance of integrated image recovery, which provides a new perspective in the field. It constructs the recovery process as a multi-task learning problem, balancing dynamic optimization and mitigating inter-task conflicts by introducing task-specific rebalancing coefficients, local convergence rates, and global correction weights. Experimental results demonstrate that the proposed method performs well on various image restoration tasks. After rebuttal, this paper receives 4 positive reviews, so I recommend the acceptance of this paper.

Reviews: review: paper_summary: paper_summary: This paper introduces a new multi-task balance learning method named Art, which is used to improve the performance of integrated image recovery. It constructs the recovery process as a multi-task learning problem, balancing dynamic optimization and mitigating inter-task conflicts by introducing task-specific rebalancing coefficients, local convergence rates, and global correction weights. Experiments show that Art works well on image restoration tasks.
strength_weakness: strength_weakness: 1.The introduction of Art showcases a novel method for enhancing multi-task image restoration, offering a fresh perspective in the field.
2.Art's design as an easily integrable optimization strategy allows it to be a versatile tool for various restoration tasks. And the method significantly improves upon existing models, indicating a substantial advancement in image restoration capabilities.
3.The paper provides extensive experiments that validate the effectiveness of Art across different datasets and models, adding to its credibility.
limitations: limitations: 1.Although an effective optimization strategy is proposed in this paper, the theoretical analysis of complex relationships and different characteristics among tasks is not deep enough, and more theoretical analysis is needed to explain why it works.
2.There are some problems in the organization of the article. The Settings of the ablation experiment in Table 1 are not clear enough. Authors should make a good arrangement. Why is there an obvious low value (14.77) in the table, is the network not converging.
3.The datasets for all-in-one task may be unbalanced. The method might not perform optimally on all-in-one task with unbalanced datasets without further adjustments to handle class imbalances.
paper_summary: paper_summary: The paper introduces a novel approach called Art (Active-Reweighting Strategy) designed to enhance multi-task learning for image restoration. The proposed method addresses the common issue of inconsistent optimization rhythms among various image restoration tasks, such as denoising, deblurring, deraining, and dehazing. Art harmonizes the optimization process by introducing a local convergence rate and a global correction weight to balance the training dynamics across different tasks. This strategy is validated through extensive experiments, showing significant improvements in performance when integrated into existing models like AirNet and TransWeather.
strength_weakness: strength_weakness: This paper proposes an innovative approach to addressing multi-task optimization conflicts in image restoration. The active-reweighting strategy (Art) effectively balances the optimization process, leading to consistent performance improvements across different tasks. The method's plug-and-play nature allows it to be easily incorporated into existing models, demonstrating its versatility and robustness. Extensive experimental validation supports the efficacy of Art, showing substantial performance gains in terms of PSNR across various image restoration scenarios.
limitations: limitations: More tests on real-world datasets need to be conducted to demonstrate the practical applicability of the proposed method.
paper_summary: paper_summary: This paper proposes a active-reweighting strategy for all-in-one image restoration. It redefine the all-in-one image restoration tasks as a multi-task learning problem, and reweight the coefficients for each tasks with local convergence rate and global correction weight. The experiment results on a diverse range of all-in-one image restoration settings demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. The motivation to mitigate the potential conflicts among various restoration tasks is interesting.

2. The proposed active reweighting method with local convergence rate and global correction weight is straightforward and reasonable.

3. The experiment results on a diverse range of all-in-one image restoration settings and methods demonstrate the effectiveness of the proposed method.
limitations: limitations: The main problem about the proposed method is that the exploration about the conflicts between different restoration tasks is not very clear. The authors claim that the proposed method is designed to harmonize the optimization of multiple restoration tasks. However, the calculation of the local convergence rate and the global correction weight seems have no direct relation with the conflicts between different restoration tasks. The author needs to explain how the active reweighting harmonizes the optimization of different restoration tasks.
paper_summary: paper_summary: The motivation of this paper is both interesting and meaningful, exploring the relationships and potential conflicts between different tasks, proposing a plug-and-play Art strategy, and presenting an efficient loss function. Extensive experiments have demonstrated the effectiveness of the proposed modules.
strength_weakness: strength_weakness: 1. The motivation for exploring the relationship in multi-tasks is meaningful and novel.
2. The proposed effective task-specific loss can balance the individual task training process.
3. The experimental results are impressive.
limitations: limitations: 1. Some exciting all-in-one image restoration methods test single-task performance, I am curious about the performance of this article (although this is not closely related to the motivation of the paper).
2. The parameter, FLOPS, and inference time comparison are missed.