Title: RestoreAgent: Autonomous Image Restoration Agent via Multimodal Large Language Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Image Restoration

Low level vision

Agent

Multimodal Large Language Model
Abstract: Natural images captured by mobile devices often suffer from multiple types of degradation, such as noise, blur, and low light. Traditional image restoration methods require manual selection of specific tasks, algorithms, and execution sequences, which is time-consuming and may yield suboptimal results. All-in-one models, though capable of handling multiple tasks, typically support only a limited range and often produce overly smooth, low-fidelity outcomes due to their broad data distribution fitting. To address these challenges, we first define a new pipeline for restoring images with multiple degradations, and then introduce RestoreAgent, an intelligent image restoration system leveraging multimodal large language models. RestoreAgent autonomously assesses the type and extent of degradation in input images and performs restoration through (1) determining the appropriate restoration tasks, (2) optimizing the task sequence, (3) selecting the most suitable models, and (4) executing the restoration. Experimental results demonstrate the superior performance of RestoreAgent in handling complex degradation, surpassing human experts. Furthermore, the system’s modular design facilitates the fast integration of new tasks and models.

RestoreAgent: Autonomous Image Restoration Agent via Multimodal Large Language Models

Abstract

Natural images captured by mobile devices often suffer from multiple types of degradation, such as noise, blur, and low light. Traditional image restoration methods require manual selection of specific tasks, algorithms, and execution sequences, which is time-consuming and may yield suboptimal results. All-in-one models, though capable of handling multiple tasks, typically support only a limited range and often produce overly smooth, low-fidelity outcomes due to their broad data distribution fitting. To address these challenges, we first define a new pipeline for restoring images with multiple degradations, and then introduce RestoreAgent, an intelligent image restoration system leveraging multimodal large language models. RestoreAgent autonomously assesses the type and extent of degradation in input images and performs restoration through (1) determining the appropriate restoration tasks, (2) optimizing the task sequence, (3) selecting the most suitable models, and (4) executing the restoration. Experimental results demonstrate the superior performance of RestoreAgent in handling complex degradation, surpassing human experts. Furthermore, the system's modular design facilitates the fast integration of new tasks and models.

1 Introduction

Image restoration, a classical research area in computer vision, focuses on recovering high-quality images from degraded observations. Traditional methods are usually tailored to specific tasks like denoising [55, 61, 49, 29, 30, 12, 3], super-resolution [56, 34, 53, 4, 45, 47, 48], and deblurring [28, 22, 51, 32, 44, 17]. However, real-world images often suffer from multiple simultaneous degradations. For example, a low-quality image may exhibit noise, blur, and rain concurrently. There may exist complex interactions and dependencies among different degradation phenomena, and each degradation may require distinct handling methods. The combination and sequence of these methods are crucial for the final restoration outcome. Recent advancements in the field have been driven by leveraging expert knowledge and developing all-in-one models. To provide a thorough understanding of this field and clarify our motivation, we present a detailed analysis below.

1.1 All-in-One Models

All-in-one models [38, 31, 24, 40, 33, 14, 27, 37, 1, 25] seek to use a single framework to handle multiple degradations simultaneously. By training on multi-task datasets, these models learn to manage various restoration tasks. However, several limitations continue to impede the practicality of these models in complex real-world scenarios:

Restricted task scope. All-in-one models often struggle to process degradations outside of their training data. Even for the same type of degradation, as shown in Figure 2 a1 , these models may have difficulty effectively processing data if the degradation distribution varies between the training and testing sets. Given that existing models only cover a limited number of tasks, employing specialized single-task restoration models is often more flexible and effective.

1.00

1.50

2.00

2.50

3.00

Test Blur Kernel Width

a.

Trained on 3 di+erent noise levels

Tested across varying noise levels

b.

Trained on 3 blur kernel ranges

Tested across varying blur kernels

c.

Trained on di+erent numbers of tasks

T ested on various tasks

Mean Square Error

Trained on [2]

Trained on [0, 4]

Trained on [1.5, 2.5]

0.0

0.1

0.2

0.3

0.4

0.5

LPIPS

Trained on Level 15

Trained on Level 25

Trained on Level 50

Level 25

is Optimal

Level 50

is Optimal

Poor

Poor

Range

Generalization

Generalization

Fewer training task types,

better single-task performance

Good

Good

Performance

Performance

Poor

Poor

Performance

Performance

Generalization vs. performance

Trade-oA:

Improving one worsens the other

Good

Good

Generalization

Generalization

Level 15

is Optimal

10

30

40

20

T est Noise Level

JPEG

low light

noise

haze

blur

24

26

28

30

PSNR

Trained on 1 Task

Trained on 3 Tasks

Trained on 5 Tasks

Fewer training task types,

better single-task performance

Good

Good

Generalization

Each model has a specific

strength range

All-in-one models underperform

single-task models

Poor

Poor

Performance

Compromised performance. All-in-one models often face trade-offs between generalization and restoration accuracy, as shown in Figure 1. While these models offer improved generalization across a broader range of degradation levels, their performance at specific levels may be compromised. Additionally, because they must handle multiple tasks with largely disparate degradation patterns, the performance for individual tasks may fall short, resulting in overly smoothed outputs. As illustrated in Figure 2 a2 , single-task models typically outperform all-in-one models in most scenarios.

All-in-one models can, in fact, be integrated into an agent system comprising multiple models, thereby going beyond a single solution. Often, using task-specific models customized for particular degradations and then integrating them with an all-in-one model yields improved performance, as shown by the two examples in Figure 2 a3 . This hybrid approach maintains the adaptability of all-in-one models while leveraging the strengths of specialized models.

1.2 Task-Specific Models

An alternative approach to using all-in-one models, which struggle to effectively address various types of degradation, is to combine several specialized task-specific models, each focusing on a specific degradation type. This modular strategy allows for a more targeted and efficient handling of the different degradations present in the input images. Superior results can be achieved because these specialized models excel in their respective areas.

1.2.1 Fixed or Random Execution Order

Current methods [50, 24, 14] typically detect the types of degradation in an image and apply the appropriate restoration models in a predetermined order, or manually selected by experts, or chosen at random. Nevertheless, there is a significant drawback to this approach: the processing order has a major impact on the final performance. A predetermined order, even if established by human experts, is not ideal and might fail to successfully restore the image, as demonstrated in Figure 2 b . Two primary causes can be identified for this.

First, applying one restoration method can alter other degradation patterns , rendering the following restoration models ineffective. For example, in an image with haze and rain, if haze is performed first (Figure 2 b ), the dehazing model may address the blur but alter the rain distribution, thereby reducing the effectiveness of the deraining model.

Second, removing some degradations can be challenging if other degradations have not been addressed first . A common example is the enhancement of low-light images, which often requires denoising as a pre-processing step. Without prior denoising, the results of low-light enhancement are likely to be subpar. In Figure 2 b , we can observe that without prior denoising and deraining, the performance of the dehazing model is significantly compromised.

better single-task performanc

Generalization

Performance

Whynotall-in-one?

Step 1.

DeRaining

Step 2.

DeJPEG

Step 3.

DeHazing

Step 1.

DeHazing

Step 2.

DeRaining

Step 3.

DeJPEG

Input

noise

blur

JPEG

low light

moire

snow

…

…

Step 1.

Denosing

Step 2.

StableSR

Only

StableSR

Step 1.

DeRaining

Step 2.

PromptIR

Only

PromptIR

Input

Input

rain

Haze

Level 15

on small noise

on all noise

Level 15

on small noise

on all noise

RainyInput

Large Rain

on Large Rain

on Small Rain

DeJPEG

Model

Denoising

Model

InstructIR

All-in-One Model

MiOIR

All-in-One Model

Input

JPEG

Input

noise

Dehazing

Model

AirNet

All-in-One Model

Input

haze

a2.

Limited performance

: Specialized models outperform generalists

b.1

Wrong execution order causes wrong results.

c.

Inflexible models limit optimal performance

a1.

Not truly "all"

:  Models still fail on unseen degradation types

a3.

Single Task + All-in-One  >  All-in-One only

over smooth

over smooth

degradation

unseen

Execution Order

Incorrect

1. DeNosing 2. DeRaining

3. DeJPEG      4. DeHazing

Input

same degradation types, di>erent patterns

+noise+JPEG

rain+haze

3. DeJPEG       4. DeHazing

b.2

Same degradation types with diFerent patterns

require distinct execution orders

+

.

In light of these findings, accurate identification of degradation patterns or careful testing of various task execution sequences is necessary for high-quality restoration. However, the search space grows significantly with the number of tasks. For example, there are 24 possible execution orders for 4 degradation types. Moreover, the number of permutations increases drastically when multiple models are available for a given task, leading to a significant rise in computational complexity.

1.2.2 Fixed or Random Model for a Single Task

In some scenarios, the system may opt to use a single model for a specific task or randomly select a model from a pool of available options [50]. However, this approach has significant drawbacks. Image restoration is a rapidly evolving field with various models tailored for a specific task, each with unique capabilities and areas of expertise for managing specific scenarios. Using a fixed model

or randomly selecting from a pool of models to process complex degradations can lead to suboptimal results. As illustrated in Figure 2 c and Figure 1 a , different denoising models excel at different noise levels. Choosing the right model is crucial for achieving the best result.

Manually selecting the best model is impractical due to the numerous combinations of task execution orders and available models. For example, with 3 degradation types and 3 models per type, there are 162 possible combinations. Evaluating these permutations is time-consuming and labor-intensive. Consequently, we often rely on one or two experienced-based solutions, which may not achieve the desired restoration effect.

1.3 RestoreAgent

In response to the aforementioned challenges, we propose RestoreAgent, an autonomous and intelligent image restoration system based on a multimodal large language model (MLLM). The MLLM's exposure to vast and diverse data endows it with superior generalization capabilities and has showcased remarkable performance in visual understanding and logical reasoning [46, 35, 18, 39, 43, 6, 62]. Furthermore, its flexibility facilitates the quick addition of new tasks, the definition of desired output formats, and easier human interaction.

Our framework offers the following functionalities:

(1) Degradation Type Identification. RestoreAgent automatically identifies the types of degradation present in an input image and determines the corresponding restoration tasks required.

(2) Adaptive Restoration Sequence. RestoreAgent goes beyond the constraints of predefined, human-specified model execution orders by dynamically evaluating the individual properties of each input image to decide the best sequence for utilizing the restoration models, thereby enhancing the overall efficiency of the image restoration procedure.

(3) Optimal Model Selection. Based on the specific degradation patterns in the input image, RestoreAgent dynamically selects the most appropriate model from the available pool for each restoration task, ensuring optimal performance.

(4) Automated Execution. Once the restoration sequence and model selection are determined, RestoreAgent autonomously executes the entire restoration pipeline without the need for manual intervention.

To this end, we start by defining the multi-degradation task and constructing a training dataset. This dataset includes paired degraded images (with one or more degradation types) and their ground truth (only for evaluation), along with the optimal task execution sequence and best model choice based on user-preferred goals. We then fine-tune MLLM to enable RestoreAgent to autonomously make task decisions and determine the optimal processing sequence and models. Experiments show that RestoreAgent's decision-making capabilities significantly outperform existing methods and human experts, achieving superior performance in recovering multi-degradation images. Notably, our method can quickly adapt to unseen tasks and models.

2 Related Work

2.1 Single-Task Image Restoration

In the field of single-task image restoration, numerous methods have focused on addressing specific types of image degradation. In denoising, models like DnCNN [59] and RNAN [63] have demonstrated significant effectiveness, among others. In deblurring, algorithms like DeblurGAN [28] and MIMO-UNet [13] and others stand out. For reducing JPEG artifacts, methods such as DCSC [19] and FBCNN [23] are particularly well-suited. Additionally, there are specialized methods for restoration under adverse weather conditions, including dehazing [52, 41], deraining [11, 7], and desnowing [8, 9, 5]. Each task often requires a specialized approach, leading to highly optimized algorithms that achieve sota performance for their specific targets compared to universal approaches.

2.2 All-in-One Image Restoration

Recent research has explored the development of All-in-One models that attempt to handle multiple degradation types simultaneously within a single framework. This kind of methods are trained to recognize and correct various forms of degradation concurrently. AirNet [31] featuring the contrastivebased degraded encoder and degradation-guided all-in-one restoration network. ADMS [38] uses adaptive filters to efficiently restore images with unknown degradations. TAPE [36] embeds a taskagnostic prior into a transformer, utilizing a two-stage process of pre-training and fine-tuning to enhance image restoration. PromptIR [40] and PIP [33] both utilize uniquely designed prompts to guide their networks. MiOIR [27] employs sequential and prompt learning strategies, which guide the

network to incrementally learn individual IR tasks in a sequential manner. MPerceiver [1] employs a multimodal prompt learning approach, utilizing Stable Diffusion priors to achieve high-fidelity all-in-one image restoration.

2.3 Agent in Image Restoration

Another research direction focuses on more intelligent image restoration systems. One class of such methods employs a toolbox approach to address image degradation separately. RL-Restore [57] prepares a toolbox consisting of small-scale convolutional networks, each specialized in different tasks. The system then learns a policy to select appropriate tools from the toolbox to progressively restore the quality of a corrupted image. However, RL-Restore supports only three types of degradation: blur, noise, and JPEG compression, which constrains its application scenarios and prevents it from utilizing new state-of-the-art models. Clarity ChatGPT [50] combines the conversational intelligence of ChatGPT with multiple image restoration methods. It automatically detects types of image degradation and selects appropriate methods to restore images. Conversely, Clarity ChatGPT identifies the presence of degradation but lacks research and design on the execution order of tasks and the optimal model selection for specific degradations in the input image.

Another class involves all-in-one approaches with degradation-aware guidance. InstructIR [14] pioneers a novel approach by utilizing human-written instructions to guide the recovery from various types of degradation. AutoDIR [24] automatically detect and restore images with multiple unknown degradations. LLMRA [25] generates text descriptions and encodes them as context embeddings with degradation information, and integrates these context embeddings into the restoration network. DA-CLIP [37] presents a degradation-aware vision-language model that guides the model to learn high-fidelity image reconstruction. For these all-in-one restoration assistant methods, inherent limitations exist in the practical applications of all-in-one models.

How to overcome these limitations, fully leverage the wide array of state-of-the-art models for different tasks available on the market, and determine the optimal execution sequence of image restoration tasks and the most suitable model for specific degradation pattern remain unexplored. This gap presents a significant opportunity for future research in intelligent image restoration systems.

3 RestoreAgent

In this section, we introduce RestoreAgent, an advanced image restoration agent designed to find the optimal model and execution sequence from a model pool to process images containing multiple degradations. RestoreAgent is built upon a state-of-the-art multimodal large language model, which possesses remarkable reasoning, generalization, and cross-modal understanding capabilities. By leveraging the model's ability to draw insights from vast amounts of multimodal data, establish connections between visual and textual information, and apply that knowledge to new contexts, RestoreAgent can effectively analyze complex image degradation scenarios, infer the most suitable restoration techniques, and generate optimal pipelines that combine the strengths of various specialized models. As a result, RestoreAgent consistently produces high-quality results.

In Section 3.1, we first define the problem of identifying the most effective combination and order of models from a given pool to restore images affected by various types of degradation. Next, in Section 3.2.2, we describe the process of constructing the training data for the RestoreAgent. The training data consists of paired samples, each containing a degraded image and its corresponding optimal restoration pipeline. Finally, we detail the training process of RestoreAgent, which involves fine-tuning the Llava-Llama3-8b model using the constructed training data in Section 3.2. By learning from these examples, RestoreAgent acquires the ability to analyze degraded images and generate optimal restoration pipelines based on the available model pool.

3.1 Problem Definition

We consider a comprehensive set of degradation types, denoted as D = { d 1 , d 2 , . . . , d n } , where each d i represents a specific type of image degradation such as noise, JPEG artifacts, blur, rain streaks, fog, and low light conditions. For each degradation type d i , we tailor a model library M d i , comprising models { M 1 d i , M 2 d i , . . . } . Each model M j d i is specifically trained to mitigate the effects of degradation d i . The problem is formally defined as follows:

Input: A degraded image I subjected to various degradation types D . A model library {M d 1 , M d 2 , . . . , M d n } tailored for processing D . The user-provided scoring function S for evaluating the image restoration process.

Objective: Identify the optimal model execution sequence σ = ( M b 1 a 1 , M b 2 a 2 , . . . , M b m a m ) that maximizes the restoration quality S of the degraded image I , where a i denotes the degradation type and b i represents the corresponding model. It is formulated as:

where S ( D , M ) represents the set of all possible sequences of degradation and model pairs.

By tackling this problem, we strive to identify the optimal combination of restoration sequence and model selections, ultimately enhancing the quality of images affected by multiple degradations in real-world settings, and thus providing a more effective and efficient solution for complex image restoration tasks.

3.2 RestoreAgent: An Advanced Image Restoration System

3.2.1 RestoreAgent Pipeline

We introduce an advanced image restoration agent, dubbed RestoreAgent, implemented using the state-of-the-art multimodal model Llava-Llama3-8b [46]. LoRA [21] is utilized to fine-tune both the vision and language modules. As shown in Figure 3, given a degraded input image, RestoreAgent can provide the best decisions, including which image restoration tasks need to be performed, the order of their execution, and which model is most suitable for each task. The model's input consists of a degraded image and the prompt such as User: How to enhance the quality of this image? [Execution history: ...] . In response, RestoreAgent generates an output sequence representing the optimal restoration pipeline, comprising a series of tasks, each associated with a specific model best suited to address particular degradation patterns. In our implementation, the output template is defined as: Agent:1.<task name><model name>. 2.<task name><model name>. 3. ... , ensuring interpretability and actionability.

RestoreAgent also supports an iterative step-wise decision-making process, reevaluating the state of the image after each restoration step. During this reassessment, the execution history is provided, offering valuable context for decision-making. This allows for real-time strategy adjustments based on cumulative effects and past actions. The system also features a rollback capability, enabling it to revert to a previous state if undesirable results are detected. This combination of iterative evaluation with historical context and rollback allows for finer control over the restoration process, facilitating mid-course corrections.

3.2.2 Data Construction

To fully leverage the potential of multimodal large models, we construct a substantial dataset of paired training samples. The process begins with applying various types of degradation to an image. Subsequently, we determine the optimal restoration pipeline using model tools for processing. For each image undergoing multiple degradations, a comprehensive search is conducted to identify the best restoration pipeline, as shown in Figure 3. This involves generating all possible permutations of task execution sequences and model combinations, applying each pipeline to the degraded image, and assessing the quality of the restored outputs using a scoring function S ( I, σ ) . By comparing the scores of all permutations, the pipeline with the highest score is selected as the optimal processing strategy σ for the given image. Users can choose from various image quality assessment methods as the scoring function, customizing the evaluation process to their specific needs. Additional details regarding dataset construction are provided in the supplementary materials.

4 Experiment

4.1 Experimental Settings

Scoring function. To construct a comprehensive evaluation system, we integrate multiple diverse metrics. Specifically, we first standardized each individual metric separately and then summed the standardized results. This process can be described as follows. Let X i represent the i -th metric. We standardize each metric by calculating its z-score: Z i = X i -µ i σ i where µ i is the mean and σ i is the standard deviation of the i -th metric. After standardizing all metrics, we aggregate the standardized scores to form the comprehensive evaluation score S : S = ∑ n i =1 Z i where n is the total number of metrics. This method ensures that each metric contributes equally to the final evaluation, regardless of its original scale. Follow [26, 20], evaluation metrics primarily include PSNR, SSIM, LPIPS [60], and DISTS [16]. These metrics are widely recognized for their ability to comprehensively reflect the outcomes of image restoration. We also provided the results of models trained on individual metrics.

Dehazing

Step 1.

DeJPEG

Step 2.

Step 3.

Deraining

Step 4.

Step 1.

DeJPEG

Step 2.

Deraining

Step 3.

Dehazing

Step 4.

Step 1.

DeRaining

Step 2.

DeJPEG

Step 3.

Motion Deblur

Step 4.

Step 1.

Deblur

Step 2.

DeJPEG

Step 3.

Deraining

Step 4.

Input

Input

Step 1.

DeJPEG

Step 2.

Deraining

Step 3.

Step 1.

Deraining

Step 2.

DeHazing

Step 3.

DeRaining

Step 1.

Step 2.

Motion Deblur

Step 3.

Step 1.

Deblur

Step 2.

Deraining

Step 3.

Input

Input

Zoom-in

Zoom-in

Zoom-in

Zoom-in

Dataset and model tool setings. To explore the feasibility of automating image restoration using multimodal models, we selected six distinct image restoration tasks: denoising, motion deblurring, deJPEG, deraining, dehazing, and low-light image enhancement. Each image in the dataset can exhibit up to four types of degradation. To validate the decision-making ability of the model when multiple models are available for a single task, we constructed three specialized models for the denoising task, and three models have different noise levels: low, medium and high noise. Similarly, for the deJPEG task, we developed models specifically designed to handle severe and mild JPEG compression artifacts. For the remaining tasks, each has a corresponding dedicated model. For the testing datasets, we assemble 200 images, mirroring the degradation types found in the training datasets, to facilitate evaluation. Detailed information is in the supplementary material.

4.2 Comparisons with Other Strategies

Compared methods. In this study, we conducted a comparative analysis of RestoreAgent against several alternative approaches:

Random selection of both the task order and the models, assuming accurate determination of task types.

Random task order, but models predicted by RestoreAgent.

Random model selection, but task orders predicted by RestoreAgent.

For all images, using the human expert's predefined order and models, assuming accurate task type determination.

Human expert personally crafting a solution for each image, determining the task sequence and models for each task. This method represents the most common scenario in real-world applications, where a human decides how to restore an image on a case-by-case basis.

The human expert in this study has more than five years of research experience in low-level vision. Before crafting solutions, the expert familiarized themselves with each task degradation and the corresponding model's actual performance to ensure they could provide the best human-level solution. Results. Table 1 reports the average metric results of our RestoreAgent and other decision-making approaches on seven different degradation combination datasets. As shown in Table 1, using a random order and model selection ranked lowest, achieving only a 34.7% performance rating among all possible strategies. By setting predefined sequences and models for image processing by human experts, traditional methods rank in the top 22.1% of all possible strategies. This demonstrates that experience-based predefined rules often used in practical applications are more effective than completely random strategies. Human experts making specific decisions for each test image can further improve upon predefined rules, increasing the ranking from 22.1% to 19.5%. This proves that using the same predefined rules to process all images is not optimal, while individualized decision-making for specific images can better enhance the effects. Then, the superior performance of our RestoreAgent (12.9%) over expert-based customization (19.5%) shows that automated and

Input

(noise+JPEG)

Real-ESRGAN

StableSR

MiOIR

DA-CLIP

RestoreAgent

GT

Input

(rain+haze+noise+JPEG)

MiOIR

DA-CLIP

RestoreAgent

GT

AirNet

Input

(noise+haze)

InstructIR

PromptIR

MiOIR

AutoDIR

DA-CLIP

RestoreAgent

GT

data-driven decision-making in our method clearly outperforms traditional and experience-based human expert judgments. This is because human experts from their own experience can not make precise judgments about the advantageous scenarios of all models and the order of task execution, especially when numerous tasks and models are involved.

4.3 Comparisons with All-in-One Methods

To demonstrate the limitations of all-in-one methods in handling multi-degraded images, we compared our approach with various types of all-in-one models. To ensure a fair comparison, tests were only conducted on degradation types and datasets that these all-in-one models were trained to support. Moreover, we repeatedly run the all-in-one model as many times as the number of degradation types of the test images to fully leverage its capabilities, thus ensuring a fair comparison. The results are shown in Figure 5 and Table 2. Our RestoreAgent achieved a significant leading advantage across all tested degradation combinations. For the degradation types commonly encountered in traditional image super-resolution, such as noise and JPEG compression artifacts, our approach significantly outperformed established methods like Real-ESRGAN and the sota SR method, StableSR. For a broader range of degradation types, our method retained a considerable advantage. Among these all-in-one approaches, InstructIR and AutoDIR face two major issues: manually predetermined or

randomly decided execution order, and using single model to address all types of degradations. These limitations often result in incomplete restoration, as depicted in Figure 5. These results underscore the limitations of all-in-one models, validating our initial hypothesis.

4.4 Adapting to Different Optimization Objectives

As discussed in the method, our proposed method can adapt to various optimization objectives, enabling the decision-making results tailored to specific target criteria. To verify it, we present the results of models trained with different individual metrics as the optimization objective in Table 3. The results indicate that when a model is trained with a single metric, the performance of the corresponding metric can be significantly improved compared to the balanced model. This showcases the adaptability and effectiveness of our method in catering to specific optimization goals.

4.5 Extending for New Tasks and Models

The proposed RestoreAgent demonstrates remarkable adaptability and extensibility, allowing for swift fine-tuning to accommodate new task types and incorporate additional models. To validate this capability, we introduced a new task, desnowing, along with its corresponding model. Building upon the RestoreAgent previously trained on six tasks, we performed rapid fine-tuning by integrating the desnowing task. Within thirty minutes, our model achieved exceptional performance on the new task type. As shown in Table 5, our approach quickly surpassed human expert-level proficiency on the new task and model. This validation underscores the practical value of our method, allowing efficient integration of additional tasks with minimal resource expenditure.

4.6 Step-wise Re-planning and Rollback

As mentioned in Section 3.2, RestoreAgent supports iterative decision-making with historical context awareness. It dynamically adjusts strategies during image restoration, reassessing image state after each step and rolling back if needed. As demonstrated in Table 6, we conducted experiments on a complex dataset incorporating four distinct types of image degradation: Motion Blur, Rain, Noise, and JPEG compression. Results show that while the single prediction approach performs well, iterative step-wise replanning further enhances restoration outcomes, allowing for precise control and mid-course corrections. The initial decision's performance is already strong, step-wise replanning thus offers incremental yet valuable improvements to an already effective process.

5 Conclusion

Our research identifies key factors in processing multi-degraded images, including task execution order, model selection, and the limitations of the all-in-one approach. Based on these insights, we present RestoreAgent, an agent model that makes intelligent processing decisions based on image degradation and user objectives. Experiments show that our pipeline outperforms the all-in-one method and surpasses human experts in decision-making performance.

Acknowledgments. This work is supported by the Guangzhou-HKUST(GZ) Joint Funding Program (No. 2023A03J0671), the Guangzhou Municipal Science and Technology Project (Grant No. 2024A04J4230), Guangdong Provincial Key Lab of Integrated Communication, Sensing and Computation for Ubiquitous Internet of Things(No.2023B1212010007), and the National Natural Science Foundation of China (Project No. 61902275).

References

Yuang Ai, Huaibo Huang, Xiaoqiang Zhou, Jiexiang Wang, and Ran He. Multimodal prompt perceiver: Empower adaptiveness, generalizability and fidelity for all-in-one image restoration. arXiv preprint arXiv:2312.02918 , 2023.

Yuanhao Cai, Hao Bian, Jing Lin, Haoqian Wang, Radu Timofte, and Yulun Zhang. Retinexformer: One-stage retinex-based transformer for low-light image enhancement. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 12504-12513, 2023.

Haoyu Chen, Jinjin Gu, Yihao Liu, Salma Abdel Magid, Chao Dong, Qiong Wang, Hanspeter Pfister, and Lei Zhu. Masked image training for generalizable deep image denoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1692-1703, 2023.

Haoyu Chen, Wenbo Li, Jinjin Gu, Jingjing Ren, Haoze Sun, Xueyi Zou, Youliang Yan, Zhensong Zhang, and Lei Zhu. Low-res leads the way: Improving generalization for super-resolution by self-supervised learning. CVPR , 2024.

Haoyu Chen, Jingjing Ren, Jinjin Gu, Hongtao Wu, Xuequan Lu, Haoming Cai, and Lei Zhu. Snow removal in video: A new dataset and a novel method. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 13211-13222, October 2023.

Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478 , 2023.

Sixiang Chen, Tian Ye, Jinbin Bai, Erkang Chen, Jun Shi, and Lei Zhu. Sparse sampling transformer with uncertainty-driven ranking for unified removal of raindrops and rain streaks. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 13106-13117, 2023.

Sixiang Chen, Tian Ye, Yun Liu, Jinbin Bai, Haoyu Chen, Yunlong Lin, Jun Shi, and Erkang Chen. Cplformer: Cross-scale prototype learning transformer for image snow removal. In Proceedings of the 31st ACM International Conference on Multimedia , MM '23, page 4228-4239, New York, NY, USA, 2023. Association for Computing Machinery.

Sixiang Chen, Tian Ye, Yun Liu, and Erkang Chen. Snowformer: Context interaction transformer with scale-awareness for single image desnowing. arXiv preprint arXiv:2208.09703 , 2022.

Sixiang Chen, Tian Ye, Yun Liu, and Erkang Chen. Snowformer: Context interaction transformer with scale-awareness for single image desnowing. arXiv preprint arXiv:2208.09703 , 2022.

Xiang Chen, Hao Li, Mingqiang Li, and Jinshan Pan. Learning a sparse transformer network for effective image deraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5896-5905, 2023.

Jun Cheng, Tao Liu, and Shan Tan. Score priors guided deep variational inference for unsupervised realworld single image denoising. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 12937-12948, 2023.

Sung-Jin Cho, Seo-Won Ji, Jun-Pyo Hong, Seung-Won Jung, and Sung-Jea Ko. Rethinking coarse-tofine approach in single image deblurring. In Proceedings of the IEEE/CVF international conference on computer vision , pages 4641-4650, 2021.

Marcos V Conde, Gregor Geigle, and Radu Timofte. High-quality image restoration following human instructions. arXiv preprint arXiv:2401.16468 , 2024.

XTuner Contributors. Xtuner: A toolkit for efficiently fine-tuning llm. https://github.com/InternLM/ xtuner , 2023.

Keyan Ding, Kede Ma, Shiqi Wang, and Eero P. Simoncelli. Image quality assessment: Unifying structure and texture similarity. CoRR , abs/2004.07728, 2020.

Jiangxin Dong, Jinshan Pan, Zhongbao Yang, and Jinhui Tang. Multi-scale residual low-pass filter network for image deblurring. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 12345-12354, 2023.

Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378 , 2023.

Xueyang Fu, Zheng-Jun Zha, Feng Wu, Xinghao Ding, and John Paisley. Jpeg artifacts reduction via deep convolutional sparse coding. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , October 2019.

Jinjin Gu, Haoming Cai, Haoyu Chen, Xiaoxing Ye, Jimmy Ren, and Chao Dong. Image quality assessment for perceptual image restoration: A new dataset, benchmark and metric. arXiv preprint arXiv:2011.15002 , 2020.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 , 2021.

Seo-Won Ji, Jeongmin Lee, Seung-Wook Kim, Jun-Pyo Hong, Seung-Jin Baek, Seung-Won Jung, and Sung-Jea Ko. Xydeblur: Divide and conquer for single image deblurring. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 17421-17430, June 2022.

Jiaxi Jiang, Kai Zhang, and Radu Timofte. Towards flexible blind jpeg artifacts removal. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4997-5006, 2021.

Yitong Jiang, Zhaoyang Zhang, Tianfan Xue, and Jinwei Gu. Autodir: Automatic all-in-one image restoration with latent diffusion. arXiv preprint arXiv:2310.10123 , 2023.

Xiaoyu Jin, Yuan Shi, Bin Xia, and Wenming Yang. Llmra: Multi-modal large language model based restoration assistant. arXiv preprint arXiv:2401.11401 , 2024.

Gu Jinjin, Cai Haoming, Chen Haoyu, Ye Xiaoxing, Jimmy S Ren, and Dong Chao. Pipal: a large-scale image quality assessment dataset for perceptual image restoration. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16 , pages 633-651. Springer, 2020.

Xiangtao Kong, Chao Dong, and Lei Zhang. Towards effective multiple-in-one image restoration: A sequential and prompt learning strategy. arXiv preprint arXiv:2401.03379 , 2024.

Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych, Dmytro Mishkin, and Jiˇ rí Matas. Deblurgan: Blind motion deblurring using conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 8183-8192, 2018.

Samuli Laine, Tero Karras, Jaakko Lehtinen, and Timo Aila. High-quality self-supervised deep image denoising. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila. Noise2noise: Learning image restoration without clean data. arXiv preprint arXiv:1803.04189 , 2018.

Boyun Li, Xiao Liu, Peng Hu, Zhongqin Wu, Jiancheng Lv, and Xi Peng. All-in-one image restoration for unknown corruption. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17452-17462, 2022.

Ji Li, Weixi Wang, Yuesong Nan, and Hui Ji. Self-supervised blind motion deblurring with deep expectation maximization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 13986-13996, June 2023.

Zilong Li, Yiming Lei, Chenglong Ma, Junping Zhang, and Hongming Shan. Prompt-in-prompt learning for universal image restoration. arXiv preprint arXiv:2312.05038 , 2023.

Xinqi Lin, Jingwen He, Ziyan Chen, Zhaoyang Lyu, Ben Fei, Bo Dai, Wanli Ouyang, Yu Qiao, and Chao Dong. Diffbir: Towards blind image restoration with generative diffusion prior. arXiv preprint arXiv:2308.15070 , 2023.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023.

Lin Liu, Lingxi Xie, Xiaopeng Zhang, Shanxin Yuan, Xiangyu Chen, Wengang Zhou, Houqiang Li, and Qi Tian. Tape: Task-agnostic prior embedding for image restoration. In European Conference on Computer Vision , pages 447-464. Springer, 2022.

Ziwei Luo, Fredrik K Gustafsson, Zheng Zhao, Jens Sjölund, and Thomas B Schön. Controlling visionlanguage models for universal image restoration. arXiv preprint arXiv:2310.01018 , 2023.

Dongwon Park, Byung Hyun Lee, and Se Young Chun. All-in-one image restoration for unknown degradations using adaptive discriminative filters for specific degradations. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 5815-5824. IEEE, 2023.

Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824 , 2023.

Vaishnav Potlapalli, Syed Waqas Zamir, Salman Khan, and Fahad Shahbaz Khan. Promptir: Prompting for all-in-one blind image restoration. arXiv preprint arXiv:2306.13090 , 2023.

Xu Qin, Zhilin Wang, Yuanchao Bai, Xiaodong Xie, and Huizhu Jia. Ffa-net: Feature fusion attention network for single image dehazing. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 11908-11915, 2020.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Erix Xing, Ming-Hsuan Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. arXiv preprint arXiv:2311.03356 , 2023.

Mengwei Ren, Mauricio Delbracio, Hossein Talebi, Guido Gerig, and Peyman Milanfar. Multiscale structure guided diffusion for image deblurring. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10721-10733, 2023.

Haoze Sun, Wenbo Li, Jianzhuang Liu, Haoyu Chen, Renjing Pei, Xueyi Zou, Youliang Yan, and Yujiu Yang. Coser: Bridging image and language for cognitive super-resolution. arXiv preprint arXiv:2311.16512 , 2023.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Jianyi Wang, Zongsheng Yue, Shangchen Zhou, Kelvin CK Chan, and Chen Change Loy. Exploiting diffusion prior for real-world image super-resolution. arXiv preprint arXiv:2305.07015 , 2023.

Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan. Real-esrgan: Training real-world blind superresolution with pure synthetic data. In Proceedings of the IEEE/CVF international conference on computer vision , pages 1905-1914, 2021.

Zejin Wang, Jiazheng Liu, Guoqing Li, and Hua Han. Blind2unblind: Self-supervised image denoising with visible blind spots. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2027-2036, June 2022.

Yanyan Wei, Zhao Zhang, Jiahuan Ren, Xiaogang Xu, Richang Hong, Yi Yang, Shuicheng Yan, and Meng Wang. Clarity chatgpt: An interactive and adaptive processing system for image restoration and enhancement. arXiv preprint arXiv:2311.11695 , 2023.

Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G. Dimakis, and Peyman Milanfar. Deblurring via stochastic refinement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 16293-16303, June 2022.

Haiyan Wu, Yanyun Qu, Shaohui Lin, Jian Zhou, Ruizhi Qiao, Zhizhong Zhang, Yuan Xie, and Lizhuang Ma. Contrastive learning for compact single image dehazing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10551-10560, 2021.

Rongyuan Wu, Tao Yang, Lingchen Sun, Zhengqiang Zhang, Shuai Li, and Lei Zhang. Seesr: Towards semantics-aware real-world image super-resolution. arXiv preprint arXiv:2311.16518 , 2023.

Ruiqi Wu, Zhengpeng Duan, Chunle Guo, Zhi Chai, and Chongyi Li. Ridcp: Revitalizing real image dehazing via high-quality codebook priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023.

Xiaohe Wu, Ming Liu, Yue Cao, Dongwei Ren, and Wangmeng Zuo. Unpaired learning of deep image denoising. In European conference on computer vision , pages 352-368. Springer, 2020.

Fanghua Yu, Jinjin Gu, Zheyuan Li, Jinfan Hu, Xiangtao Kong, Xintao Wang, Jingwen He, Yu Qiao, and Chao Dong. Scaling up to excellence: Practicing model scaling for photo-realistic image restoration in the wild. arXiv preprint arXiv:2401.13627 , 2024.

Ke Yu, Chao Dong, Liang Lin, and Chen Change Loy. Crafting a toolchain for image restoration by deep reinforcement learning. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2443-2452, 2018.

Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 5728-5739, 2022.

Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE transactions on image processing , 26(7):31423155, 2017.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 586-595, 2018.

Yi Zhang, Dasong Li, Ka Lung Law, Xiaogang Wang, Hongwei Qin, and Hongsheng Li. Idr: Selfsupervised image denoising via iterative data refinement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2098-2107, June 2022.

Yihua Zhang, Pingzhi Li, Junyuan Hong, Jiaxiang Li, Yimeng Zhang, Wenqing Zheng, Pin-Yu Chen, Jason D Lee, Wotao Yin, Mingyi Hong, et al. Revisiting zeroth-order optimization for memory-efficient llm fine-tuning: A benchmark. arXiv preprint arXiv:2402.11592 , 2024.

Yulun Zhang, Kunpeng Li, Kai Li, Bineng Zhong, and Yun Fu. Residual non-local attention networks for image restoration. arXiv preprint arXiv:1903.10082 , 2019.

A Appendix / supplemental material

A.1 Model Tool Setings

As shown in Table 7, for the tasks of denoising and deJPEG, as well as deraining, we employ Restormer [58] as our model. For dehazing, we utilize RIDCP [54], while for motion deblurring, we use DeblurGANv2 [28]. For desnowing, we implement Snowformer [10]. For low-light enhancement, we use Retinexformer [2]. It is noteworthy that the models we are using are not the latest state-of-theart models, indicating that there is significant room for improvement in our models.

A crucial consideration in image restoration is the limited generalization capability of many current models, which often fail to maintain performance when faced with subtle variations in image degradation. This necessitates the selection of more robust models. For example, in our approach to denoising, we enhance model generalization by incorporating not only Gaussian noise but also random blur and other noise types during training. This strategy enables the model to address more complex degradation scenarios effectively.

A.2 Dataset Construction Details

Input

Answer

How to enhance the quality of this image?

Execution history: None.

1. Denoising, low noise level,

2. Dehazing,

3. DeJPEG, high quality factor

How to enhance the quality of this image?

Execution history: 1. Denoising, low noise level.

1. Denoising, low noise level,

2. Dehazing,

3. DeJPEG, high quality factor

How to enhance the quality of this image?

Execution history: 1. Dehazing.

Rollback.

How to enhance the quality of this image?

Execution history: None.

Rollback from Dehazing.

1. Denoising, low noise level,

2. Dehazing,

3. DeJPEG, high quality factor

How to enhance the quality of this image?

Execution history:

1. Denoising, low noise level,

2. Dehazing, 3. DeJPEG, high quality factor

Stop.

Input

1. Denoising

1. Dehazing

Input

1. Denoising

2. Dehazing

3. DeJPEG

1

(Primary)

2

3

4

5

Initiate full enhancement

sequences for degraded

images.

Dynamically adjust

strategies based on

intermediate results.

Identify and correct

suboptimal steps through

rollback mechanisms.

Avoid repetition of

ineffective procedures

post-rollback.

Recognize when image

quality has reached its

optimal state.

Function

Figure 6 illustrates 5 scenarios incorporated into our dataset, designed to enhance the versatility and robustness of the RestoreAgent model:

(1) Once we obtain a degraded image along with its corresponding optimal decision results, we can construct the primary part of our dataset. This part consists of degraded images in their original, unprocessed state. For these inputs, the RestoreAgent receives a prompt: "How to enhance the quality of this image? Execution history: None." This scenario trains the model to formulate comprehensive enhancement strategies from scratch, encompassing multiple restoration steps. This part of the data exceeds 23k pairs.

(2) To foster dynamic decision-making capabilities, we introduce a second category of training instances. Here, the input comprises partially processed images (e.g., after denoising) along with their execution history. This approach enables the RestoreAgent to adapt its predictions based on intermediate results, promoting a more flexible and context-aware enhancement process.

(3) The third scenario addresses situations where the model identifies suboptimal outcomes from a particular enhancement step. In such cases, the RestoreAgent is trained to output "Rollback," indicating the need to revert to a previous state and recalibrate its strategy. This feature is crucial for maintaining high-quality outputs and avoiding the propagation of errors through the enhancement pipeline. We select from erroneous paths (the decisions with the worst metric results) to construct this portion of the paired data, as the worst paths require a rollback.

(4) Following a rollback event, our fourth data category provides the model with information about the specific step that triggered the rollback. This guidance is essential in preventing the model from repeating ineffective procedures, thus streamlining the enhancement process and improving efficiency.

(5) The final scenario in our training regime represents fully processed images that require no further enhancement. In these instances, the RestoreAgent is trained to recognize optimal image quality and output "Stop", effectively terminating the enhancement sequence.

By incorporating these diverse scenarios, we aim to develop a highly adaptive and efficient image restoration system capable of addressing a wide array of real-world image degradation challenges. For computational efficiency, unless specifically mentioned otherwise, our default experiments are based on a single planning for the initial image rather than using iterative step-wise replanning.

A.3 Testset details

The specific details of our test set are presented in Table 8, which demonstrates our construction of various combinations of degradation types. Each image in the set contains a minimum of one and a maximum of four types of degradation, with the entire set comprising 200 images.

A.4 Training Setups

In this study, we incorporate the CLIP pre-trained Vision Transformer (ViT-L/14) [42] as the image encoder to convert input images into visual tokens. For the language model, we utilize the Llama37B [46]. Despite their capabilities, pre-trained LLMs fail to provide accurate responses without dataset-specific fine-tuning. To address this, we adopt LoRA [21], a fine-tuning technique that efficiently modifies a limited number of parameters within the model. Following [21], we apply LoRA to adjust the projection layers in all self-attention modules of both the vision encoder and the LLM, thereby generating our RestoreAgent. We employ the Xtuner framework [15] to facilitate the training process. For our experimental setup, we configure the LoRA rank to 16. The RestoreAgent undergoes training across ten epochs on 4 NVIDIA RTX A100 GPUs, with a batch size of 32. We employ the Adam optimizer and a learning rate of 0.00002. The total duration of the training process approximates ten hours.

A.5 Analysis

Figure 4 and 8 illustrate RestoreAgent's decision-making process and the importance of model selection, respectively. Figure 6 further demonstrates why human decision-making often yields suboptimal results in image restoration tasks. Figure 8a exemplifies the nuanced challenges in degradation assessment. Despite identical backgrounds and degradation types, subtle variations in degradation features lead to divergent optimal restoration sequences. For instance, the sequence

same degradation types, di0erent patterns

Input

Input

1. DeNosing 2. DeRaining

3. DeJPEG      4. DeHazing

Input

3. DeJPEG       4. DeHazing

+noise+JPEG

rain+haze

a.

Same degradation types with di2erent patterns

require distinct execution orders

b.

Finding the sole e2ective strategy amidst a myriad of choices

is exceedingly challenging for human

3. DeHazing

3. DeHazing

3. DeRaining

3. DeHazing

1. DeRaining

2. DeHazing

3. DeRaining

3. DeRaining

3. DeHazing

Step 1.

10.54

0.3756

PSNR

LPIPS

Low Light

Enhancement

Step 2.

Large

noise level

26.53

0.1999

PSNR

LPIPS

Step 1.

Small

noise level

Step 2.

10.63

0.4002

PSNR

LPIPS

16.96

0.3754

PSNR

LPIPS

Step 1.

11.78

0.3129

PSNR

LPIPS

Step 2.

Medium

noise level

21.41

0.2552

PSNR

LPIPS

Step 1.

Large

noise level

Step 2.

11.78

0.3636

PSNR

LPIPS

20.17

0.2775

PSNR

LPIPS

most suitable model

most suitable model

over smooth

Low Light

Enhancement

DeHazing

DeHazing

Input

Input

Zoom-in

Zoom-in

ill-suited model ill-suited model

"DeNoising → DeRaining → DeJPEG → DeHazing" proves effective for the upper row images but fails for the lower row. Conversely, the sequence "DeRaining → DeNoising → DeJPEG → DeHazing" yields optimal results for the lower row but is suboptimal for the upper row. This dichotomy underscores the difficulty human experts face in discerning minute degradation differences, thereby compromising effective decision-making.

The complexity of optimal restoration sequencing is further accentuated in Figure 8b. Here, we demonstrate scenarios where only one specific sequence among numerous permutations yields satisfactory results. This observation highlights the formidable challenge posed to human decisionmakers in identifying the singular effective restoration pathway amidst a multitude of possibilities.

These findings collectively emphasize the superiority of automated, data-driven approaches in navigating the intricate landscape of image restoration. The RestoreAgent's ability to discern and adapt to subtle degradation variations surpasses human capabilities, particularly in scenarios where the optimal restoration sequence is non-intuitive and highly specific to individual image characteristics.

A.6 Discussion

Comparison with assistants with all-in-one models. Assistants that employ unified models, such as LLMRA [25] and AutoDIR [24], attempt to handle diverse tasks, degradation patterns, and intensities using a single model. As discussed in Section 1.1, these all-in-one models face significant challenges, including restricted task scope and compromised performance, which greatly limit their effectiveness in real-world applications. Conversely, our method leverages various model experts to address specific situations, the upper bound of our pipeline is determined by the latest SOTA models, allowing us to maximally leverage the latest advancements in the field without being constrained by

the limitations of an all-in-one model. Furthermore, as detailed in Section 4.4, our RestoreAgent exhibits high efficiency in incorporating new tasks and models, showcasing greater flexibility.

Comparison with assistants with tool use. Image restoration assistants that utilize tool libraries, such as Clarity ChatGPT [50] and RL-Restore [57]. Clarity ChatGPT merely identifies the degradation in images, follows a rigid execution strategy, lacking the ability to make dynamic decisions on task execution order and select the best model. As discussed in Section 1.2.1 and 1.2.2, an inappropriate task execution sequence and model selection can leading to lower performance in subsequent operations. RL-Restore, on the other hand, uses reinforcement learning for sequence decision-making and model selection. However, its task definition is overly simplistic, limited to three degradation types (noise, blur, and JPEG) with a narrow degradation range. Also, training reinforcement learningbased methods is more challenging and may result in lower precision, making it difficult to achieve high performance in complex and varied scenarios. Conversely, the integration of a comprehensive task definition with advanced multimodal models allows our method to effectively manage various degradation types and intensities. This adaptability enhances its efficacy, positioning our approach as a promising solution for image restoration tasks.

A.7 Alation Study

Training data amount. To investigate the effect of training data volume on our method, we evaluated the performance of the RestoreAgent model trained on datasets consisting of 7,000, 14,000, and 23,000 data pairs; see Table 4 The results demonstrate that even with the smallest dataset of 7k pairs, our RestoreAgent achieves superior performance over both random and human expert benchmarks. More notably, the training data volume increasing from 7k to 14k incurs a substantial performance improvement with the ranking percentage decreasing from 16.2% to 13.6%. With 23k data pairs, the performance further improves, achieving a ranking percentage of 12.9%. This indicates that using more training data boosts our RestoreAgent model. These findings emphasize the robustness of our approach, demonstrating that while larger datasets do enhance performance, our model already provides significant benefits even with relatively smaller datasets.

A.8 Limitation and Future Work

The primary limitation of our study is the confined scope of models and tasks examined. While our research offers valuable insights into RestoreAgent's performance across several degradation scenarios, it does not encompass the full spectrum of restoration models or image degradation tasks currently available.

Another limitation pertains to the limited generalization capability of current image restoration models. These models often exhibit a notable decrease in performance or fail to respond adequately when faced with even minor variations in image degradation patterns. This limitation greatly narrows our selection of model tools, requiring us to choose more robust and generalizable model tools. The challenge underscores a critical need in the field of image restoration: future models must go beyond simply overfitting training data. Rather, they should exhibit better generalization and increased efficiency in handling real-world degradation cases.

Our future work will focus on significantly expanding the range of image restoration models incorporated into our multimodal large language model. This expansion aims to enhance RestoreAgent's capabilities across a broader scope of restoration tasks and degradation types. By integrating a more diverse set of state-of-the-art models, we seek to create a more comprehensive and versatile restoration framework.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Yes, the main claims made in the abstract and introduction accurately reflect the paper's contributions

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of the work in the supplemental material section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Yes, for each theoretical result, we provide the full set of assumptions and a complete proof.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Yes, our method is easy to be reproduced, and we provide all information.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We use public code and data.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Yes, we give all the details.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: See Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We conform, in every respect, to the NeurIPS Code of Ethics as outlined at the provided https://neurips.cc/public/EthicsGuidelines.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The creators or original owners of assets used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [Yes]

Justification: See supplemental materia.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [Yes]

Justification: There are no potential risks.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nFor real-world images corrupted by multiple simultaneous degradations, this paper first analyzes the limitations of using all-in-one restoration models and various task-specific models. The authors then introduce RestoreAgent, which automatically identifies the types of degradation in a degraded image, determines the sequence of restoration tasks, and selects suitable models from the model pool. RestoreAgent presents an automated restoration pipeline that requires only an input image and a general human instruction, without any prior knowledge of the involved degradation tasks or manually predefined task sequences.\n\nSTRENGTHS:\n1. The paper comprehensively analyzes the challenges and limitations of employing all-in-one models and multiple task-specific expert models with fixed or random task sequences, as well as fixed or random models for each task.\n2. The authors evaluate various configurations of RestoreAgent using diverse objective image quality metrics (PSNR, SSIM, LPIPS, DISTS, and their combinations), all of which outperform the human expert model on the corresponding metric. \n3. RestoreAgent exhibits the scalability by extending to new tasks and models with minimal computational resource. \n4. The presentation, including writing, analysis, and visualization, is clear and easy to follow.\n\nWEAKNESSES:\n1. Incomplete descriptions about data construction. \n\n- Authors randomly select up to four types of degradation from a degradation set (noise, blur, JPEG, rain, haze, and low-light) to construct paired training data. According to data synthesis strategies in [1,2], JPEG compression is typically performed after noise and blur, and in the final order. Is the degradation order of JPEG compression in this paper the same? If not, the authors should discuss the reasonableness of random sampling.\n\n- What are the components of 23k paired data? One degraded image for each high-quality image or many degraded versions for each high-quality image? \n\n- What is the configuration in ablation studies about training data amount? Simultaneously scaling up low & high-quality images or synthesizing more low-quality images for each high-quality image? If it’s the former, will increasing the number of degraded images while keeping the number of high-quality images unchanged improve performance?\n\n2. Inference time for input images with diverse resolution.\n- The authors are suggested to report the running time for input images of various resolutions. This should include the total time, the running time for the RestoreAgent, and the running time for the subsequent restoration models. The reviewer is curious whether the agent's response time exceeds that of the restoration models when processing high-resolution images, such as those with 4K resolution.\n\n3. Scalability for new tasks and models. \n- Section 4.5 demonstrates that the proposed RestoreAgent can extend to new tasks and models in just half an hour, surpassing human expert-level performance on the new task. However, it is unclear whether adaptation to the new task results in performance degradation on prior tasks, similar to the catastrophic forgetting problem in continual learning. The authors are encouraged to report the performance of the fine-tuned model on the previous tasks to address this concern.\n\n[1] Wang X, Xie L, Dong C, et al. Real-esrgan: Training real-world blind super-resolution with pure synthetic data[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2021: 1905-1914.\n\n[2] Zhang K, Liang J, Van Gool L, et al. Designing a practical degradation model for deep blind image super-resolution[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021: 4791-4800.\n\nQUESTIONS:\nAddressing concerns in the Weaknesses with thorough explanations and additional experiments would significantly enhance my confidence in this work. A satisfactory response to these points may lead to a reconsideration of the current evaluation.\n\nLIMITATIONS:\nThe manuscript includes the checklist guidelines.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new pipeline to address multiple degradation, like noise, blur and low light. Besides, a RestoreAgent with multimodal large language models is introduced to assess the type and extent of degradations in the input images and perform dynamic restorations.\n\nSTRENGTHS:\n1. The paper is well-written and well-organised. \n2. The whole pipeline seems to be novel and reasonable. \n3. The method achieves SOTA performance on several benchmarks and different degradation tasks.\n\nWEAKNESSES:\nThe overall motivation of this paper is commendable, but I have a few concerns:\n\n1. The author mentions that RestoreAgent can autonomously assess the type and extent of degradation in input images and perform restoration. This strategy is interesting. However, I am wondering how the order of different enhancement techniques is defined. For example, if the input has noise and rain streaks, how is the order of dehazing and denoising techniques determined? Will this affect performance?\n\n2. In contrast to other image enhancement techniques, the proposed RestoreAgent should first find a suitable restoration task and then select the most appropriate model to enhance the quality of the input. Therefore, I am concerned whether this process will increase the inference time. The authors should provide some computational analysis.\n\n3. The enhancement capabilities of this work rely heavily on existing enhancement frameworks. If existing frameworks cannot work well in some cases, such as extreme noise effects, I guess the proposed RestoreAgent may also fail. Is this true? If so, I suggest the authors mention this in the limitations section.\n\n4. The explanation of \"ranking\" and \"balanced\" in Table 1 is still unclear. The authors should clarify the definitions of these terms.\n\n5. It would be better to show more visual comparisons of the RestoreAgent.\n\nQUESTIONS:\nPlease see weaknesses.\n\nLIMITATIONS:\nLimitations are not mentioned in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an image restoration pipeline designed to handle various degradation types and levels by leveraging MLLM’s capabilities to select the appropriate model and determine the execution order. It begins with an analysis of why execution order and utilizing multiple models for different degradation levels are crucial for restoring complexly degraded images. The paper then constructs an instruction dataset and fine-tunes the MLLM. Experimental results demonstrate the effectiveness of the proposed restoration pipeline.\n\nSTRENGTHS:\n1.\nThis work presents a compelling analysis of complex image restoration. This insight is valuable given that degraded images in real-world scenarios often involve multiple types of degradation.\n2.\nThis approach leverages the strengths of different models for handling specific noise levels, thereby eliminating the trade-off between generalization and performance.\n3.\nThis paper formally defines the problem of handling multiple degradations and model selection in image restoration.\n4.\nExtensive experiments demonstrate superiority of such pipeline in processing degraded images with multiple degradations.\n\nWEAKNESSES:\n1.\nIn the introduction, it would be helpful to explain how the Multi-Level Learning Model (MLLM) excels at understanding different types and levels of image degradation. This will show why MLLM is well-suited for handling complex combinations of image degradation. Providing this clarity will make the benefits of using MLLM for image restoration more evident.\n2.\nWhen incorporating a new type of degradation, the cost extends beyond merely training the MLLM. Please also discuss the process of constructing training data for the newly added degradation and how it integrates with previously trained data.\n3.\nIn lines 211-212, please clarify what the mean and standard deviation are calculated over. The subscript \"i\" is already used for degradation type and it might be clearer to use another character.\n\nQUESTIONS:\n1.\nWhat if the degradation of the input image falls outside the predefined degradation scope? This could present a generalization issue, as the model might not perform well on unseen types or levels of degradation not covered in the predefined scope. Please discuss it.\n2.\nIn Table 2, it would be clearer to highlight the best method for each evaluation criterion. Additionally, please specify which methods the ranking improvement is compared against for better context.\n\nLIMITATIONS:\nThe authors have addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces RestoreAgent, an innovative image restoration system that leverages multimodal large language models to autonomously handle images with multiple types of degradation. The system addresses limitations of existing all-in-one models and fixed task sequences by dynamically adapting to each image's specific degradations. RestoreAgent can identify degradation types, determine appropriate restoration tasks, optimize the execution sequence, select the most suitable models, and execute the restoration process autonomously. The authors present a method for constructing training data and demonstrate that RestoreAgent outperforms existing methods and human experts in handling complex image degradations.\n\nSTRENGTHS:\n1. This paper represents a innovation and a good contribution in image restoration and potentially opens up a new research direction for this area.\n2. The motivation is strong. The authors effectively demonstrate the importance of task execution order and model selection in multi-task scenarios. The designed system adeptly addresses these issues.\n3. Experimental results indicate that RestoreAgent's decision-making capabilities in handling complex degradations surpass those of human experts. This kind of pipeline also surpass all-in-one models.\n4. The paper is generally well written and clear to understand.\n\nWEAKNESSES:\n1. The paper constructs a training dataset for training the multimodal large language model and a testing dataset as a benchmark for evaluating performance across multiple tasks. More details and explanations regarding the construction methods of these datasets would be beneficial.\n2. Table 1 presents performance rankings using both ordinal and percentage forms. The definitions and explanations for these ranking forms are somewhat lacking, which might require readers to spend extra time understanding them. Clearer explanations would facilitate better comprehension.\n3. The proposed Autonomous Restoration Agent represents a novel paradigm that is likely to encounter numerous new challenges. Beyond the issues already mentioned in the paper, the authors could consider discussing additional limitations and future research directions for this paradigm. This would help future researchers better follow and improve upon this work.\n\nQUESTIONS:\n1. The current method appears to predict all execution steps at once for a given input image. In Figure 3, each image has a dashed line pointing to the input. Does this imply that after each execution, the result can be fed back as input? (Based on my understanding, this system supports this) The paper seems to lack analysis and experiments related to this aspect. Could the authors provide more details on this part?\n2. The authors have proposed a testing dataset to evaluate multi-task processing capabilities. Will this dataset be made publicly available to facilitate further research by other researchers?\n\nLIMITATIONS:\nSee Weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}