RestoreAgent: Autonomous Image Restoration Agent via Multimodal Large Language Models
Authors: Haoyu Chen
Wenbo Li
Jinjin Gu
Jingjing Ren
Sixiang Chen
Tian Ye
Renjing Pei
Kaiwen Zhou
Fenglong Song
Lei Zhu
Venue: NeurIPS 2024 poster (2024)
Keywords: Image Restoration
Low level vision
Agent
Multimodal Large Language Model
Primary area: machine_vision

Abstract: Natural images captured by mobile devices often suffer from multiple types of degradation, such as noise, blur, and low light. Traditional image restoration methods require manual selection of specific tasks, algorithms, and execution sequences, which is time-consuming and may yield suboptimal results. All-in-one models, though capable of handling multiple tasks, typically support only a limited range and often produce overly smooth, low-fidelity outcomes due to their broad data distribution fitting. To address these challenges, we first define a new pipeline for restoring images with multiple degradations, and then introduce RestoreAgent, an intelligent image restoration system leveraging multimodal large language models. RestoreAgent autonomously assesses the type and extent of degradation in input images and performs restoration through (1) determining the appropriate restoration tasks, (2) optimizing the task sequence, (3) selecting the most suitable models, and (4) executing the restoration. Experimental results demonstrate the superior performance of RestoreAgent in handling complex degradation, surpassing human experts. Furthermore, the system’s modular design facilitates the fast integration of new tasks and models.

RestoreAgent: Autonomous Image Restoration Agent via Multimodal Large Language Models
Haoyu Chen', Wenbo Lid, Tian Ye Renjing Pei
Jinjin Gu Jingjing Ren', Sixiang Chen',
Kaiwen Zhou². Fenglong Song", Lei 74u 4.4
1 The Hong Kong University of Science and Technology (Guangzhou) Huawei Noah's 's Ark Lab 'hhe University of Sydney Project page: https://haoyuchen. wot.///on..ooeeeetooeeeeeee
'The Hong Kong University Science and Technology
Abstract
Natural images captured by mobile devices often suffer from multiple types of degradation, such as noise, blur, and low light. Traditional image restoration methods require manual selection of specific tasks, algorithms, and execution sequences, which is time-consuming and may yield suboptimal results. All-in-one models, though capable of handling multiple tasks, typically support only a limited range and often produce overly smooth, low-fidelity outcomes due their broad data distribution fitting. address these challenges, we first define a new pipeline for restoring images with multiple degradations, and then introduce RestoreAgent, an intelligent image restoration system leveraging multimodal large language models. RestoreAgent autonomously assesses the type and extent of degradation in input images and performs restoration through (1) determining the appropriate restoration tasks, (2) optimizing the task sequence, (3) selecting the most suitable models, and (4) executing the restoration. Experimental results demonstrate the superior performance of Agent in handling complex degradation, surpassing human experts. Furthermore, the system's 's modular design facilitates the fast integration of new tasks and models.
Introduction
Image restoration, classical research area in computer vision, focuses on recovering high-quality images from degraded observations. Traditional methods are usually tailored to specific tasks like denoising [53
and deblur- However, real real-world images often suffer from multiple simultaneous degradations. For example, low-quality image may exhibit noise, blur, and rain concurrently. There may exist complex interactions and dependencies among different degradation phenomena, and each degradation may require distinct handling methods. The combination and sequence of these methods are crucial for the final restoration outcome. Recent advancements in the field have been driven by leveraging expert knowledge and developing all-in-one models. To provide a thorough understanding of this field and clarify our motivation, we present detailed analysis below.
61
49
29
30
super-resolution [56] 34
ring [28 22
51
32
44
17
1.1 All-in-One Models
seek to use single framework to handle multiple degradations simultaneously. By training on multi-task datasets, these models learn to manage various restoration tasks. However, several limitations continue to impede the practicality of these models in complex real- world scenarios:
All-in-one models [38] 31 40 33 14 27 37 25
Restricted task scope. All-in-one models often struggle to process degradations outside of their training data. Even for the same type of degradation, as shown in Vigure2[a1. these models may have difficulty effectively processing data the degradation distribution varies between the training and testing sets. Given that existing models only cover limited number of tasks, employing specialized single-task restoration models is often more flexible and effective.
LLei Zhu (leizhu @ust.hk) is corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Trained Level 15
30
0.5
Trained on Level 50
0.4
Mean Square Error
Good Pefforment celtriiztionn
28
LPIPS
0.3
PSNR
0.2
0.1
Genenazattoo
Wide Tichinn Range
0.0
3.00
JPEG
low light
nose
haze
blur
Test Noise Levil
Test Blur Kernel Width
Level 15 Optmmall
Level 25 Optimal
Level 5D is Optimal
Trained on |2]
Trained on [0,
Trained
Tasks
Trained on Task
Trained 11.5,
Trained Tasks
Trained on different noise levels Tested Each model has specific strength range
b. Trained on biur kamal ranges
Trained different numbers tasks on
noise levels
Tested across varying, blur kemels
Generalization vs, performance Trade-off: improving one worsens the
models underperform single-task models
Figure 1: Limitations all-in-one models. (a) Models trained on different noise levels excel in
specific areas, so choosing models on demand leads better results. (b) Models trained on wider range of blur degradations offer improved generalization but compromised performance, showing trade-off. (c) Multi-task models underperform on individual tasks compared single-task models, illustrating that all-in-one models trade performance for generalization.
Compromised performance. All-in-one models often face trade-offs between generalization and restoration accuracy, as shown in Figure [ While these models offer improved generalization across broader range of degradation levels, their performance at specific levels may be compromised. Additionally, because they must handle multiple tasks with largely disparate degradation patterns, performance for individual tasks may fall short, resulting in overly smoothed outputs. As illustrated Figure a2, single-task models typically outperform all-in-one models in most scenarios.
All-in-one models can, in fact, be integrated into an agent system comprising multiple models, thereby going beyond a single solution. Often, using task-specific models customized for particular degradations and then integrating them with an all-in-one model yields improved performance, as shown by the two examples in Figure 2 a3. This hybrid approach maintains the adaptability of all-in-one models while leveraging the strengths of specialized models.
1.2 Task-Specific Models
An alternative approach to using all-in-one models, which struggle to effectively address various types of degradation, to combine several specialized task sssc-specfiic models, each focusing on specific degradation type. This modular strategy allows for more targeted and efficient handling of different degradations present the input images. Superior results can be achieved because these specialized models excel in their respective areas.
1.2.1 Fixed or Random Execution Order
Current methods [50 24 14 typically detect the types of degradation in an image and apply the appropriate restoration models in a predetermined order, or manually selected by experts, or chosen random. Nevertheless, there is significant drawback to this approach: the processing order has major impact on the final performance. A predetermined order, even if established by human experts, not ideal and might fail to successfully restore the image, as demonstrated in Figure 2 b. Two primary causes can be identified for this.
First, applying one restoration method can alter other degradation patterns, rendering the following restoration models ineffective. For example, in an image with haze and rain, if haze is performed first (Figure b). the dehazing model may address the blur but alter the rain distribution, thereby reducing the effectiveness the deraining model.
Second, removing some degradations can be challenging if other degradations have not been addressed first. common example the enhancement low-light images, which often requires denoising as pre-processing step. Without prior denoising, the results of low-light enhancement are likely to be subpar. In Figure b, we can observe that without prior denoising and deraining, the performance of the dehazing model is significantly compromised.

---

Why not all-in-one? a1. Not truly "all": Models still fail on unseen degradation types
unzeon Goddaoo
Limited performance: outperform
Input JPEG
DeJPEG Model
MOIR Model
Input noise
Danoising Model
InstructIR All in One Model
Input haze
Dehazing Model
AirNet All One Model
a3. Single Task All-in-One All-in-One only
Input
Step Denoeing
Step StableSR
Only StableSR
Input
Step DeRaining
Only PromptiR
PromptlR
Why not use fixed random task execution order?
b.1 Wrong execution order causes wrong results.
b.2 Same degradation types with cilffernn patterns require distinct execution orders
yppdt dif
Input
Step DePairing
Step DeJPEG
Step 3. Del lazing
Incerret Exccition Oroir
Step DeHazing
Step 2. DeRaining
Stap 3. DeJPEG
Input main-haze
B. DeIPEG
1. DeNosing DeRaining Deliazing
1. Dellaining DeNosing 3.DDPE
Dolrazing
Why not use single fixed model for a task?
Infiexible optimal performance
Nokey
Noley
Model
Rainylnout Large
Model Large
Model
noise
Figure 2: Limitation illustration of all-in-one models, fixed task execution order, and fixed model. Images with pink background indicate negative examples
In light of these findings, accurate identification of degradation patterns or careful testing of various task execution sequences is necessary for high-quality restoration. However, the search space grows significantly with the number of tasks. For example, there are 24 possible execution orders for degradation types. Moreover, the number of permutations increases drastically when multiple models are available for given task, leading significant rise computational complexity.
1.2.2 Fixed or Random Model for a Single Task
In some scenarios, the system may opt to use a single model for specific task or randomly select model from pool of available options [50]. However, this approach has significant drawbacks. Image restoration rapidly evolving field with various models tailored for a specific task, each with unique capabilities and areas of expertise for managing specific scenarios. Using fixed model

---

or randomly selecting from pool of models to process complex degradations can lead to suboptimal results. As illustrated in Figure 2 e and Figure[Ja. different denoising models excel at different noise levels. Choosing the right model is crucial for achieving the best result.
Manually selecting the best model is impractical due to the numerous combinations of task execution orders and available models. For example, with 3 degradation types and 3 models per type, there are 162 possible combinations. Evaluating these permutations is time-consuming and labor-intensive. Consequently, we often rely on one or two experienced-basee solutions, which may not achieve the desired restoration effect.
1.3 RestoreAgent
In response to the aforementioned challenges, we propose RestoreAgent, an autonomous and intelli- gent image restoration system based on multimodal large language model (MLLM). The MLLM's exposure to vast and diverse data endows it with superior generalization capabilities and has show- cased remarkable performance in visual understanding and logical reasoning 46 35 39
Furthermore, its flexibility facilitates the quick addition of new tasks, the definition of desired output formats, and easier human interaction.
Our framework offers the following functionalities:
(1) Degradation Type Identification. RestoreAgent automatically identifies the types of degradation present in an input image and determines the corresponding restoration tasks required.
(2) Adaptive Restoration Sequence. RestoreAgent goes beyond the constraints of predefined, ouman-specified model execution orders by dynamically evaluating the individual properties of each input image to decide the best sequence for utilizing the restoration models, thereby enhancing the overall efficiency of the image restoration procedure.
(3) Optimal Model Selection. Based on the specific degradation patterns in the input image, RestoreAgent dynamically selects the most appropriate model from the available pool for each restoration task, ensuring optimal performance.
(4) Automated Execution. Once the restoration sequence and model selection are determined, RestoreAgent autonomously executes the entire restoration pipeline without the need for manual intervention.
To this end, we start by defining the multi-degradation task and constructing a training dataset. This dataset includes paired degraded images (with one or more degradation types) and their ground truth (only for evaluation), along with the optimal task execution sequence and best model choice based on user- user-preferred goals. We then fine-tune MILM enable RestoreAgent to autonomously make task decisions and determine the optimal processing sequence and models. Experiments show that RestoreAgent
decision-making capabilities significantly outperform existing methods and human experts, achieving superior performance in recovering multi-degradation images. Notably, our method can quickly adapt to unseen tasks and models.
2
Related Work
2.1
Single-Task Image Restoration
In the field of single-task image restoration, numerous methods have focused addressing specific types of image degradation. In denoising, models like DrCNN [59] and RNAN 63 have demon- strated significant effectiveness, among others. In deblurring. algorithms like DeblurGAN (28 and MIMO-UNet 133 and others stand out. For reducing JPEG artifacts, methods such as DCSC [9 and FBCNN 23 are particularly well-suited. Additionally, there are specialized methods for restoration
under ad verse weather conditions, including dehazing [52] 41). deraining [II, 77. and desnow-
ing 88 9 5. Each task often requires specialized approach, leading highly optimized algorithms that achieve sota performance for their specific targets compared to universal approaches.
2.2 All-in-One Image Restoration
Recent research has explored the development of All-in-One models that attempt to handle multiple degradation types simultaneously within a single framework. This kind of methods are trained to
recognize and correct various forms of degradation concurrently. AirNet [I] featuring the contrastive- based degraded encoder and degradation -uideed all-in-one restoration network. ADMS (38] uses adaptive filters to efficiently restore images with unknown degradations. TAPE (36 embeds a task- agnostic prior into transformer, utilizing a two-stage process of pre-training and fine-tuning to
enhance image restoration. PromptIR 40 and PIP (33 both utilize uniquely designed prompts to guide their networks. MiOIR [27] employs sequential and prompt learning strategies, which guide the

---

network to incrementally learn individual IR tasks in sequential manner. MPereeiver [ employs multimodal prompt learning approach, utilizing Stable Diffusion priors to achieve high-fidelity all-in-one image restoration.
2.3 Agent in Image Restoration
Another research direction focuses on more intelligent image restoration systems. One class of such methods employs toolbox approach to address image degradation separately. RL-Restore [57] prepares toolbox consisting of small-scale convolutional networks, each specialized different tasks. The system then learns policy to select appropriate tools from the toolbox progressively restore the quality of a corrupted image. However, RL-Restore supports only three types of degradation: blur, noise, and JPEG compression, which constrains its application scenarios and prevents it from utilizing new state-of-the-ae models. Clarity ChatGPT [50] combines the conversational intelligence of ChatGPT with multiple image restoration methods. It automatically detects types of image degradation and selects appropriate methods to restore images. Conversely, Clarity ChatGPT identifies the presence of degradation but lacks research and design on the execution order of tasks and the optimal model selection for specific degradations in the input image.
Another class involves all-in-one approaches with degradation-aware guidance. InstructIR [14 pioneers a novel approach by utilizing human-written instructions to guide the recovery from various types of degradation. AutoDIR (24) automatically detect and restore images with multiple unknown degradations. LLMRA 25 generates text descriptions and encodes them as context embeddings with degradation information, and integrates these context embeddings into the restoration network. DA-CLIP (37 presents degradation-awar vision-language model that guides the model to learn high-fidelity image reconstruction. For these all-in-one restoration assistant methods, inherent limitations exist in the practical applications of all-in-one models.
How to overcome these limitations, fully leverage the wide array of state-of-the-art models for different tasks available on the market, and determine the optimal execution sequence of image restoration tasks and the most suitable model for specific degradation pattern remain unexplored. This gap presents a significant opportunity for future research in intelligent image restoration systems.
3 RestoreAgent
In this section, we introduce RestoreAgent, an advanced image restoration agent designed to find the optimal model and execution sequence from a model pool to process images containing multiple degradations. RestoreAgent is built upon state-of-the the-art multimodal large language model, which possesses remarkable reasoning, generalization, and cross-modal understanding capabilities. By leveraging the model's ability to draw insights from vast amounts of multimodal data, establish connections between visual and textual information, and apply that knowledge to new contexts, RestoreAgent can effectively analyze complex image degradation scenarios, infer the most suit- able restoration techniques, and generate optimal pipelines that combine the strengths various specialized models. As result, RestoreAgent consistently produces high-quality results.
In Section 3.1 we first define the problem of identifying the most effective combination and order of models from given pool to restore images affected by various types of degradation. Next, in Section 3.2.2 we describe the process of constructing the training data for the RestoreAgent. The training data consists of paired samples, each containing degraded image and its corresponding optimal restoration pipeline. Finally, we detail the training process of RestoreAgent, which involves fine-tuning the Llava-Llama3-88 model using the constructed training data in Section (3.2, By learning from these examples, RestoreAgent acquires the ability to analyze degraded images and generate optimal restoration pipelines based on the available model pool.
3.1 Problem Definition
We consider a comprehensive set of degradation types, denoted as D = {dl, d2...., .......d d, where each d, represents specific type of image degradation such noise, JPEG artifacts, blur, rain streaks, fog, and low light conditions. For each degradation type d, we tailor a model library Ma,, comprising models { [M,, , ...... Each model M is specifically trained to mitigate the effects of degradation d, The problem is formally defined as follows:
Input: A degraded image I subjected to various degradation types D. A model library :MMd:Miiy Ma, tailored for processing D. The user-provided scoring function S for evalu- ating the image restoration process.
5

---

Data Construction
Task
Models Library
Determination of the Optimal Execution Sequence
Data Structuro
Delewing np
DeuPèo
soar unvi dee
2 Rest PONA
Detlnneg
Dediar
Dedkur
Dedkar
Dedkar
@ bedikur
W LPPS
boHazzing
and encuiiiing roge
and
201
(2 Rest Preference
Detenne
Low Light
beficingg
Derlazing
DeMlazing
Derkozing
Coller
Eshonce
Erilance
Enhonice
Erioance
Devieing my se Hig Uiges
RestoreAgent Pipeline
23 trahing
Input
Perception
Task Planning
Selection
DeMtesing
the cualts
Muttimodal Large Language Model
RestoreAgent
Step1
Shop
Exscuton
littry
Tilbedi
Dnd
DeNlciin
DeRaining
PIG Hag
DAJPEG
Stop-wise lee-planning
Figure 3: Illustration of the data construction workflow and RestoreAgent pipeline.
Identify the optimal model execution sequence (M& M& 1.... Mam) ) that maxi- mizes the restoration quality S of the degraded image I, where a; denotes the degradation type and represents the corresponding model. is formulated as:
Objective:
arg max GEG(D.M)
S(I,o),
where G(D,M) represents the set of all possible sequences of degradation and model pairs.
By tackling this problem, we strive to identify the optimal combination of restoration sequence and model selections, ultimately enhancing the quality of images affected by multiple degradations in real-world settings, and thus providing a more effective and efficient solution for complex image restoration tasks.
3.2 RestoreAgent: An Advanced Image Restoration System
3.2.1 RestoreAgent Pipeline
We introduce an advanced image restoration agent, dubbed RestoreAgent, implemented using the state-of-the-an multimodal model Llava-Llama3-8h 46) LoRA [2I] is utilized to fine-tune both the vision and language modules. As shown in Figure given a degraded input image, RestoreAgent can provide the best decisions, including which image restoration tasks need to be performed, the order of their execution, and which model is most suitable for each task. The model's input consists of degraded image and the prompt such as User: How to enhance the quality of this image? [Execution history: ...J. In response, RestoreAgent generates an output sequence representing the optimal restoration pipeline, comprising a series of tasks, each associated with specific model best suited to address particular degradation patterns. In our implementation, the output template is defined as: Agent: 1. <task hame><model name> 2. <task ensuring interpretability and actionability.
name>. 3.
RestoreAgent also supports an iterative step-wise decision-making process, reevaluating the state of the image after each restoration step. During this reassessment, the execution history is provided, offering valuable context for decision-makinn on cumulative effects and past actions. The system also features rollback capability, enabling it to revert to previous state if undesirable results are detected. This combination of iterative evaluation with historical context and rollback allows for finer control over the restoration process, facilitating mid-course corrections.
This allows for real-time strategy adjustments based

---

Table I: Comparison of RestoreAgent with other decision- maaiong strategies for multi-degraded image restoration. The "balanced" column represents the sum of the four normalized metrics, which is our score function to train the model. The "ranking" column indicates the ranking of the given decision among all possible decisions, with the total number of decisions for each test set provided.
The final group presents the Average Result Across All Datasets providing an overall performance.
Noise + JPEG
Low Light + Noise
PSNK
LPIPS ; DISTS
balanced
ranking /17
PSNR
5SIM
LPPPPS
; DISTS
balanced
ranking /10
Kandom Order Model
24.52
0.7273
0,2589
0.2212
1.47
6.7
15.57
0.6541
11.4351
0.2588
1:58
3.9
Rondom Ofler Prodict Modil
25.24
0.7765
0.2327
0.1960
3.07
15.62
0.6857
0.3651
0.2281
3.03
3.0
Random Model a Predict Order
24.90
0.7568
0.2597
0.2132
2.03
6.0
17.57
0.7044
1.3685
0.2324
3.75
2.3
delined Oder and Model
25.29
0.7528
0.2037
2.47
0.7098
0.3385
0.2260
3.90
2.1
Hatian Expert
25.06
0.7588
0.2551
0.2121
2.25
5.5
18.05
0.7239
113278
0.2220
4.29
1.9
RestoreAgent
25.32
0.7506
0.1958
11.6
17.80
0.7226
0.3259
0.2138
10.2
Motion Blur Noise + JPEG
Rain + Noise + JPEG
PSNR
55IM
LPPIS
DISTS | balanced
ranking /64
PSNR
55IM
LPIPS
DISTS
balanced
ranking /64
Random Order& Model
24.81
0,7816
0.2381
0.1747
2:32
19.5
25.64
0,7970
0.2412
0.3020
2.90
16.1
Random Oder Predict Model
24.73
0.7787
0.2261
0.1684
2.69
16.1
25.67
0,8008
0,2368
0.1956
3.11
15.0
Random Modal + Predict Order
24.95
0,7912
0,2363
0.1647
3.18
13.6
26.14
0,8074
0.2314
0.1996
3.49
13.3
Pre-detinid Oder and Model
24.84
0.7895
0.2305
0.1662
2.97
15.0
25,80
0.7981
0.2360
0.2041
2.83
16.7
Human Expert
25.20
0.795
0,2006
0.1646
3.82
9.0
25.99
0 0.8063
0.2258
0. 0.1992
3.58
12.6
Rgent
25.16
0.7939
0.2042
0.1546
4.35
14.4
26.38
0.8136
0.2300
0.1891
4.67
6.4 16.2
Haze + Noise + JPEG
Haze + Rain + Noise + JPEG
PSNK
SSIM
LPIPS
DISTS ) balanced
ranking /64
PSNR
SSIM
LPIPS ; DISTS ) balanced
ranking /287
Rundom Order Model
18.98
0,7156
0.3267
0.2212
1.52
23.4
15.13
0.6300
0.4464
0.2800
1.28
102.5
Random Oderr Predict Model
19.00
0.7235
0.3155
0.2081
2.03
211.4
17.45
0.6897
0,3692
0.24000
2.36
72.8
Rundom Model Predict Order
19.67
0,7683
0,2778
0.2010
2.95
15.9
19.79
0,7833
0.2815
0.1991
5,66
16.9
Pre Pre-delined Oder and Model
19:47
0.7603
0.2641
0.1912
3.51
12.4
19.29
0.7785
0,2815
0.1974
5.502
26.1
Human Expert
19.50
0,7753
0.2700
0.1982
3.36
12.7
19.39
0,7802
0.2928
0.2043
5.500
21.3
RestoreAgent
19.55
0.7794
0.25663
0.1863
3.93
8.4
14.
19.72
0,7816
0.2741
0.1903
5.36
9.7 111.6
Molion Blur
Rain Noise JPBG
Average Resull Across All Dalasets
PSNK
SSIM
LPIPS ; DISTS ) balanced
ranking /287
PSNR
SSIM
LPIPS ; DISTS ) balanced
ranking (1)
Randem Order Model
21.96
0,6672
0.3306
0.2239
2.57
85.3
0,7139
0.3246
0.2241
1,922
34.7
Random Odeer Prodict Modell
22.11
0.6667
0.303%
0.2122
3.66
55.8
21.74
0.7585
0.2848
0.2144
2.89
26.1
Random Model Predict Urder
22.74
0,6696
0.1979
5.39
24.7
0,7574
0,2750
0.2027
3.44
Pre defined Oder and Model
22.35
0.5862
0,2858
0.1997
4.65
35.7
22.38
0.7639
0,2614
0.1986
3.48
22.1
Haman Expert RestoreAgent
22.96
0,7082
0.2861
0.2051
21.2
22.51
0.7634
0,2670
0.2014
3.73
22.95
0.7097
0.2615
0.1887
6.35
3.7
115.5
22.61
0,7700
0,2513
0.1890
4.38
12.9
16.6
3.2.2 Data Construction
To fully leverage the potential of multimodal large models, we construct a substantial dataset of paired training samples. The process begins with applying various types of degradation to an image. Subsequently, we determine the optimal restoration pipeline using model tools for processing. For each image undergoing multiple degradations, a comprehensive search is conducted to identify the best restoration pipeline, as shown Figure 3 This involves generating all possible permutations of task execution sequences and model combinations, applying each pipeline to the degraded image, and assessing the quality of the restored outputs using a scoring function ST, By comparing the scores of all permutations, the pipeline with the highest score is selected as the optimal processing strategy a for the given image. Users can choose from various image quality assessment methods as the scoring function, customizing the evaluation process to their specific needs. Additional details regarding dataset construction are provided in the supplementary materials.
4 Experiment
4.1 Experimental Settings
Scoring function. To construct a comprehensive evaluation system, we integrate multiple diverse metrics. Specifically, we first standardized each individual metric separately and then summed the standardized results. This process can be described as follows. Let X; represent the i-th metric. We standardize each metric by calculating its z-score: Z, = X,-Ms where μ4 the mean and o; the standard deviation of the i-th metric. After standardizing all metrics, we aggregate the standardized scores to form the comprehensive evaluation score S: S = > Z, where n is the total number of metries. This method ensures that each metric contributes equally to the final evaluation, regardless of its original scale. Follow 226 20), evaluation metrics primarily include PSNR, SSIM, LPIPS (60), and DISTS 16.. These metrics are widely recognized for their ability to comprehensively reflect the outcomes of image restoration. We also provided the results of models trained on individual metrics.
7

---

Input
Drep
Canceiing
2
Defaming
Srpp
Vatton Detie
Cancarg
Amppat
Seep
Delowrng
Prep
Centnng
Deftern:
Zeemih
Aby
2kp-
Denin
Zewerdn
Figure 4: Illustrations of RestoreAgent's choices demonstrate that our approach predicts the correct task sequence. Images with pink background show inappropriate decisions.
Dataset and model tool setings. To explore the feasibility of automating image restoration using multimodal models, we selected six distinct image restoration tasks: denoising, motion deblurring, deJPEG, deraining, dehazing, and low-light image enhancement. Each image in the dataset can exhibit up to four types of degradation. To validate the decision-making ability of the model when multiple models are available for single task, we constructed three specialized models for the denoising task, and three models have different noise levels: low, medium and high noise. Similarly, for the deIPEG task, we developed models specifically designed handle severe and mild JPEG compression artifacts. For the remaining tasks, each has a corresponding dedicated model. For the testing datasets, we assemble 200 images, mirroring the degradation types found in the training datasets, to facilitate evaluation. Detailed information is in the supplementary material. 4.2 Comparisons with Other Strategies
Compared methods. In this study, we conducted a comparative analysis of RestoreAgent against several alternative approaches:
Random selection of both the task order and the models, assuming accurate determination of task types.
Random task order, but models predicted by RestoreAgent.
Random model selection, but task orders predicted by Restore/gent.
For all images, using the human expert's predefined order and models, assuming accurate task type determination.
Human expert personally crafting solution for each image, determining the task sequence and models for each task. This method represents the most common scenario in real-world applications, where human decides how restore an image on case-by-case basis.
The human expert in this study has more than five years of research experience in low-level vision. Before crafting solutions, the expert familiarized themselves with each task degradation and the corresponding model's actual performance to ensure they could provide the best human-level solution. Results. Table reports the average metric results of our RestoreAgent and other decision-making approaches on seven different degradation combination datasets. As shown in Table using random order and model selection ranked lowest, achieving only 34.7% performance rating among all possible strategies. By setting predefined sequences and models for image processing by human experts, traditional methods rank in the top 22.1% of all possible strategies. This demonstrates that experience- based predefined rules often used in practical applications are more effective than completely random strategies. Human experts making specific decisions for each test image can further improve upon predefined rules, increasing the ranking from 22. 1% to 19.5%. This proves that using the same predefined rules to process all images is not optimal, while individualized decision-making for specific images can better enhance the effects. Then, the superior performance of our RestoreAgent (12.9%) over expert-based customization (19.5% shows that automated and
8

---

Imput (noiee thone)
AirNet
InstructIA
Prompt
MOIA
AutoDIR
DA-CUPP
RestoreAgent
Imput (PEG))
ESRAAN
StableSA
MOI
DA-CLIP
RestoreAgent
GT
WA WS
Input
MOIR
DA-CUP
RestoreAgent
Visual comparisons with All-in-One Methods. To ensure a fair comparison, All-in-One methods are tested only on the degradation types and datasets they support. The all-in-one approach still lacks the ability effectively handle images containing multiple types of degradation.
Figure 5:
Table 2:
Comparison of RestoreAgent with All-in-One methods for multi multi-degraded image restoration. We highlight best and second-best values for each metric.
noise + JPEG
haze + noise
rain + haze + noise
rain + baze nolse + JPGG
LPIPS
DISTS
PSNR
SSEM
LPIPS
DISTS
PSNK
SSIM
LPIPS
DISTS
Real ESRGAN
23.43
0,7242
0,3022
0.2106
Stabk-RR
47
17.61
0.4464
0.3735
0.2124
0.5897
PronpuX
30
16.13
0.5428
0.6696
0.3544
17.81
0.7099
0.6505
0.2317
MCCIR 27
23.95
0,6961
0.3266
0.2325
15.79
0.4790
0.7118
0.3628
16.22
0.638
0.719
0.2771
13,50
0.6410
0.4875
0,2939
0.2170
DA CLIP
3
22.47
0,6128
0.3525
0.2257
1698
0.7061
0.390
0.2757
15.44
0.0011
0.6597
0.2754
15:30
0.6863
0.3871
0,262)
AunDDRR
17.51
0.6942
0.4248
0.2444
19.22
0.7705
0.3023
0.1802
Restore Apent
23
0.2300
0.7816
02774
1901
data-driven decision-making in our method clearly outperforms traditional and experience- based human expert judgments. This is because human experts from their own experience can not make precise judgments about the advantageous scenarios of all models and the order of task execution, especially when numerous tasks and models are involved.
4.3 Comparisons with All-in-One Methods
To demonstrate the limitations of all-in-one methods in handling multi-degraded images, we compared our approach with various types of all-in-one models. To ensure fair comparison, tests were only conducted on degradation types and datasets that these all-in-one models were trained to support. Moreover, we repeatedly run the all-in-one model as many times as the number of degradation types of the test images to fully leverage its capabilities, thus ensuring fair comparison. The results are shown in Figure (5) and Table 2 Our RestoreAgent achieved a significant leading advantage across all tested degradation combinations. For the degradation types commonly encountered in traditional image resorttes,, such as noise and JPEG compression artifacts, our approach significantly outperformed established methods like Real-ESRGAN and the sota SR method, StableSR. For broader range of degradation types, our method retained considerable advantage. Among these all-in-one approaches, InstructIR and AutoDIR face two major issues; manually predetermined or

---

PSNR
Table 3: RestoreAgent
Value
ronking (16)
Value
ranking (56
Value
ranking (5)
Value
making (k)
possesses the flexibility to adapt to various opti- mization objectives, en- abling the generation of decision-man makiing re- sults tailored to specific target criteria.
Pre-delined and Model
22.38
25.4
0,7639
26.5
0.2644
25.2
0.1956
22.1
Human Expert
22.51
20155
0.7634
22.6
0,2670
25.1
0.2014
25.8
RestorAgent balanced
22.61
19.9
0,7700
21.4
022513
17.1
0.1890
RestorAgent- PSNR
22.72
RestorAgent-SSe SSIM
0.7766
16.5
RestorAgent LPIPS
RestorAgent- DISTS
0.1875
129
Table 4: Analysis of the effect of training data size. Our model shows strong performance with smaller datasets (7k), but increasing the data volume (23k) results in further enhanced outcomes.
Table 5: Fast adaptation new task (desnowing) in half an hour.
balanced ranking 161
Random
0.54
27.1
Pre-defined Order Model
3.82
9.2
PSNR
SSIM
LPIPS
DISTS
balanced
ranking (%)
Human Expert
3.91
8.5
RestoreAgent
4.23
4.3
Random
21.31
0.7139
0.3246
0.2241
1.92
34.7
Human Expert
22.51
0.7634
0.2670
0.2014
3.73
19.5
Table 6: Step-wise planning.
RestoreAgent
balanced
ranking /64
.7k
22.63
0.7669
0.2568
0.1922
4.10
16.2
Human Expert
5.42
21.2
- 14k
22.57
0,7664
0.2528
0.1902
4.26
13.6
Restore Agent
6,35
5.7
23k
22.61
0.7700
0.2513
0.1890
4.38
12.9
RestoreAgent + Step-wise
6.38
4.5
randomly decided execution order, and using single model to address all types of degradations. These limitations often result in incomplete restoration, as depicted Figure These results underscore the limitations of all-in-one models, validating our initial hypothesis.
4.4 Adapting Different Optimization Objectives
As discussed in the method, our proposed method can adapt to various optimization objectives, enabling the decision decision-making results tailored to specific target criteria. To verify it, we present the results of models trained with different individual metrics as the optimization objective in Table 3 The results indicate that when model is trained with a single metric, the performance of the corresponding metric can be significantly improved compared to the balanced model. This showcases the adaptability and effectiveness of our method in catering to specific optimization goals.
4.5 Extending for New Tasks and Models
The proposed RestoreAgent demonstrates remarkable adaptability and extensibility, allowing for swift fine-tuning accommodate new task types and incorporate additional models. To validate this capability, we introduced a new task, desnowing, along with its corresponding model. Building upon the RestoreAgent previously trained on six tasks, we performed rapid fine-tuning by integrating the desnowing task. Within thirty minutes, our model achieved exceptional performance on the new task type. As shown in T our approach quickly surpassed human expert-level proficiency on the new task and model. This validation underscores the practical value of our method, allowing efficient integration of additional tasks with minimal resource expenditure.
4.6 Step-wise Re-planning and Rollback
As mentioned in Section 3.2] RestoreAgent supports iterative decision-making with historical context awareness. dynamically adjusts strategies during image restoration, reassessing image state after each step and rolling back if needed. As demonstrated in Table 6 we conducted experiments complex dataset incorporating four distinct types of image degradation: Motion Blur, Rain, Noise, and JPEG compression. Results show that while the single prediction approach performs well, iterative step-wise replanning further enhances restoration outcomes, allowing for precise control and mid-course corrections. The initial decision's 's performance is already strong, step-wise replanning thus offers incremental yet valuable improvements to an already effective process.
5 Conclusion
Our research identifies key factors in processing multi- multi-degraded images, including task execution order, model selection, and the limitations of the all-in-one approach. Based on these insights, we present RestoreAgent, an agent model that makes intelligent processing decisions based on image degradation and user objectives. Experiments show that our pipeline outperforms the all-in-one method and surpasses human experts in eeeiisoonmmkinn performance.
10

---

Acknowledgments. This work is supported by the Grangzhou-HKUSST(GZ)) Joint Funding Pro- gram (No. 2023A0310671), the Guangzhou Municipal Science and Technology Project (Grant No. 2024A0414230), Guangdong Provincial Key Lab of Integrated Communication, Sensing and Computation for Ubiquitous Internet of Things(No.2023B12 220000007), and the National Natural Science Foundation of China (Project No. 61902275).
References
[1] Yuang Ai, Huaibo Huang, Xiaoqiang Zhou, Jiexiang Wang, and Ran He. Multimodal prompt perceiver: Empower adaptiveness, generalizability and fidelity for all-in-one image restoration. arXiv preprint arXiv:2312.0299 2023.
[2] Yuanhao Cai, Hao Bian, Jing Lin, Haoqian Wang. Radu Timofte, and Yulun Zhang. Retinexformer: One-stage caanee--based transformer for low-light image enhancement. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 12504-12511 2023.
[3] Haoyu Chen, Jinjin Gu, Yihao Liu, Salma Abdel Magid, Chao Dong. Qiong Wang. Hanspeter Pfister, and Lei Zhu. Masked image training for generalizable deep image denoising. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 1692-1703, 2023.
[4] Haoyu Chen, Wenbo Li, Jinjin Gu, Jingjing Ren, Haoze Sun, Xueyi Zou, Youliang Yan, Zhensong Zhang, and Lei Zhu. Low-res leads the way: Improving generalization for super-resolution by self-supervised learning. CVPR, 2024.
[5] Haoyu Chen, Jingjing Ren, Jinjin Gu, Hongtao Wu, Xuequan Lu, Haoming Cai, and Lei Zhu. Snow removal in video: A new dataset and a novel method. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 13211-13222 October 2023.
[6] Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang I,, Zechun Liu, Pengchuan Zhang, Raghuraman Krish- namoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Flhoseeiyy Minigpt-v2: large language model as unified interface for vision-language multi-task learning. arXiv preprint 2rX1v:2310.09477 2023.
[7] Sixiang Chen, Tian Ye, Jinbin Bai, Erkang Chen, Jun Shi, and Lei Zhu. Sparse sampling transformer with uncertainty-driven ranking for unified removal of raindrops and rain streaks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13106-13117. 2023.
[8] Sixiang Chen, Tian Ye, Yun Liu, Jinbin Bai, Haoyu Chen, Yunlong Lin, Jun Shi, and Erkang Chen. Cplformer: Cross-scale prototype learning transformer for image snow removal. In Proceedings of the 31 ACM International Conference on Multimedia, MM '23, page 4228-4444488 4239, New York, NY, USA, 2023. Association for Computing Machinery.
[9] Sixiang Chen, Tian Ye, Yun Liu, and Erkang Chen, Snowformer: Context interaction transformer with scale-awareness for single image desnowing. arXiv preprint ar0iv:2208....... 09703, 2022.
[10] Sixiang Chen, Tian Ye, Yun Liu, and Erkang Chen. Snowformer: Context interaction transformer with scale-awareness for single image desnowing. arXiv preprint arXiv:2208.099 09703, 2022.
[11] Xiang Chen, Hao Li, Minggiang Li, and Jinshan Pan. Learning a sparse transformer network for effective image deraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5896-58666 59066590552222 2023.
[12] Jun Cheng. Tao Liu, and Shan Tan. Score priors guided deep variational inference for unsupervised real- world single image denoising. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 12937-12948, 2023.
[13] Sung-lin Cho, Sco- Sco-Won Ji, Jun-Pyo Hong, Seung-Won Jung, and Sung-Jea Ko. Rethinking coarse-to- fine approach in single image deblurring. In Proceedings of the IEEF/CVF international conference on computer vision, pages 4641-4651 2021.
[14] Marcos V Conde, Gregor Geigle, and Radu Timofte. High-quality image restoration following human instructions. arXiv preprint arXiv:2401. 16468, 2024.
[15] XTuner Contributors. Xtuner: A toolkit for efficiently fine-tuning 1lm. xtuner 2023.
[16] Keyan Ding, Kede Ma. Shiqi Wang, and Eero P. Simoncelli. Image quality assessment: Unifying structure and texture similarity. CoRR, abs/2004.07728, 2020.

---

[17] Jiangxin Dong, Jinshan Pan. Zhongbao Yang, and Jinhui Tang. Multi-scale residual low-pass filter network for image deblurring. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 12345-12354, 2023.
[18] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong. Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2.333.33377 2023.
[19] Xueyang Fu, Zheng-Jun Zha, Feng Wu, Xinghao Ding, and John Paisley. Jpeg artifacts reduction via deep convolutional sparse coding. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
[20] Jinjin Gu, Haoming Cai, Haoyu Chen, Xiaoxing Ye, Jimmy Ren, and Chao Dong. Image quality assessment for perceptual image restoration: A new dataset, benchmark and metric. arXiv preprint arXiv:2011.15002. 2020.
[21] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu. Yuanzhi Li, Shean Wang. Wang, and Weizhu Chen. Lora: Low-rank adaptation large language models, arXiv preprint (06.:i::666000000 09685, 2021.
[22] Seo-Won Ji, Jeongmin Lee, Seung- Wook Kim, Jun-Pyo Hong, Seung-Jin Back, Seung- Won Jung, and Sung-Jea Ko. Xydeblur: Divide and conquer for single image deblurring. In Proceedingy of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 17421-17430, June 2022.
Seung-Won
[23] Jiaxi Jiang, Kai Zhang, and Radu Timofte. Towards flexible hlind jpeg artifacts removal. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 4997-5006, 2021.
[24] Yitong Jiang. Zhaoyang Zhang, Tianfan Xue, and Jinwei Gu. Autodir: Automatic all-in-one image restoration with latent diffusion. arXiv preprint arXiv:2310. 2023.
[25] Xiaoyu Jin, Yuan Shi, Bin Xia, and Wenming Yang. Llmra: Multi-modal large language model based restoration assistant. arXiv preprint arXiv:2401. 11001, 2024.
[26] Gu Jinjin, Cai Haoming, Chen Haoyu, Ye Xiaoxing, Jimmy S Ren, and Dong Chao. Pipal: a large-scale image quality assessment dataset for perceptual image restoration. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part XI 16, pages 633-66 651. Springer, 2020.
[27] Xiangtao Kong, Chao Dong, and Lei Zhang, Towards effective multiple-in-on image restoration: A sequential and prompt learning strategy. arXiv preprini 2:Xiv:2401.03377 2024.
[28] Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych, Dmytro Mishkin, and Jifí Matas. Deblurgan: Blind motion deblurring using conditional adversarial networks. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 8183-8192, 2018.
[29] Samuli Laine, Tero Karras, Jaakko Lehtinen, and Timo Aila. High-quality self-supervised deep image denoising. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'A d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc., 2019.
[30] Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila. Noise2noise: Learning image restoration without clean data. arXiv preprint arXiv Arxiv:1003. 1803.08 2018.
[31] Boyun Li, Xiao Liu, Peng Hu, Zhongqin Wu, Jiancheng Lv. and Xi Peng. All-in-one image restoration for unknown corruption. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17452 17462, 2022.
[32] Ji Li, Weixi Wang, Yuesong Nan, and Hui Ji. Self-supervised blind motion deblurring with deep expectation maximization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13986-13996. June 2023.
[33] Zilong Li, Yiming Lei, Chenglong Ma, Junping Zhang, and Hongming Shan. Prompt-in in in-prompt learning for universal image restoration. arXiv preprint arXiv:2312.05008 2023.
[34] Xingi Lin, Jingwen He, Ziyan Chen, Zhaoyang Lyu, Ben Fei, Bo Dai, Wanli Ouyang, Yu Qiao, and Chao Dong. Diffbir: Towards blind image restoration with generative diffusion prior. arXiv preprint arXiv:2308.15 15070, 2023.
[35] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. 2023.
12

---

[36] Lin Liu, Lingxi Xie, Xiaopeng Zhang, Shanxin Yuan, Xiangyu Chen, Wengang Zhou, Houqiang Li, and Qi Tian. Tape: Task-agnostic prior embedding for image restoration. In European Conference on Computer Vision, pages 447-464, Springer, 2022.
[37] Ziwei Luo, Fredrik K Gustafsson, Zheng Zhao, Jens Sjölund, and Thomas B Schön. Controlling vision- language models for universal image restoration. arXiv preprint arXiv:2310.0101. 2023.
[38] Dongwon Park, Byung Hyun Lee, and Se Young Chun. All-in-one image restoration for unknown degradations using adaptive discriminative filters for specific degradations. In 2023 JEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 5815-5824. IEEE, 2023.
[39] Zhiliang Peng, Wenhui Wang. Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 14824, 2023.
[40] Vaishnav Potlapalli, Syed Waqus Zamir, Salman Khan, and Fahad Shahbaz Khan. Promptir: Prompting for all-in-one blind image restoration. arXiv preprint arXiv:2306. 13090, 2023.
[41] Xu Qin, Zhilin Wang. Yuanchao Bai, Xiaodong Xie, and Huizhu Jia. Ffa-net: Feature fusion attention network for single image dehazing. In Proceedings of the AAAI conference artificial intelligence, volume 34, pages 1 11908-11915, 2020.
[42] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transfcrable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763 PMLR, 2021.
[43] Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Ahdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Erix Xing, Ming-Hsuan Yang. and Fahad Khan. Glamm: Pixel grounding large multimodal model. arXiv preprint 1I.i:33311033366 2023.
[44] Mengwei Ren, Mauricio Delbracio, Hossein Talebi, Guido Gerig, and Peyman Milanfar. Multiscale structure guided diffusion for image deblurring. In Proceedings of the JEEE/CVF International Conference Computer Vision, pages 10721-10733, 2023.
[45] Haoze Sun, Wenbo Li, Jianzhuang Liu, Haoyu Chen, Renjing Pei, Xueyi Zou, Youliang Yan, and Yujiu Yang. Coser: Bridging image and language for cognitive super-resolution. arXiv preprint 2....:2311..161/11 2023.
[46] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anna Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 13971, 2023.
[47] Jianyi Wang, Zongsheng Yue, Shangchen Zhou, Kelvin CK Chan, and Chen Change Loy. Exploiting diffusion prior for real-world image super-resolution. arXiv preprint 20X1v:2805.07005, 2023.
[48] Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan. Real-esrgam: Training real-world blind super- resolution with pure synthetic data. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1914, 2021.
[49] Zejin Wang, Jiazheng Liu, Guoqing Li, and Hua Han. BlindZunblind: Self Self-supervised image denoising with visible blind spots. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 2027-2036, June 2022.
[50] Yanyan Wei, Zhao Zhang, Jiahuan Ren, Xiaogang Xu, Richang Hong, Yi Yang, Shuicheng Yan, and Meng Wang. Clarity chatgpt: An interactive and adaptive processing system for image restoration and enhancement. arXiv preprint arXiv:2311. 1665, 2023.
[51] Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G. Dimakis, and Peyman Milanfar. Deblurring via stochastic refinement. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), pages 16293-16303, June 2022.
[52] Haiyan Wu, Yanyun Qu, Shaohui Lin, Jian Zhou, Ruizhi Qiao, Zhizhong Zhang, Yuan Xie, and Lizhuang Ma. Contrastive learning for compact single image dehazing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10551-10560, 2021.
[53] Rongyuan Wu, Tao Yang. Lingchen Sun, Zhengqiang Zhang, Shuai Li, and Lei Zhang. Seesr: Towards semantics-awatt aware real- world image super-resolution. arXiv preprint 2rXiv:2311.16518 2023.
13

---

[54] Ruiqi Wu, Zhengpeng Duan, Chunle Guo, Zhi Chai, and Chongyi Li. Ridep: Revitalizing real image dehazing via high-quality codebook priors. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2023.
[55] Xiaohe Wu, Ming Liu, Yue Cao, Dongwei Ren, and Wangmeng Zuo. Unpaired learning of deep image denoising. In European conference on computer vision, pages 352-368. Springer, 2020.
[56] Fanghua Yu, Jinjin Gu, Zheyuan Li, Jinfan Hu, Xiangtao Kong. Xintao Wang. Jingwen He, Yu Qiao, and Chao Dong. Scaling up to excellence: Practicing model scaling for photo-realistic image restoration in the wild. arXiv preprint ar Xiv:24001. 13627, 2024.
[57] Yu, Chao Dong, Liang Lin, and Chen Change Loy. Crafting toolchain for image restoration by deep reinforcement learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
2018.
[58] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming Ling-Hssuan Yang. Restormer: Elficient transformer for high-resolution image restoration. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 5728-5739, 2022.
[59] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang, Beyond gaussian denoiser: Residual learning of deep for image denoising. IEEE transactions image processing, 26(7):3142- 3155, 2017.
[60] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep fcatures perceptual metric. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 186-595, 2018.
[61] Zhang, Dasong Li, Lung Law, Xiaogang Wang, Hongwei Qin, and Hongsheng Li. Idr: Self- supervised image denoising via iterative data refinement. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2098-2107, June 2022.
[62]
Yihua Zhang, Pingzhi Li, Junyuan Hong, Jiaxiang Li, Yimeng Zhang. Wenqing Zheng, Pin- Yu Chen, Jason D Lee, Wotao Yin, Mingyi Hong, et al. Revisiting zeroth-order optimization for memory-efficient Ilm fine-tuning: A benchmark. arXiv preprint arXiv:2402.1 1592, 2024.
[63] Yulun Zhang, Kumpeng Li, Kai Li, Bineng Zhong, and Yun Fu, Residual non-local attention networks for image restoration. arXiv preprint arXiv:/903 10082, 2019.
14

---

A Appendix supplemental material
A.1 Model Tool Setings
As shown in Table 7 for the tasks of denoising and deJPEG, as well as deraining, we employ Restormer [58] as model. For dehazing, we utilize RIDCP [54), while for motion deblurring, we use DeblurGANv2 28). For desnowing, we implement Snowformer [0]. For low-light enhancement, use Retinexformer [2]. It is noteworthy that the models we using not the latest state-of-the the- art models, indicating that there is significant room for improvement in our models.
A crucial consideration in image restoration is the limited generalization capability of many current models, which often fail to maintain performance when faced with subtle variations in image degradation. This necessitates the selection of more robust models. For example, in approach to denoising. we enhance model generalization by incorporating not only Gaussian noise but also random blur and other noise types during training. This strategy enables the model to address more complex degradation scenarios effectively.
Table 7: Model tools for different restoration tasks.
Table 8: Testset details.
Task
Model Tools
Degradatio
Number Images
Restormer (trained on large noise level)
Gaussian denosing
Restormer (trained medium noise level)
Naise JPEG
50
Restormer (trained on noise level)
Noise Lew light
30
Motion Blur + Noise JPBG
30
Restormer (trained high quality factor)
DeIPEG
Restormer (truined on low quality factor)
Rain Noise JPBG
20
Haze Noise + IPEG
30
Dehazing
RIDCP
Haze Rain Noise JPEG
20
Deraining
Restormer
58
Motion Blur + Rain Noise JPEG
20
Motion deblurring
DeblurGANvv
Total
200
Low-light enhancement
Retinexfomner
A.2
Dataset Construction Details
Input
Answer
Function
How to enhance the quality this image? Execution history:
Denoising, noise level, Dehazing,
Intiiate full enharccement images.
(Primary)
Imput
DeJPEG, high quelity factor
How enhance the quality this image?
Danoising, low noise lovel,
Dynamically adjust strategles based Intermediate rosults.
2
Denolsing
Exacution history: 1, Dencising, low noisa Inval.
Dehazing, DeJPPG, high quality tactor
eriancc quality image?
Idantity and cornect suboptimal through roliback machanisms.
Execution history: 1, Dehozingg
How enhance the quality of this image?
Denoising, low noize level, Dehazing,
Avaid repatition of procedures post: roltbacic
Inout
history: Rollback from Dehexiing
DeJPEG, high quelity factor
Denoising DcJPEG
enhencce quality image?
Stop-
Recognize when image quality has reached its optimal
Execution history: Denoising, kow noise level,
Dohazing, DaJPEG, high quality
Figure 6: Five scenarios for dataset construction and their corresponding examples.
15

---

Figure iillustrates 5 scenarios incorporated into our dataset, designed to enhance the versatility and robustness of the RestoreAgent model:
(1) Once we obtain a degraded image along with its corresponding optimal decision results, we can construct the primary part of dataset. This part consists of degraded images in their original, unprocessed state. For these inputs, the RestoreAgent receives prompt: "How to enhance the quality of this image? Execution history: None." This scenario trains the model to formulate comprehensive enhancement strategies from scratch, encompassing multiple restoration steps. This part of the data exceeds 23k pairs.
(2) To foster dynamic decision-making capabilities, we introduce a second category of training instances. Here, the input comprises partially processed images (e.g., after denoising) along with their execution history. This approach enables the RestoreAgent to adapt its predictions based on intermediate results, promoting more flexible and context-aware enhancement process.
(3) The third scenario addresses situations where the model identifies suboptimal outcomes from particular enhancement step. In such cases, the RestoreAgent is trained to output "Rollback," indicating the need to revert to a previous state and recalibrate its strategy. This feature is crucial for maintaining high-quality outputs and avoiding the propagation of errors through the enhancement pipeline. We select from erroneous paths (the decisions with the worst metric results) to construct this portion of the paired data, the worst paths require rollback.
(4) Following a rollback event, our fourth data category provides the model with information about the specific step that triggered the rollback. This guidance is essential in preventing the model from repeating ineffective procedures, thus streamlining the enhancement process and improving efficiency.
(5) The final scenario in our training regime represents fully processed images that require no further enhancement. In these instances, the RestoreAgent is trained to recognize optimal image quality and output "Stop". effectively terminating the enhancement sequence.
By incorporating these diverse scenarios, we aim to develop highly adaptive and efficient image restoration system capable addressing a wide array of real-world image degradation challenges. For computational efficiency, unless specifically mentioned otherwise, our default experiments are based on single planning for the initial image rather than using iterative step-wise replanning.
A.3 Testset details
The specific details of our test set presented in Table which demonstrates our construction of various combinations of degradation types. Each image in the set contains a minimum of one and a maximum of four types of degradation, with the entire set comprising 200 images.
A.4 Training Setups
In this study, we incorporate the CLIP pre-trained Vision Transformer (ViT-L/14) 42 encoder to convert input images into visual tokens. For the language model, we utilize the Llama3- 7B [46]. Despite their capabilities, pre-trained LLMSs fail to provide accurate responses without dataset-specific fine-tuning. To address this, we adopt LoRA [2I). a fine-tuning technique that efficiently modifies limited number of parameters within the model. Following (21). apply LoRA to adjust the projection layers in all self-attention modules of both the vision encoder and the LLM, thereby generating our RestoreAgent. We employ the Xtuner framework [5] to facilitate the training process. For our experimental setup, we configure the LoRA rank to 16. The RestoreAgent undergoes training across ten epochs 4 NVIDIA RTX A 1100 GPUs, with batch size of 32. We employ the Adam optimizer and a learning rate of 0.00002. The total duration of the training process approximates hours.
the image
A.5 Analysis
Figure 4 and illustrate RestoreAgent's decision-main making process and the importance of model selection, respectively. Figure further demonstrates why human decision-making often yields suboptimal results in image restoration tasks. Figure 8h exemplifies the nuanced challenges in degradation assessment. Despite identical backgrounds and degradation types, subtle variations in degradation features lead to divergent optimal restoration sequences. For instance, the sequence
16

---

Dereecin
dag ratuttioinnnnn
Input 10000
DaBaniing Deliceing D.DPPP det song
DciPso
1.Ouhannng C.CoNcaing
Same types
Finding the sole offective strategy amidst myllad of exceecingly challenging human
requira distiics
orders
Figure 7: Challenges in human expert decision-maker This figure illustrates the difficulty faced by human experts in discerning minute differences between degradation patterns, leading to suboptimal restoration strategics.
CONn
Stop
light
Stop
DzNosing
Step
DcHazing
Zoonsin
Large nocee level
Modium noleo
Inpot
LEGS
Step
DeNisiing
Stap
Del baring
Zeem-In
Smalll noiae levsig
Lango
Figure 8: Examples of model decisions made by RestoreAgent. This figure demonstrates how choosing the appropriate model for a specific restoration task significantly affects the outcome quality, We present PSNR and LPIPS metrics for each image. Images with a pink background indicate examples of inappropriate decisions (zoom-in for better view).
"DeNoising DeRaining DeJPEG DeHazing" proves effective for the upper images but fails for the lower row. Conversely, the sequence "DeRaining ~ DeNoising - DeJPEG -> DeHazing" yields optimal results for the lower row but is suboptimal for the upper row. This dichotomy underscores the difficulty human experts face in discerning minute degradation differences, thereby compromising effective decision-making.
The complexity of optimal restoration sequencing is further accentuated in Figure Sp. Here, we demonstrate scenarios where only one specific sequence among numerous permutations yields satisfactory results. This observation highlights the formidable challenge posed to human decision- makers in identifying the singular effective restoration pathway amidst multitude of possibilities.
These findings collectively emphasize the superiority of automated, data-driven approaches in navigating the intricate landscape of image restoration. The Restore/gent's ability to discern and adapt to subtle degradation variations surpasses human capabilities, particularly in scenarios where optimal restoration sequence is non-intuitive and highly specific to individual image characteristics.
A.6 Discussion
Comparison with assistants with all-in-one models. Assistants that employ unified models, such LLMRA (25 and AutoDIR (24) attempt to handle diverse tasks, degradation patterns, and intensities using single model. As discussed in Section I.1 these all-in-one models face significant challenges, including restricted task scope and compromised performance, which greatly limit their effectiveness in real-world applications. Conversely, our method leverages various model experts to address specific situations, the upper bound of our pipeline is determined by the latest SOTA models, allowing us maximally leverage the latest advancements in the field without being constrained by
17

---

the limitations of an all-in-one model. Furthermore, as detailed in Section 4.4 our RestoreAgent exhibits high efficiency incorporating new tasks and models, showcasing greater flexibility,
Comparison with assistants with tool use. Image restoration assistants that utilize tool libraries, such as Clarity ChatGPT [50] and RL-Restore [57]. Clarity ChatGPT merely identifies the degradation in images, follows rigid execution strategy, lacking the ability to make dynamic decisions on task execution order and select the best model. As discussed in Section 1.2.1 [.2. an inappropriate task execution sequence and model selection can leading to lower performance in subsequent operations. RI.-Restore, on the other hand, uses reinforcement learning sequence decision-making and model selection. However, its task definition is overly simplistic, limited to three degradation types (noise, blur, and JPEG) with a narrow degradation range. Also, training reinforcement learning- based methods is more challenging and may result in lower precision, making it difficult to achieve high performance in complex and varied scenarios. Conversely, the integration task definition with advanced multimodal models allows method to effectively manage various degradation types and intensities. This adaptability enhances its efficacy, positioning our approach as promising solution for image restoration tasks.
comprehensive
A.7 Alation Study
Training data amount. To investigate the effect of training data volume on our method, we evaluated the performance of the RestoreAgent model trained datasets consisting of 7,000, 14,000, and 23,000 data pairs; see Table 4 The results demonstrate that even with the smallest dataset of 7k pairs, RestoreAgent achieves superior performance over both random and human expert benchmarks. More notably, the training data volume increasing from 7k to 14k incurs a substantial performance improvement with the ranking percentage decreasing from 16.2% 13.6%. With 23k data pairs, the performance further improves, achieving a ranking percentage of 12.9%. This indicates that using more training data boosts RestoreAgent model. These findings emphasize the robustness of our approach, demonstrating that while larger datasets do enhance performance, our model already provides significant benefits with relatively smaller datasets.
A.8 Limitation and Future Work
The primary limitation of our study is the confined scope of models and tasks examined. While research offers valuable insights into Restore performance across several degradation scenarios, it does not encompass the full spectrum of restoration models or image degradation tasks currently available.
Another limitation pertains to the limited generalization capability of current image restoration models. These models often exhibit a notable decrease in performance or fail respond adequately when faced with even minor variations in image degradation patterns. This limitation greatly narrows our selection of model tools, requiring us choose more robust and generalizable model tools. The challenge underscores a critical need in the field of image restoration: future models must go beyond simply overfitting training data. Rather, they should exhibit better generalization and increased efficiency in handling worldworld degradation cases.
Our future work will focus on significantly expanding the range of image restoration models incorpo- rated into our multimodal large language model. This expansion aims to enhance RestoreAgent's capabilities across broader scope of restoration tasks and degradation types. By integrating more diverse set of state-of-the-art models, we seek to create a more comprehensive and versatile restoration framework.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the main claims made in the abstract and introduction accurately reflect the paper's 's contributions
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the work in the supplemental material section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
19

---

Yes, for each theoretical result, we provide the full set of assumptions and complete proof.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof' provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Yes, our method is easy to be reproduced, and provide all information.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
Justification:
We use public code and data.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cpippi.ccppulli//uuuii wwwiiicgguie//eeeeeeeeeic CoodeSubeiiie ssi wsionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, we give all the details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: See Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
21

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Section 4.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://wwww....ooollllcccccceeea
Answer:
[Yes]
Justification: We conform, every respect, to the NeurIPS Code of Ethics as outlined at the provided
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
There is no societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
22

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The creators or original owners of assets used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
23

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification:
See supplemental materia.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: There are no potential risks.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "cj46TZGoFY", "reviewer": "Reviewer_DJ9b", "summary": "This paper introduces RestoreAgent, an innovative image restoration system that leverages multimodal large language models to autonomously handle images with multiple types of degradation. The system addresses limitations of existing all-in-one models and fixed task sequences by dynamically adapting to each image's specific degradations. RestoreAgent can identify degradation types, determine appropriate restoration tasks, optimize the execution sequence, select the most suitable models, and execute the restoration process autonomously. The authors present a method for constructing training data and demonstrate that RestoreAgent outperforms existing methods and human experts in handling complex image degradations.", "questions": "1. The current method appears to predict all execution steps at once for a given input image. In Figure 3, each image has a dashed line pointing to the input. Does this imply that after each execution, the result can be fed back as input? (Based on my understanding, this system supports this) The paper seems to lack analysis and experiments related to this aspect. Could the authors provide more details on this part?\n2. The authors have proposed a testing dataset to evaluate multi-task processing capabilities. Will this dataset be made publicly available to facilitate further research by other researchers?", "limitations": "See Weaknesses", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper represents a innovation and a good contribution in image restoration and potentially opens up a new research direction for this area.\n2. The motivation is strong. The authors effectively demonstrate the importance of task execution order and model selection in multi-task scenarios. The designed system adeptly addresses these issues.\n3. Experimental results indicate that RestoreAgent's decision-making capabilities in handling complex degradations surpass those of human experts. This kind of pipeline also surpass all-in-one models.\n4. The paper is generally well written and clear to understand.", "weaknesses": "1. The paper constructs a training dataset for training the multimodal large language model and a testing dataset as a benchmark for evaluating performance across multiple tasks. More details and explanations regarding the construction methods of these datasets would be beneficial.\n2. Table 1 presents performance rankings using both ordinal and percentage forms. The definitions and explanations for these ranking forms are somewhat lacking, which might require readers to spend extra time understanding them. Clearer explanations would facilitate better comprehension.\n3. The proposed Autonomous Restoration Agent represents a novel paradigm that is likely to encounter numerous new challenges. Beyond the issues already mentioned in the paper, the authors could consider discussing additional limitations and future research directions for this paradigm. This would help future researchers better follow and improve upon this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "`Q1`. **More details regarding the construction methods of these datasets**\n\nThank you for your feedback. We have significantly expanded the relevant sections in the revised version of our paper to offer a much more comprehensive explanation of our data preparation process.\n\nRegarding the training dataset, the following is part of the part of newly added detailed content:\n\nThe training pairs construction begins with applying various types of degradation to an image. Subsequently, we determine the optimal restoration pipeline using model tools for processing. This involves generating all possible permutations of task execution sequences and model combinations, applying each pipeline to the degraded image, and assessing the quality of the restored outputs using a scoring function.  The model for different restoration tasks are shown in **Tbale 1**. \n\n**Table 1: Model tools for different restoration tasks.**\n\n| Task | Model Tools |\n|---|---|\n| Gaussian denosing | Restormer (trained on small / medium / large noise level) |\n| DeJPEG \t  | Restormer (trained on low / high quality factor)  |\n| Dehazing  | RIDCP |\n| Deraining | Restormer |\n| Motion deblurring | DeblurGANv2 |\n| Low-light enhancement | Retinexformer |\n\n **Table 2** illustrates 5 scenarios incorporated into our dataset, designed to enhance the versatility and robustness of the RestoreAgent model:  \n\n**Table 2: Five scenarios for dataset construction and corresponding examples.**\n\n| Index | Input | Answer | Function |\n|---|---|---|---|\n| 1 (Primary) | How to enhance the quality of this image? Execution history: None. | 1. Denoising, low noise level, 2. Dehazing, 3. DeJPEG, high quality factor. | Initiate full enhancement sequences for degraded images. |\n| 2 | How to enhance the quality of this image? Execution history: 1. Denoising, low noise level. | 1. Denoising, low noise level, 2. Dehazing, 3. DeJPEG, high quality factor | Dynamically adjust strategies based on intermediate results. |\n| 3 | How to enhance the quality of this image? Execution history: 1. Denoising, low noise level, 2. Dehazing | Rollback. | Identify and correct suboptimal steps through rollback mechanisms. |\n| 4 | How to enhance the quality of this image? Execution history: 1. Denoising, low noise level. Rollback from Dehazing. | 1. Denoising, low noise level, 2. DeJPEG, high quality factor, 3. Dehazing. | Avoid repetition of ineffective procedures post-rollback. |\n| 5 | How to enhance the quality of this image? Execution history: 1. Denoising, low noise level, 2. DeJPEG, high quality factor, 3. Dehazing. | Stop. | Recognize when image quality has reached its optimal state. |\n\nAs for the testing dataset used as a benchmark, the details are presented in **Table 3**, which demonstrates our construction of various combinations of degradation types. Each image in the set contains a minimum of one and a maximum of four types of degradation, with the entire set comprising 200 images. \n\n**Table 3: Testset details.**\n\n| Degradation | # Images |\n|---|---|\n| Noise + JPEG | 50 |\n| Noise + Low light | 30 |\n| Motion Blur + Noise + JPEG | 30 |\n| Rain + Noise + JPEG | 20 |\n| Haze + Noise + JPEG | 30 |\n| Haze + Rain + Noise + JPEG | 20 |\n| Motion Blur + Rain + Noise + JPEG | 20 |\n| Total | 200 |\n\n---\n\n`Q2`. **Explanations of \"rankings\" in Table 1**\n\nWe've clarified the ranking system:\n\nFor individual test sets:    \nX/Y format: X is the average ranking, Y is total combinations.    \nExample: 6.4/64 means average 6.4th best out of 64 options.   \n\nFor overall average:    \nWe use percentages due to varying combination counts of different restoration tasks.   \nProcess: Calculate percentage ranking per set, then average.    \nExample: 12.9% means top 12.9% on average across datasets.\n\n---\n\n`Q3`. **Additional limitations and future research directions**\n\nThe primary limitation of our study is the confined scope of models and tasks examined. While our research offers valuable insights into RestoreAgent’s performance across several degradation scenarios, it does not encompass the full spectrum of restoration models or image degradation tasks currently available.  \n\nAnother limitation pertains to the limited generalization capability of current image restoration models. These models often exhibit a notable decrease in performance or fail to respond adequately when faced with even minor variations in image degradation patterns. This limitation greatly narrows our selection of model tools, requiring us to choose more robust and generalizable model tools. The challenge underscores a critical need in the field of image restoration: future models must go beyond simply overfitting training data. \n\nOur future work will focus on significantly expanding the range of image restoration models incorporated into our multimodal large language model. This expansion aims to enhance RestoreAgent’s capabilities across a broader scope of restoration tasks and degradation types. By integrating a more diverse set of state-of-the-art models, we seek to create a more comprehensive and versatile restoration framework.\n\n---\n\n`Q4`. **Step-wise Re-planning and Rollback.**\n\nYou are correct in your understanding. In the revised version of our paper, we have added a detailed discussion on this feature, which we call \"Step-wise Re-planning and Rollback.\" \n\nIn the answer to Q1, we have actually added some details (indexes 2, 3, and 4 of **Table 2**).\n**Table 4** shows experiments on a complex dataset with four image degradation types. While single prediction performs well, iterative step-wise replanning offers modest improvements. This indicates strong initial performance, with replanning serving as a refinement tool for incremental enhancements.\n\n**Table 4: Step-wise planning.**\n\n|            | balanced | ranking /64 |\n|---|---|---|\n| Human Expert | 5.42 | 21.2 |\n| RestoreAgent | 6.35 | 5.7  |\n| RestoreAgent + Step-wise | **6.38**| **4.5** |\n\n---\n\n`Q5`. **Will this test set be made publicly available?**\n\nYes, we will.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NIwuCn7b0A", "reviewer": "Reviewer_NhBR", "summary": "This paper presents an image restoration pipeline designed to handle various degradation types and levels by leveraging MLLM’s capabilities to select the appropriate model and determine the execution order. It begins with an analysis of why execution order and utilizing multiple models for different degradation levels are crucial for restoring complexly degraded images. The paper then constructs an instruction dataset and fine-tunes the MLLM. Experimental results demonstrate the effectiveness of the proposed restoration pipeline.", "questions": "1.\nWhat if the degradation of the input image falls outside the predefined degradation scope? This could present a generalization issue, as the model might not perform well on unseen types or levels of degradation not covered in the predefined scope. Please discuss it.\n2.\nIn Table 2, it would be clearer to highlight the best method for each evaluation criterion. Additionally, please specify which methods the ranking improvement is compared against for better context.", "limitations": "The authors have addressed the limitations.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1.\nThis work presents a compelling analysis of complex image restoration. This insight is valuable given that degraded images in real-world scenarios often involve multiple types of degradation.\n2.\nThis approach leverages the strengths of different models for handling specific noise levels, thereby eliminating the trade-off between generalization and performance.\n3.\nThis paper formally defines the problem of handling multiple degradations and model selection in image restoration.\n4.\nExtensive experiments demonstrate superiority of such pipeline in processing degraded images with multiple degradations.", "weaknesses": "1.\nIn the introduction, it would be helpful to explain how the Multi-Level Learning Model (MLLM) excels at understanding different types and levels of image degradation. This will show why MLLM is well-suited for handling complex combinations of image degradation. Providing this clarity will make the benefits of using MLLM for image restoration more evident.\n2.\nWhen incorporating a new type of degradation, the cost extends beyond merely training the MLLM. Please also discuss the process of constructing training data for the newly added degradation and how it integrates with previously trained data.\n3.\nIn lines 211-212, please clarify what the mean and standard deviation are calculated over. The subscript \"i\" is already used for degradation type and it might be clearer to use another character.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "`Q1`: **In the introduction, it would be helpful to explain how the MLLM excels at understanding different types and levels of image degradation.**\n\nThank you for your valuable suggestion. We will incorporate the following explanation into our introduction to clarify how MLLMs excel at understanding and handling different types and levels of image degradation.\n\nMLLMs are well-equipped to handle various types and levels of image degradation due to their extensive training on diverse datasets. This training enables them to develop a deep understanding of the relationships between different modalities, which is crucial for identifying and addressing complex degradation patterns.\n﻿    \nMLLMs excel in image restoration due to their robust reasoning abilities and advanced pattern recognition skills. They can analyze and interpret intricate details of various degradations, discerning subtle variations in noise, blur, compression artifacts, and other forms of image deterioration. This capability allows MLLMs to make precise decisions on applying the most appropriate restoration techniques, even when faced with complex combinations of degradation effects. For instance, when encountering an image with mixed noise, blur, and compression artifacts, an MLLM can accurately determine the optimal sequence and type of restoration techniques to apply. \n﻿    \nFurthermore, the adaptability of MLLMs allows them to quickly learn and optimize for new degradation scenarios through fine-tuning. This flexibility is essential for real-world applications where the types and combinations of degradations can vary widely. By continually updating their knowledge base, MLLMs can maintain high performance across a broad spectrum of image restoration tasks.\n﻿    \n\n\n`Q2`: **Please also discuss the process of constructing training data for the newly added degradation and how it integrates with previously trained data.**\n\nThe process of constructing training data for new degradations follows the same methodology as for existing ones. For instance, if we were to add a new degradation type such as snow, we would randomly combine it with other existing degradations like noise and JPEG.\nSpecifically, we experiment with different permutations and combinations of these degradations and find the best one to generate diverse training data. \nOnce the new training data is generated, it is directly added to the original training dataset. This combined dataset is then used to fine-tune the model.\nAdding a new task requires a relatively small amount of data and has a quick training time. It allows for continuous improvement of the model without the need for complete retraining from scratch.\nWe have updated our manuscript to include a more detailed discussion of this process.\n\n\n\n`Q3`: **In lines 211-212, please clarify what the mean and standard deviation are calculated over. The subscript \"i\" is already used for degradation type and it might be clearer to use another character.**\n\nWe have revised lines 211-212. Our mean and standard deviation are calculated separately for each metric. Specifically, for each metric, we have results from all possible permutations and combinations restoration pipelines. We calculate the mean and variance using all these data points for each respective metric.\n\nAs you correctly pointed out, using the subscript \"i\" for both degradation type and in the summation could lead to confusion. We have updated our notation to use different characters for clarity. \n\n\n\n`Q4`: **What if the degradation of the input image falls outside the predefined degradation scope?**\n\nWe appreciate the reviewer's concern regarding the generalization capability of our model when faced with degradations outside the predefined scope. Here, we would like to further elaborate on this aspect.\n\nFirst of all, it is true that there might be generalization issues when encountering unseen types or levels of degradation. We acknowledge this limitation and have discussed it in detail in our manuscript.\n\nHowever, the primary and most significant contribution of our work is to demonstrate the feasibility and effectiveness of the proposed pipeline. The essence of our work lies in utilizing multimodal large models as agents for image restoration, which has shown promising results across various conditions. Our experiments show that the RestoreAgent performs exceptionally well in most complex scenarios, thus proving its viability.\n\nIn addition, our approach allows for flexible integration of different tools. If the model encounters degradations beyond the predefined scope, additional tools can be seamlessly incorporated. This adaptability ensures that our pipeline remains robust even when faced with unexpected degradation types.\nEssentially, the generalization issue is more related to the specific image restoration models (tools) used within the pipeline, rather than the pipeline itself. \n\nLastly, one of our future research directions is to significantly expand the scope of restoration tasks and incorporate a greater variety of restoration models. By doing so, we aim to enhance the system's capability to handle real-world complex degradations more effectively.\nThis expansion will involve integrating models trained on a broader spectrum of degradation types and more generalizable restoration models, ensuring that the pipeline can generalize better to unseen conditions.\n\n\n\n`Q5`: **Specify Table 2**\n\nThank you for your suggestion to improve the clarity of Table 2. We have updated Table 2 to highlight the best performing method for each evaluation criterion. We also have added a clear note to Table 2 specifying that the ranking improvement is compared against human experts.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OJHKrxIv4K", "reviewer": "Reviewer_Ux72", "summary": "This paper proposes a new pipeline to address multiple degradation, like noise, blur and low light. Besides, a RestoreAgent with multimodal large language models is introduced to assess the type and extent of degradations in the input images and perform dynamic restorations.", "questions": "Please see weaknesses.", "limitations": "Limitations are not mentioned in the paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and well-organised. \n2. The whole pipeline seems to be novel and reasonable. \n3. The method achieves SOTA performance on several benchmarks and different degradation tasks.", "weaknesses": "The overall motivation of this paper is commendable, but I have a few concerns:\n\n1. The author mentions that RestoreAgent can autonomously assess the type and extent of degradation in input images and perform restoration. This strategy is interesting. However, I am wondering how the order of different enhancement techniques is defined. For example, if the input has noise and rain streaks, how is the order of dehazing and denoising techniques determined? Will this affect performance?\n\n2. In contrast to other image enhancement techniques, the proposed RestoreAgent should first find a suitable restoration task and then select the most appropriate model to enhance the quality of the input. Therefore, I am concerned whether this process will increase the inference time. The authors should provide some computational analysis.\n\n3. The enhancement capabilities of this work rely heavily on existing enhancement frameworks. If existing frameworks cannot work well in some cases, such as extreme noise effects, I guess the proposed RestoreAgent may also fail. Is this true? If so, I suggest the authors mention this in the limitations section.\n\n4. The explanation of \"ranking\" and \"balanced\" in Table 1 is still unclear. The authors should clarify the definitions of these terms.\n\n5. It would be better to show more visual comparisons of the RestoreAgent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "`Q1`: **How the order of different enhancement techniques is defined. For example, if the input has noise and rain streaks, how is the order of dehazing and denoising techniques determined? Will this affect performance?**\n\nThank you for highlighting this important aspect. The order of applying different enhancement techniques can significantly impact the restoration results, as illustrated in Figure 2 (b.1, b.2) of our manuscript.\n\n**Impact of Degradation Order on Patterns**\n\nWhen constructing training data, the sequence of applying degradations like noise, JPEG compression, and blur affects the resulting degradation patterns. For instance, adding noise before JPEG compression/blur alters the noise characteristics, making it smoother due to the compression/blur effects. Conversely, adding JPEG compression/blur before noise results in unaltered noise characteristics. This interplay between degradations creates distinct patterns based on their application order. The MLLM can identify different task sequences through degradation patterns.\n\n**Influence of Restoration Order on Performance**\n\nSimilarly, the order of applying restoration techniques affects the degradation characteristics. For example, applying a deblurring model first can sharpen noise and rain streaks, altering their features. Conversely, applying a denoising model first can reduce noise but may impact rain streak features, making subsequent deraining less effective. An interesting case is when noise and rain streaks coexist: low noise levels may not impact deraining, but high noise levels can render deraining ineffective. Conversely, denoising first might alter rain streak features, affecting deraining performance.\n\nOur proposed RestoreAgent, being a learning-based method, is trained on extensive data to recognize degradation patterns and make informed decisions based on subtle variations. It autonomously determines the optimal sequence of restoration steps to effectively remove degradations. This is achieved through a comprehensive understanding of how different degradations and restoration techniques interact.\n\n\n\n`Q2`: **Inference time.**\n\nRestoreAgent is built on a MLLM. In our experiments. The image patch size we used is 512x512 pixels. In practice, the MLLM resizes all inputs to a fixed size, such as 224x224, and the inference time of RestoreAgent is approximately 0.4 seconds in RTX4090. The increase in inference time is very minimal. \n\n\n`Q3`: **If existing frameworks cannot work well in some cases, such as extreme noise effects, I guess the proposed RestoreAgent may also fail. Is this true? If so, I suggest the authors mention this in the limitations section.**\n\nWe appreciate the reviewer's concern regarding the generalization capability of our model when faced with degradations outside the predefined scope. Here, we would like to further elaborate on this aspect.\n\nFirst of all, it is true that there might be generalization issues when encountering unseen types or levels of degradation. We acknowledge this limitation and have discussed it in detail in our manuscript.\n\nHowever, the primary and most significant contribution of our work is to demonstrate the feasibility and effectiveness of the proposed pipeline. The essence of our work lies in utilizing multimodal large models as agents for image restoration, which has shown promising results across various conditions. Our experiments show that the RestoreAgent performs exceptionally well in most complex scenarios, thus proving its viability.\n\nIn addition, our approach allows for flexible integration of different tools. If the model encounters degradations beyond the predefined scope, additional tools can be seamlessly incorporated. This adaptability ensures that our pipeline remains robust even when faced with unexpected degradation types.\nEssentially, the generalization issue is more related to the specific image restoration models (tools) used within the pipeline, rather than the pipeline itself. \n\nLastly, one of our future research directions is to significantly expand the scope of restoration tasks and incorporate a greater variety of restoration models. By doing so, we aim to enhance the system's capability to handle real-world complex degradations more effectively.\nThis expansion will involve integrating models trained on a broader spectrum of degradation types and more generalizable restoration models, ensuring that the pipeline can generalize better to unseen conditions.\n\n`Q4`: **The explanation of \"ranking\" and \"balanced\" in Table 1 is still unclear. The authors should clarify the definitions of these terms.**\n\n\nWe have added the following clarification to the paper:\n\n\n**The explanation of \"ranking\":**\n\nWe've clarified the ranking system:\n\nFor individual test sets:    \nX/Y format: X is the average ranking, Y is total combinations.    \nExample: 6.4/64 means average 6.4th best out of 64 options.   \n\nFor overall average:    \nWe use percentages due to varying combination counts of diffrent restoration tasks.   \nProcess: Calculate percentage ranking per set, then average.    \nExample: 12.9% means top 12.9% on average across datasets.\n\n**The explanation of \"balanced\":**\n\nWe have also provided a detailed explanation of \"balanced\" in Section 4.1 of our manuscript, specifically within the Scoring Function subsection. Balanced refers to the sum of the PSNR, SSIM, LPIPS, and DISTS metrics after each has been standardized. To ensure clarity, we have also added additional explanatory notes directly in Table 1. We hope these changes address your concerns.\n\n`Q5`: **Show more visual comparisons.**\n\nThank you for your valuable feedback. We have updated our manuscript to include additional visual comparisons of the RestoreAgent. These enhancements provide a clearer demonstration of the effectiveness of our proposed method. Please refer to the updated **PDF file** we have uploaded, which contains these new visual results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KiPpDc5CQ6", "reviewer": "Reviewer_4Dbc", "summary": "For real-world images corrupted by multiple simultaneous degradations, this paper first analyzes the limitations of using all-in-one restoration models and various task-specific models. The authors then introduce RestoreAgent, which automatically identifies the types of degradation in a degraded image, determines the sequence of restoration tasks, and selects suitable models from the model pool. RestoreAgent presents an automated restoration pipeline that requires only an input image and a general human instruction, without any prior knowledge of the involved degradation tasks or manually predefined task sequences.", "questions": "Addressing concerns in the Weaknesses with thorough explanations and additional experiments would significantly enhance my confidence in this work. A satisfactory response to these points may lead to a reconsideration of the current evaluation.", "limitations": "The manuscript includes the checklist guidelines.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper comprehensively analyzes the challenges and limitations of employing all-in-one models and multiple task-specific expert models with fixed or random task sequences, as well as fixed or random models for each task.\n2. The authors evaluate various configurations of RestoreAgent using diverse objective image quality metrics (PSNR, SSIM, LPIPS, DISTS, and their combinations), all of which outperform the human expert model on the corresponding metric. \n3. RestoreAgent exhibits the scalability by extending to new tasks and models with minimal computational resource. \n4. The presentation, including writing, analysis, and visualization, is clear and easy to follow.", "weaknesses": "1. Incomplete descriptions about data construction. \n\n- Authors randomly select up to four types of degradation from a degradation set (noise, blur, JPEG, rain, haze, and low-light) to construct paired training data. According to data synthesis strategies in [1,2], JPEG compression is typically performed after noise and blur, and in the final order. Is the degradation order of JPEG compression in this paper the same? If not, the authors should discuss the reasonableness of random sampling.\n\n- What are the components of 23k paired data? One degraded image for each high-quality image or many degraded versions for each high-quality image? \n\n- What is the configuration in ablation studies about training data amount? Simultaneously scaling up low & high-quality images or synthesizing more low-quality images for each high-quality image? If it’s the former, will increasing the number of degraded images while keeping the number of high-quality images unchanged improve performance?\n\n2. Inference time for input images with diverse resolution.\n- The authors are suggested to report the running time for input images of various resolutions. This should include the total time, the running time for the RestoreAgent, and the running time for the subsequent restoration models. The reviewer is curious whether the agent's response time exceeds that of the restoration models when processing high-resolution images, such as those with 4K resolution.\n\n3. Scalability for new tasks and models. \n- Section 4.5 demonstrates that the proposed RestoreAgent can extend to new tasks and models in just half an hour, surpassing human expert-level performance on the new task. However, it is unclear whether adaptation to the new task results in performance degradation on prior tasks, similar to the catastrophic forgetting problem in continual learning. The authors are encouraged to report the performance of the fine-tuned model on the previous tasks to address this concern.\n\n[1] Wang X, Xie L, Dong C, et al. Real-esrgan: Training real-world blind super-resolution with pure synthetic data[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2021: 1905-1914.\n\n[2] Zhang K, Liang J, Van Gool L, et al. Designing a practical degradation model for deep blind image super-resolution[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021: 4791-4800.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "`Q1`: **Degradation order of JPEG compression.**\n\nThank you for bringing up this important point regarding the degradation order. In our study, the order of JPEG compression is not fixed and is entirely random, unlike the sequence suggested in references [1,2]. The strategy of placing JPEG compression at the end, as mentioned by the reviewer, represents a subset of the data we constructed. Therefore, our dataset inherently includes this configuration while also supporting a broader range of degradation combinations.\n\nOur primary objective is to validate the robustness and feasibility of our proposed pipeline. By incorporating a wider variety of degradation sequences, we can more effectively demonstrate the generalizability and effectiveness of our method across different scenarios. Moreover, the specific degradation order can be flexibly defined and adjusted based on user requirements, ensuring adaptability to various applications.\n\n\n`Q2`: **What are the components of 23k paired data? One degraded image for each high-quality image or many degraded versions for each high-quality image?**\n\n\nOur dataset comprises one degraded image for each high-quality image. This means that each image pair consists of a unique degraded image and its corresponding high-quality counterpart, ensuring diverse image backgrounds. Specifically, we utilized images from the DIV2K and Flickr8K datasets, as well as the RESIDE dataset to construct images containing haze. Additionally, the LOLv2 Real dataset was employed to create low-light images.\n\n\n\n`Q3`: **What is the configuration in ablation studies about training data amount?**\n\n\nAs previously mentioned, each high-quality image in our dataset corresponds to a single degraded image. Therefore, when we scale up or down the training data, both low-quality and high-quality images increase or decrease simultaneously.\n\nIncreasing the number of degraded images while keeping the number of high-quality images unchanged would indeed improve performance. As discussed in our manuscript, image degradation is complex, and even minor variations can lead to different degradation characteristics, requiring varied handling strategies. Thus, adding more degraded images extends the range of degradations the model is exposed to, enhancing its ability to manage a broader spectrum of degradation scenarios.\n\n\n\n`Q4`: **running time**\n\nThe RestoreAgent in our approach is constructed based on a multimodal large language model. Specifically, the computational time for the RestoreAgent is approximately 0.4 seconds. Therefore, the additional running time introduced by the RestoreAgent is minimal. In our experiments, the image patch size used is 512x512 pixels. In practice, the MLLM resizes all inputs to a fixed size, such as 224x224. For processing 4K resolution images, there are several strategies to consider:      \n(1) Single Patch Representation: Often, image degradation across different regions is uniform. In such cases, a single patch can represent the overall degradation of a 4K image.     \n(2) Patch-wise Processing: The image can be divided into smaller blocks, with each block represented by a patch.       \n(3) Resizing: The 4K image can be resized appropriately before making decisions using the multimodal model.    \nGiven these strategies, decision-making for 4K images does not significantly increase the required time.\n\nRegarding the running time of the restoration models, it is important to note that our pipeline utilizes restoration models based on user preferences. Consequently, we can select models of varying sizes and running times according to specific needs. Therefore, concerns about the running time of the restoration models may not be necessary as the flexibility in our pipeline allows for optimization and customization based on user requirements.\n\n\n\n\n`Q5`: **it is unclear whether adaptation to the new task results in performance degradation on prior tasks, similar to the catastrophic forgetting problem in continual learning.**\n\n\nThank you for pointing out this important concern. As shown in Table 1, we report the performance of the fine-tuned model on previous tasks after adaptation. The changes in performance are minimal. We believe that these minor variations in performance are primarily attributable to the **inherent randomness in the training process**. We believe that with careful tuning, the model's performance can be further enhanced.\n\nThis demonstrates a crucial point: the current capabilities of multimodal large language models are more than sufficient to handle this task. Their robust reasoning abilities support the management of scenarios far more complex than those presented in our experiments.\n\nWe believe one reason for this is that the newly added degradation task does not conflict with the existing tasks; in some aspects, it is very similar. For example, in the combination of snow + noise + JPEG, the original model already has extensive knowledge of handling noise + JPEG, which can be easily applied to the new combination. \n\nWe are confident that in more complex scenarios (with more restoration tasks and more restoration models), RestoreAgent will demonstrate its significant advantages compared with human. In such complex scenarios, human decision-making will be less effective since the space is much bigger, whereas the powerful capabilities of learning-based multimodal large language models will exhibit even greater superiority.\n\n\n\n\n\n\n|       method   | Average Ranking on All Previous Datasets | Ranking on the New Task |\n|---|---|---|\n| Human Expert            | 19.5%    | 21.2 / 64 |\n| RestoreAgent             | 12.9%    | -    |\n| RestoreAgent + New Task  | 13.1%  | 4.5 / 64  |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear AC and all reviewers,

We sincerely appreciate your time and efforts in reviewing our paper. We are glad to find that reviewers recognized the following merits of our work:

- **Innovative contribution and strong motivation [DJ9b, NhBR, Ux72, 4Dbc]**: 
The proposed RestoreAgent addresses the challenges of image restoration by effectively demonstrating the importance of task execution order and model selection in multi-task scenarios. This novel approach opens up new research directions in the field.

- **Impressive performance [DJ9b, NhBR, Ux72, 4Dbc]**: Experimental results indicate that RestoreAgent's decision-making capabilities in handling complex degradations surpass those of human experts and existing all-in-one models. 


- **Well-written and clear [DJ9b, Ux72, 4Dbc]**: The paper is generally well written and clear to understand, with comprehensive analysis and visualization.



We also thank all reviewers for their insightful and constructive suggestions, which help further improve our paper. In addition to the pointwise responses below, we summarize the major revision in the rebuttal according to the reviewers’ suggestions:


- **Detailed data construction process [DJ9b, 4Dbc]**:
We have added detailed explanations on the construction methods for the training and testing datasets.


- **Discussion on Limitations and Future Directions [DJ9b, NhBR, Ux72]**:
We have expanded the discussion on the potential challenges and future research directions, including the integration of new degradation types, generalization to unseen degradations, and the limitations of existing frameworks.


- **Manuscript update [DJ9b, Ux72, 4Dbc, NhBR]**: We have included more visual comparisons, inference time, experimental results, and discussions in the main paper and Appendix, Furthermore, we have clarified descriptions that were previously unclear.


We hope our pointwise responses below can clarify all reviewers' confusion and address the raised concerns. We thank all reviewers' efforts and time again.

Best,
Authors

Meta-review: This paper explores the multimodal large language models for image restoration. All reviewers are in favor of this manuscript. The AC agrees with the recommendations of reviewers.
Decision: Accept (poster)