RestoreAgent: Autonomous Image Restoration Agent via Multimodal Large Language Models

Authors: haoyu  chen
wenbo  li
jinjin  gu
jingjing  ren
sixiang  chen
tian  ye
renjing  pei
kaiwen  zhou
fenglong  song
lei  zhu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 15:41:26

Abstract: Natural images captured by mobile devices often suffer from multiple types of degradation, such as noise, blur, and low light. Traditional image restoration methods require manual selection of specific tasks, algorithms, and execution sequences, which is time-consuming and may yield suboptimal results. All-in-one models, though capable of handling multiple tasks, typically support only a limited range and often produce overly smooth, low-fidelity outcomes due to their broad data distribution fitting. To address these challenges, we first define a new pipeline for restoring images with multiple degradations, and then introduce RestoreAgent, an intelligent image restoration system leveraging multimodal large language models. RestoreAgent autonomously assesses the type and extent of degradation in input images and performs restoration through (1) determining the appropriate restoration tasks, (2) optimizing the task sequence, (3) selecting the most suitable models, and (4) executing the restoration. Experimental results demonstrate the superior performance of RestoreAgent in handling complex degradation, surpassing human experts. Furthermore, the system’s modular design facilitates the fast integration of new tasks and models.

Main paper: 
 introduction: 
 Image restoration, a classical research area in computer vision, focuses on recovering high-quality images from degraded observations. Traditional methods are usually tailored to specific tasks like denoising [55, 61, 49, 29, 30, 12, 3] , super-resolution [56, 34, 53, 4, 45, 47, 48] , and deblurring [28, 22, 51, 32, 44, 17] . However, real-world images often suffer from multiple simultaneous degradations. For example, a low-quality image may exhibit noise, blur, and rain concurrently. There may exist complex interactions and dependencies among different degradation phenomena, and each degradation may require distinct handling methods. The combination and sequence of these methods are crucial for the final restoration outcome. Recent advancements in the field have been driven by leveraging expert knowledge and developing all-in-one models. To provide a thorough understanding of this field and clarify our motivation, we present a detailed analysis below.All-in-one models [38, 31, 24, 40, 33, 14, 27, 37, 1, 25] seek to use a single framework to handle multiple degradations simultaneously. By training on multi-task datasets, these models learn to manage various restoration tasks. However, several limitations continue to impede the practicality of these models in complex real-world scenarios:Restricted task scope. All-in-one models often struggle to process degradations outside of their training data. Even for the same type of degradation, as shown in Figure 2 a1, these models may have difficulty effectively processing data if the degradation distribution varies between the training and testing sets. Given that existing models only cover a limited number of tasks, employing specialized single-task restoration models is often more flexible and effective. Compromised performance. All-in-one models often face trade-offs between generalization and restoration accuracy, as shown in Figure 1 . While these models offer improved generalization across a broader range of degradation levels, their performance at specific levels may be compromised. Additionally, because they must handle multiple tasks with largely disparate degradation patterns, the performance for individual tasks may fall short, resulting in overly smoothed outputs. As illustrated in Figure 2 a2, single-task models typically outperform all-in-one models in most scenarios.All-in-one models can, in fact, be integrated into an agent system comprising multiple models, thereby going beyond a single solution. Often, using task-specific models customized for particular degradations and then integrating them with an all-in-one model yields improved performance, as shown by the two examples in Figure 2 a3. This hybrid approach maintains the adaptability of all-in-one models while leveraging the strengths of specialized models.An alternative approach to using all-in-one models, which struggle to effectively address various types of degradation, is to combine several specialized task-specific models, each focusing on a specific degradation type. This modular strategy allows for a more targeted and efficient handling of the different degradations present in the input images. Superior results can be achieved because these specialized models excel in their respective areas.Current methods [50, 24, 14] typically detect the types of degradation in an image and apply the appropriate restoration models in a predetermined order, or manually selected by experts, or chosen at random. Nevertheless, there is a significant drawback to this approach: the processing order has a major impact on the final performance. A predetermined order, even if established by human experts, is not ideal and might fail to successfully restore the image, as demonstrated in Figure 2 b . Two primary causes can be identified for this.First, applying one restoration method can alter other degradation patterns, rendering the following restoration models ineffective. For example, in an image with haze and rain, if haze is performed first (Figure 2 b ), the dehazing model may address the blur but alter the rain distribution, thereby reducing the effectiveness of the deraining model.Second, removing some degradations can be challenging if other degradations have not been addressed first. A common example is the enhancement of low-light images, which often requires denoising as a pre-processing step. Without prior denoising, the results of low-light enhancement are likely to be subpar. In Figure 2 b , we can observe that without prior denoising and deraining, the performance of the dehazing model is significantly compromised. In light of these findings, accurate identification of degradation patterns or careful testing of various task execution sequences is necessary for high-quality restoration. However, the search space grows significantly with the number of tasks. For example, there are 24 possible execution orders for 4 degradation types. Moreover, the number of permutations increases drastically when multiple models are available for a given task, leading to a significant rise in computational complexity.In some scenarios, the system may opt to use a single model for a specific task or randomly select a model from a pool of available options [50] . However, this approach has significant drawbacks. Image restoration is a rapidly evolving field with various models tailored for a specific task, each with unique capabilities and areas of expertise for managing specific scenarios. Using a fixed model or randomly selecting from a pool of models to process complex degradations can lead to suboptimal results. As illustrated in Figure 2 c and Figure 1 a, different denoising models excel at different noise levels. Choosing the right model is crucial for achieving the best result.Manually selecting the best model is impractical due to the numerous combinations of task execution orders and available models. For example, with 3 degradation types and 3 models per type, there are 162 possible combinations. Evaluating these permutations is time-consuming and labor-intensive. Consequently, we often rely on one or two experienced-based solutions, which may not achieve the desired restoration effect.In response to the aforementioned challenges, we propose RestoreAgent, an autonomous and intelligent image restoration system based on a multimodal large language model (MLLM). The MLLM's exposure to vast and diverse data endows it with superior generalization capabilities and has showcased remarkable performance in visual understanding and logical reasoning [46, 35, 18, 39, 43, 6, 62] . Furthermore, its flexibility facilitates the quick addition of new tasks, the definition of desired output formats, and easier human interaction. Our framework offers the following functionalities:(1) Degradation Type Identification. RestoreAgent automatically identifies the types of degradation present in an input image and determines the corresponding restoration tasks required.(2) Adaptive Restoration Sequence. RestoreAgent goes beyond the constraints of predefined, human-specified model execution orders by dynamically evaluating the individual properties of each input image to decide the best sequence for utilizing the restoration models, thereby enhancing the overall efficiency of the image restoration procedure.(3) Optimal Model Selection. Based on the specific degradation patterns in the input image, RestoreAgent dynamically selects the most appropriate model from the available pool for each restoration task, ensuring optimal performance. (4) Automated Execution. Once the restoration sequence and model selection are determined, RestoreAgent autonomously executes the entire restoration pipeline without the need for manual intervention.To this end, we start by defining the multi-degradation task and constructing a training dataset. This dataset includes paired degraded images (with one or more degradation types) and their ground truth (only for evaluation), along with the optimal task execution sequence and best model choice based on user-preferred goals. We then fine-tune MLLM to enable RestoreAgent to autonomously make task decisions and determine the optimal processing sequence and models. Experiments show that RestoreAgent's decision-making capabilities significantly outperform existing methods and human experts, achieving superior performance in recovering multi-degradation images. Notably, our method can quickly adapt to unseen tasks and models.As shown in Table 7 , for the tasks of denoising and deJPEG, as well as deraining, we employ Restormer [58] as our model. For dehazing, we utilize RIDCP [54] , while for motion deblurring, we use DeblurGANv2 [28] . For desnowing, we implement Snowformer [10] . For low-light enhancement, we use Retinexformer [2] . It is noteworthy that the models we are using are not the latest state-of-theart models, indicating that there is significant room for improvement in our models.A crucial consideration in image restoration is the limited generalization capability of many current models, which often fail to maintain performance when faced with subtle variations in image degradation. This necessitates the selection of more robust models. For example, in our approach to denoising, we enhance model generalization by incorporating not only Gaussian noise but also random blur and other noise types during training. This strategy enables the model to address more complex degradation scenarios effectively. Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: Yes, the main claims made in the abstract and introduction accurately reflect the paper's contributions Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
single-task image restoration: 
 In the field of single-task image restoration, numerous methods have focused on addressing specific types of image degradation. In denoising, models like DnCNN [59] and RNAN [63] have demonstrated significant effectiveness, among others. In deblurring, algorithms like DeblurGAN [28] and MIMO-UNet [13] and others stand out. For reducing JPEG artifacts, methods such as DCSC [19] and FBCNN [23] are particularly well-suited. Additionally, there are specialized methods for restoration under adverse weather conditions, including dehazing [52, 41] , deraining [11, 7] , and desnowing [8, 9, 5] . Each task often requires a specialized approach, leading to highly optimized algorithms that achieve sota performance for their specific targets compared to universal approaches.Recent research has explored the development of All-in-One models that attempt to handle multiple degradation types simultaneously within a single framework. This kind of methods are trained to recognize and correct various forms of degradation concurrently. AirNet [31] featuring the contrastivebased degraded encoder and degradation-guided all-in-one restoration network. ADMS [38] uses adaptive filters to efficiently restore images with unknown degradations. TAPE [36] embeds a taskagnostic prior into a transformer, utilizing a two-stage process of pre-training and fine-tuning to enhance image restoration. PromptIR [40] and PIP [33] both utilize uniquely designed prompts to guide their networks. MiOIR [27] employs sequential and prompt learning strategies, which guide the network to incrementally learn individual IR tasks in a sequential manner. MPerceiver [1] employs a multimodal prompt learning approach, utilizing Stable Diffusion priors to achieve high-fidelity all-in-one image restoration.Another research direction focuses on more intelligent image restoration systems. One class of such methods employs a toolbox approach to address image degradation separately. RL-Restore [57] prepares a toolbox consisting of small-scale convolutional networks, each specialized in different tasks. The system then learns a policy to select appropriate tools from the toolbox to progressively restore the quality of a corrupted image. However, RL-Restore supports only three types of degradation: blur, noise, and JPEG compression, which constrains its application scenarios and prevents it from utilizing new state-of-the-art models. Clarity ChatGPT [50] combines the conversational intelligence of ChatGPT with multiple image restoration methods. It automatically detects types of image degradation and selects appropriate methods to restore images. Conversely, Clarity ChatGPT identifies the presence of degradation but lacks research and design on the execution order of tasks and the optimal model selection for specific degradations in the input image.Another class involves all-in-one approaches with degradation-aware guidance. InstructIR [14] pioneers a novel approach by utilizing human-written instructions to guide the recovery from various types of degradation. AutoDIR [24] automatically detect and restore images with multiple unknown degradations. LLMRA [25] generates text descriptions and encodes them as context embeddings with degradation information, and integrates these context embeddings into the restoration network. DA-CLIP [37] presents a degradation-aware vision-language model that guides the model to learn high-fidelity image reconstruction. For these all-in-one restoration assistant methods, inherent limitations exist in the practical applications of all-in-one models.How to overcome these limitations, fully leverage the wide array of state-of-the-art models for different tasks available on the market, and determine the optimal execution sequence of image restoration tasks and the most suitable model for specific degradation pattern remain unexplored. This gap presents a significant opportunity for future research in intelligent image restoration systems.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We discuss the limitations of the work in the supplemental material section. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
restoreagent: 
 In this section, we introduce RestoreAgent, an advanced image restoration agent designed to find the optimal model and execution sequence from a model pool to process images containing multiple degradations. RestoreAgent is built upon a state-of-the-art multimodal large language model, which possesses remarkable reasoning, generalization, and cross-modal understanding capabilities. By leveraging the model's ability to draw insights from vast amounts of multimodal data, establish connections between visual and textual information, and apply that knowledge to new contexts, RestoreAgent can effectively analyze complex image degradation scenarios, infer the most suitable restoration techniques, and generate optimal pipelines that combine the strengths of various specialized models. As a result, RestoreAgent consistently produces high-quality results.In Section 3.1, we first define the problem of identifying the most effective combination and order of models from a given pool to restore images affected by various types of degradation. Next, in Section 3.2.2, we describe the process of constructing the training data for the RestoreAgent. The training data consists of paired samples, each containing a degraded image and its corresponding optimal restoration pipeline. Finally, we detail the training process of RestoreAgent, which involves fine-tuning the Llava-Llama3-8b model using the constructed training data in Section 3.2. By learning from these examples, RestoreAgent acquires the ability to analyze degraded images and generate optimal restoration pipelines based on the available model pool.We consider a comprehensive set of degradation types, denoted as D = {d 1 , d 2 , . . . , d n }, where each d i represents a specific type of image degradation such as noise, JPEG artifacts, blur, rain streaks, fog, and low light conditions. For each degradation type d i , we tailor a model library M di , comprising models {M 1 di , M 2 di , . . .}. Each model M j di is specifically trained to mitigate the effects of degradation d i . The problem is formally defined as follows:Input: A degraded image I subjected to various degradation types D. A model library {M d1 , M d2 , . . . , M dn } tailored for processing D. The user-provided scoring function S for evaluating the image restoration process. Step 1Step 2Step 3Step 1DeNosing Noise level: MidStep 2Step 3DeJPEG JPEG level: HighStep 4 By tackling this problem, we strive to identify the optimal combination of restoration sequence and model selections, ultimately enhancing the quality of images affected by multiple degradations in real-world settings, and thus providing a more effective and efficient solution for complex image restoration tasks.We introduce an advanced image restoration agent, dubbed RestoreAgent, implemented using the state-of-the-art multimodal model Llava-Llama3-8b [46] . LoRA [21] is utilized to fine-tune both the vision and language modules. As shown in Figure 3 , given a degraded input image, RestoreAgent can provide the best decisions, including which image restoration tasks need to be performed, the order of their execution, and which model is most suitable for each task. The model's input consists of a degraded image and the prompt such as User: How to enhance the quality of this image? [Execution history: ...]. In response, RestoreAgent generates an output sequence representing the optimal restoration pipeline, comprising a series of tasks, each associated with a specific model best suited to address particular degradation patterns. In our implementation, the output template is defined as: Agent:1.<task name><model name>. 2.<task name><model name>. 3. ..., ensuring interpretability and actionability.RestoreAgent also supports an iterative step-wise decision-making process, reevaluating the state of the image after each restoration step. During this reassessment, the execution history is provided, offering valuable context for decision-making. This allows for real-time strategy adjustments based on cumulative effects and past actions. The system also features a rollback capability, enabling it to revert to a previous state if undesirable results are detected. This combination of iterative evaluation with historical context and rollback allows for finer control over the restoration process, facilitating mid-course corrections. To fully leverage the potential of multimodal large models, we construct a substantial dataset of paired training samples. The process begins with applying various types of degradation to an image. Subsequently, we determine the optimal restoration pipeline using model tools for processing. For each image undergoing multiple degradations, a comprehensive search is conducted to identify the best restoration pipeline, as shown in Figure 3 . This involves generating all possible permutations of task execution sequences and model combinations, applying each pipeline to the degraded image, and assessing the quality of the restored outputs using a scoring function S(I, σ). By comparing the scores of all permutations, the pipeline with the highest score is selected as the optimal processing strategy σ for the given image. Users can choose from various image quality assessment methods as the scoring function, customizing the evaluation process to their specific needs. Additional details regarding dataset construction are provided in the supplementary materials.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes]• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experimental settings: 
 Scoring function. To construct a comprehensive evaluation system, we integrate multiple diverse metrics. Specifically, we first standardized each individual metric separately and then summed the standardized results. This process can be described as follows. Let X i represent the i-th metric. We standardize each metric by calculating its z-score:Z i = Xi-µi σiwhere µ i is the mean and σ i is the standard deviation of the i-th metric. After standardizing all metrics, we aggregate the standardized scores to form the comprehensive evaluation score S: S = n i=1 Z i where n is the total number of metrics. This method ensures that each metric contributes equally to the final evaluation, regardless of its original scale. Follow [26, 20] , evaluation metrics primarily include PSNR, SSIM, LPIPS [60] , and DISTS [16] . These metrics are widely recognized for their ability to comprehensively reflect the outcomes of image restoration. We also provided the results of models trained on individual metrics. • For all images, using the human expert's predefined order and models, assuming accurate task type determination. • Human expert personally crafting a solution for each image, determining the task sequence and models for each task. This method represents the most common scenario in real-world applications, where a human decides how to restore an image on a case-by-case basis. The human expert in this study has more than five years of research experience in low-level vision. Before crafting solutions, the expert familiarized themselves with each task degradation and the corresponding model's actual performance to ensure they could provide the best human-level solution.Results. Table 1 reports the average metric results of our RestoreAgent and other decision-making approaches on seven different degradation combination datasets. As shown in Table 1 , using a random order and model selection ranked lowest, achieving only a 34.7% performance rating among all possible strategies. By setting predefined sequences and models for image processing by human experts, traditional methods rank in the top 22.1% of all possible strategies. This demonstrates that experience-based predefined rules often used in practical applications are more effective than completely random strategies. Human experts making specific decisions for each test image can further improve upon predefined rules, increasing the ranking from 22.1% to 19.5%. This proves that using the same predefined rules to process all images is not optimal, while individualized decision-making for specific images can better enhance the effects. Then, the superior performance of our RestoreAgent (12.9%) over expert-based customization (19.5%) shows that automated and data-driven decision-making in our method clearly outperforms traditional and experience-based human expert judgments. This is because human experts from their own experience can not make precise judgments about the advantageous scenarios of all models and the order of task execution, especially when numerous tasks and models are involved.To demonstrate the limitations of all-in-one methods in handling multi-degraded images, we compared our approach with various types of all-in-one models. To ensure a fair comparison, tests were only conducted on degradation types and datasets that these all-in-one models were trained to support. Moreover, we repeatedly run the all-in-one model as many times as the number of degradation types of the test images to fully leverage its capabilities, thus ensuring a fair comparison. The results are shown in Figure 5 and Table 2 . Our RestoreAgent achieved a significant leading advantage across all tested degradation combinations. For the degradation types commonly encountered in traditional image super-resolution, such as noise and JPEG compression artifacts, our approach significantly outperformed established methods like Real-ESRGAN and the sota SR method, StableSR. For a broader range of degradation types, our method retained a considerable advantage. Among these all-in-one approaches, InstructIR and AutoDIR face two major issues: manually predetermined or randomly decided execution order, and using single model to address all types of degradations. These limitations often result in incomplete restoration, as depicted in Figure 5 . These results underscore the limitations of all-in-one models, validating our initial hypothesis.As discussed in the method, our proposed method can adapt to various optimization objectives, enabling the decision-making results tailored to specific target criteria. To verify it, we present the results of models trained with different individual metrics as the optimization objective in Table 3 .The results indicate that when a model is trained with a single metric, the performance of the corresponding metric can be significantly improved compared to the balanced model. This showcases the adaptability and effectiveness of our method in catering to specific optimization goals.The proposed RestoreAgent demonstrates remarkable adaptability and extensibility, allowing for swift fine-tuning to accommodate new task types and incorporate additional models. To validate this capability, we introduced a new task, desnowing, along with its corresponding model. Building upon the RestoreAgent previously trained on six tasks, we performed rapid fine-tuning by integrating the desnowing task. Within thirty minutes, our model achieved exceptional performance on the new task type. As shown in Table 5 , our approach quickly surpassed human expert-level proficiency on the new task and model. This validation underscores the practical value of our method, allowing efficient integration of additional tasks with minimal resource expenditure.As mentioned in Section 3.2, RestoreAgent supports iterative decision-making with historical context awareness. It dynamically adjusts strategies during image restoration, reassessing image state after each step and rolling back if needed. As demonstrated in Table 6 , we conducted experiments on a complex dataset incorporating four distinct types of image degradation: Motion Blur, Rain, Noise, and JPEG compression. Results show that while the single prediction approach performs well, iterative step-wise replanning further enhances restoration outcomes, allowing for precise control and mid-course corrections. The initial decision's performance is already strong, step-wise replanning thus offers incremental yet valuable improvements to an already effective process. 
conclusion: 
 Our research identifies key factors in processing multi-degraded images, including task execution order, model selection, and the limitations of the all-in-one approach. Based on these insights, we present RestoreAgent, an agent model that makes intelligent processing decisions based on image degradation and user objectives. Experiments show that our pipeline outperforms the all-in-one method and surpasses human experts in decision-making performance.A Appendix / supplemental material A. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: See Section 4.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The creators or original owners of assets used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: How to enhance the quality of this image? Execution history: None. (1) Once we obtain a degraded image along with its corresponding optimal decision results, we can construct the primary part of our dataset. This part consists of degraded images in their original, unprocessed state. For these inputs, the RestoreAgent receives a prompt: "How to enhance the quality of this image? Execution history: None." This scenario trains the model to formulate comprehensive enhancement strategies from scratch, encompassing multiple restoration steps. This part of the data exceeds 23k pairs.(2) To foster dynamic decision-making capabilities, we introduce a second category of training instances. Here, the input comprises partially processed images (e.g., after denoising) along with their execution history. This approach enables the RestoreAgent to adapt its predictions based on intermediate results, promoting a more flexible and context-aware enhancement process.(3) The third scenario addresses situations where the model identifies suboptimal outcomes from a particular enhancement step. In such cases, the RestoreAgent is trained to output "Rollback," indicating the need to revert to a previous state and recalibrate its strategy. This feature is crucial for maintaining high-quality outputs and avoiding the propagation of errors through the enhancement pipeline. We select from erroneous paths (the decisions with the worst metric results) to construct this portion of the paired data, as the worst paths require a rollback.(4) Following a rollback event, our fourth data category provides the model with information about the specific step that triggered the rollback. This guidance is essential in preventing the model from repeating ineffective procedures, thus streamlining the enhancement process and improving efficiency.(5) The final scenario in our training regime represents fully processed images that require no further enhancement. In these instances, the RestoreAgent is trained to recognize optimal image quality and output "Stop", effectively terminating the enhancement sequence.By incorporating these diverse scenarios, we aim to develop a highly adaptive and efficient image restoration system capable of addressing a wide array of real-world image degradation challenges. For computational efficiency, unless specifically mentioned otherwise, our default experiments are based on a single planning for the initial image rather than using iterative step-wise replanning.The specific details of our test set are presented in Table 8 , which demonstrates our construction of various combinations of degradation types. Each image in the set contains a minimum of one and a maximum of four types of degradation, with the entire set comprising 200 images.In this study, we incorporate the CLIP pre-trained Vision Transformer (ViT-L/14) [42] as the image encoder to convert input images into visual tokens. For the language model, we utilize the Llama3-7B [46] . Despite their capabilities, pre-trained LLMs fail to provide accurate responses without dataset-specific fine-tuning. To address this, we adopt LoRA [21] , a fine-tuning technique that efficiently modifies a limited number of parameters within the model. Following [21] , we apply LoRA to adjust the projection layers in all self-attention modules of both the vision encoder and the LLM, thereby generating our RestoreAgent. We employ the Xtuner framework [15] to facilitate the training process. For our experimental setup, we configure the LoRA rank to 16. The RestoreAgent undergoes training across ten epochs on 4 NVIDIA RTX A100 GPUs, with a batch size of 32. We employ the Adam optimizer and a learning rate of 0.00002. The total duration of the training process approximates ten hours.Figure 4 and 8 illustrate RestoreAgent's decision-making process and the importance of model selection, respectively. Figure 6 further demonstrates why human decision-making often yields suboptimal results in image restoration tasks. Figure 8a exemplifies the nuanced challenges in degradation assessment. Despite identical backgrounds and degradation types, subtle variations in degradation features lead to divergent optimal restoration sequences. For instance, the sequence Step 2. Large noise level 26.53 0.1999Step 1.Step 2. Step 2. "DeNoising → DeRaining → DeJPEG → DeHazing" proves effective for the upper row images but fails for the lower row. Conversely, the sequence "DeRaining → DeNoising → DeJPEG → DeHazing" yields optimal results for the lower row but is suboptimal for the upper row. This dichotomy underscores the difficulty human experts face in discerning minute degradation differences, thereby compromising effective decision-making.The complexity of optimal restoration sequencing is further accentuated in Figure 8b . Here, we demonstrate scenarios where only one specific sequence among numerous permutations yields satisfactory results. This observation highlights the formidable challenge posed to human decisionmakers in identifying the singular effective restoration pathway amidst a multitude of possibilities.These findings collectively emphasize the superiority of automated, data-driven approaches in navigating the intricate landscape of image restoration. The RestoreAgent's ability to discern and adapt to subtle degradation variations surpasses human capabilities, particularly in scenarios where the optimal restoration sequence is non-intuitive and highly specific to individual image characteristics.Comparison with assistants with all-in-one models. Assistants that employ unified models, such as LLMRA [25] and AutoDIR [24] , attempt to handle diverse tasks, degradation patterns, and intensities using a single model. As discussed in Section 1.1, these all-in-one models face significant challenges, including restricted task scope and compromised performance, which greatly limit their effectiveness in real-world applications. Conversely, our method leverages various model experts to address specific situations, the upper bound of our pipeline is determined by the latest SOTA models, allowing us to maximally leverage the latest advancements in the field without being constrained by the limitations of an all-in-one model. Furthermore, as detailed in Section 4.4, our RestoreAgent exhibits high efficiency in incorporating new tasks and models, showcasing greater flexibility.Comparison with assistants with tool use. Image restoration assistants that utilize tool libraries, such as Clarity ChatGPT [50] and RL-Restore [57] . Clarity ChatGPT merely identifies the degradation in images, follows a rigid execution strategy, lacking the ability to make dynamic decisions on task execution order and select the best model. As discussed in Section 1.2.1 and 1.2.2, an inappropriate task execution sequence and model selection can leading to lower performance in subsequent operations. RL-Restore, on the other hand, uses reinforcement learning for sequence decision-making and model selection. However, its task definition is overly simplistic, limited to three degradation types (noise, blur, and JPEG) with a narrow degradation range. Also, training reinforcement learningbased methods is more challenging and may result in lower precision, making it difficult to achieve high performance in complex and varied scenarios. Conversely, the integration of a comprehensive task definition with advanced multimodal models allows our method to effectively manage various degradation types and intensities. This adaptability enhances its efficacy, positioning our approach as a promising solution for image restoration tasks.Training data amount. To investigate the effect of training data volume on our method, we evaluated the performance of the RestoreAgent model trained on datasets consisting of 7,000, 14,000, and 23,000 data pairs; see Table 4 The results demonstrate that even with the smallest dataset of 7k pairs, our RestoreAgent achieves superior performance over both random and human expert benchmarks. More notably, the training data volume increasing from 7k to 14k incurs a substantial improvement with the ranking percentage decreasing from 16.2% to 13.6%. With 23k data pairs, the performance further improves, achieving a ranking percentage of 12.9%. This indicates that using more training data boosts our RestoreAgent model. These findings emphasize the robustness of our approach, demonstrating that while larger datasets do enhance performance, our model already provides significant benefits even with relatively smaller datasets.The primary limitation of our study is the confined scope of models and tasks examined. While our research offers valuable insights into RestoreAgent's performance across several degradation scenarios, it does not encompass the full spectrum of restoration models or image degradation tasks currently available.Another limitation pertains to the limited generalization capability of current image restoration models. These models often exhibit a notable decrease in performance or fail to respond adequately when faced with even minor variations in image degradation patterns. This limitation greatly narrows our selection of model tools, requiring us to choose more robust and generalizable model tools. The challenge underscores a critical need in the field of image restoration: future models must go beyond simply overfitting training data. Rather, they should exhibit better generalization and increased efficiency in handling real-world degradation cases.Our future work will focus on significantly expanding the range of image restoration models incorporated into our multimodal large language model. This expansion aims to enhance RestoreAgent's capabilities across a broader scope of restoration tasks and degradation types. By integrating a more diverse set of state-of-the-art models, we seek to create a more comprehensive and versatile restoration framework.Justification: We conform, in every respect, to the NeurIPS Code of Ethics as outlined at the provided https://neurips.cc/public/EthicsGuidelines.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Justification: There is no societal impact of the work performed.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).DeNosingDeRainingDeHazing 


Decision: This paper explores the multimodal large language models for image restoration. All reviewers are in favor of this manuscript. The AC agrees with the recommendations of reviewers.

Reviews: review: paper_summary: paper_summary: This paper introduces RestoreAgent, an innovative image restoration system that leverages multimodal large language models to autonomously handle images with multiple types of degradation. The system addresses limitations of existing all-in-one models and fixed task sequences by dynamically adapting to each image's specific degradations. RestoreAgent can identify degradation types, determine appropriate restoration tasks, optimize the execution sequence, select the most suitable models, and execute the restoration process autonomously. The authors present a method for constructing training data and demonstrate that RestoreAgent outperforms existing methods and human experts in handling complex image degradations.
strength_weakness: strength_weakness: 1. The paper constructs a training dataset for training the multimodal large language model and a testing dataset as a benchmark for evaluating performance across multiple tasks. More details and explanations regarding the construction methods of these datasets would be beneficial.
2. Table 1 presents performance rankings using both ordinal and percentage forms. The definitions and explanations for these ranking forms are somewhat lacking, which might require readers to spend extra time understanding them. Clearer explanations would facilitate better comprehension.
3. The proposed Autonomous Restoration Agent represents a novel paradigm that is likely to encounter numerous new challenges. Beyond the issues already mentioned in the paper, the authors could consider discussing additional limitations and future research directions for this paradigm. This would help future researchers better follow and improve upon this work.
questions: questions: 1. The current method appears to predict all execution steps at once for a given input image. In Figure 3, each image has a dashed line pointing to the input. Does this imply that after each execution, the result can be fed back as input? (Based on my understanding, this system supports this) The paper seems to lack analysis and experiments related to this aspect. Could the authors provide more details on this part?
2. The authors have proposed a testing dataset to evaluate multi-task processing capabilities. Will this dataset be made publicly available to facilitate further research by other researchers?
limitations: limitations: See Weaknesses
paper_summary: paper_summary: This paper presents an image restoration pipeline designed to handle various degradation types and levels by leveraging MLLM’s capabilities to select the appropriate model and determine the execution order. It begins with an analysis of why execution order and utilizing multiple models for different degradation levels are crucial for restoring complexly degraded images. The paper then constructs an instruction dataset and fine-tunes the MLLM. Experimental results demonstrate the effectiveness of the proposed restoration pipeline.
strength_weakness: strength_weakness: 1.
In the introduction, it would be helpful to explain how the Multi-Level Learning Model (MLLM) excels at understanding different types and levels of image degradation. This will show why MLLM is well-suited for handling complex combinations of image degradation. Providing this clarity will make the benefits of using MLLM for image restoration more evident.
2.
When incorporating a new type of degradation, the cost extends beyond merely training the MLLM. Please also discuss the process of constructing training data for the newly added degradation and how it integrates with previously trained data.
3.
In lines 211-212, please clarify what the mean and standard deviation are calculated over. The subscript "i" is already used for degradation type and it might be clearer to use another character.
questions: questions: 1.
What if the degradation of the input image falls outside the predefined degradation scope? This could present a generalization issue, as the model might not perform well on unseen types or levels of degradation not covered in the predefined scope. Please discuss it.
2.
In Table 2, it would be clearer to highlight the best method for each evaluation criterion. Additionally, please specify which methods the ranking improvement is compared against for better context.
limitations: limitations: The authors have addressed the limitations.
paper_summary: paper_summary: This paper proposes a new pipeline to address multiple degradation, like noise, blur and low light. Besides, a RestoreAgent with multimodal large language models is introduced to assess the type and extent of degradations in the input images and perform dynamic restorations.
strength_weakness: strength_weakness: The overall motivation of this paper is commendable, but I have a few concerns:

1. The author mentions that RestoreAgent can autonomously assess the type and extent of degradation in input images and perform restoration. This strategy is interesting. However, I am wondering how the order of different enhancement techniques is defined. For example, if the input has noise and rain streaks, how is the order of dehazing and denoising techniques determined? Will this affect performance?

2. In contrast to other image enhancement techniques, the proposed RestoreAgent should first find a suitable restoration task and then select the most appropriate model to enhance the quality of the input. Therefore, I am concerned whether this process will increase the inference time. The authors should provide some computational analysis.

3. The enhancement capabilities of this work rely heavily on existing enhancement frameworks. If existing frameworks cannot work well in some cases, such as extreme noise effects, I guess the proposed RestoreAgent may also fail. Is this true? If so, I suggest the authors mention this in the limitations section.

4. The explanation of "ranking" and "balanced" in Table 1 is still unclear. The authors should clarify the definitions of these terms.

5. It would be better to show more visual comparisons of the RestoreAgent.
questions: questions: Please see weaknesses.
limitations: limitations: Limitations are not mentioned in the paper.
paper_summary: paper_summary: For real-world images corrupted by multiple simultaneous degradations, this paper first analyzes the limitations of using all-in-one restoration models and various task-specific models. The authors then introduce RestoreAgent, which automatically identifies the types of degradation in a degraded image, determines the sequence of restoration tasks, and selects suitable models from the model pool. RestoreAgent presents an automated restoration pipeline that requires only an input image and a general human instruction, without any prior knowledge of the involved degradation tasks or manually predefined task sequences.
strength_weakness: strength_weakness: 1. Incomplete descriptions about data construction. 

- Authors randomly select up to four types of degradation from a degradation set (noise, blur, JPEG, rain, haze, and low-light) to construct paired training data. According to data synthesis strategies in [1,2], JPEG compression is typically performed after noise and blur, and in the final order. Is the degradation order of JPEG compression in this paper the same? If not, the authors should discuss the reasonableness of random sampling.

- What are the components of 23k paired data? One degraded image for each high-quality image or many degraded versions for each high-quality image? 

- What is the configuration in ablation studies about training data amount? Simultaneously scaling up low & high-quality images or synthesizing more low-quality images for each high-quality image? If it’s the former, will increasing the number of degraded images while keeping the number of high-quality images unchanged improve performance?

2. Inference time for input images with diverse resolution.
- The authors are suggested to report the running time for input images of various resolutions. This should include the total time, the running time for the RestoreAgent, and the running time for the subsequent restoration models. The reviewer is curious whether the agent's response time exceeds that of the restoration models when processing high-resolution images, such as those with 4K resolution.

3. Scalability for new tasks and models. 
- Section 4.5 demonstrates that the proposed RestoreAgent can extend to new tasks and models in just half an hour, surpassing human expert-level performance on the new task. However, it is unclear whether adaptation to the new task results in performance degradation on prior tasks, similar to the catastrophic forgetting problem in continual learning. The authors are encouraged to report the performance of the fine-tuned model on the previous tasks to address this concern.

[1] Wang X, Xie L, Dong C, et al. Real-esrgan: Training real-world blind super-resolution with pure synthetic data[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2021: 1905-1914.

[2] Zhang K, Liang J, Van Gool L, et al. Designing a practical degradation model for deep blind image super-resolution[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021: 4791-4800.
questions: questions: Addressing concerns in the Weaknesses with thorough explanations and additional experiments would significantly enhance my confidence in this work. A satisfactory response to these points may lead to a reconsideration of the current evaluation.
limitations: limitations: The manuscript includes the checklist guidelines.

Comments: comment: Thank you to the authors for their detailed response. After reviewing all the explanations and the provided visual results, I can confirm that all of my concerns have been fully addressed.
Thanks the detailed response, my concerns were well addressed.
Thanks to the authors for their detailed responses. After considering the other reviews and the replies provided, I can confirm that the authors have addressed all my concerns about data construction, catastrophic forgetting, and running time. Additionally, the authors provided potential solutions for handling high-resolution inputs. Thus, I raised my final rating to "accept." I also encourage the authors to explore the incorporation of RLHF and human preference optimization in future updates. I believe that this agent-based paradigm has the potential to spark a new wave of research in the low-level vision community.
Thanks to the authors for their efforts and replies. The authors' rebuttal addressed my questions about implement details. It's refreshing to tackle image restoration with the Agent paradigm, and I'm sure this work can inspire the community a lot! Overall, this work is quite good and I wish to see the follow up work.