An Intelligent Agentic System for Complex Image Restoration Problems

## Abstract

Abstract Real-world image restoration (IR) is inherently complex and often requires combining multiple specialized models to address diverse degradations.
Inspired by human problem-solving, we propose AgenticIR, an agentic system that mimics the human approach to image processing by following five key stages: Perception, Scheduling, Execution, Reflection, and Rescheduling.
AgenticIR leverages large language models (LLMs) and vision-language models (VLMs) that interact via text generation to dynamically operate a toolbox of IR models.
We fine-tune VLMs for image quality analysis and employ LLMs for reasoning, guiding the system step by step.
To compensate for LLMs’ lack of specific IR knowledge and experience, we introduce a self-exploration method, allowing the LLM to observe and summarize restoration results into referenceable documents.
Experiments demonstrate AgenticIR’s potential in handling complex IR tasks, representing a promising path toward achieving general intelligence in visual processing.
The code is available at https://github.com/Kaiwen-Zhu/AgenticIR .

## 1 Introduction

page\cref@result

Figure: Figure 1: The five stages of human process of IR (some details are hidden).\cref@constructprefixpage\cref@result
Refer to caption: x1.png

Image restoration (IR) problems in real-world scenarios are inherently complex.
Over the past decades, researchers have abstracted various degradation phenomena into independent IR tasks, proposing advanced models tailored to address these problems.
However, in practice, these models are seldom used in isolation. Instead, they serve as optional tools that collaborate with other methods to solve complex problems.
For example, today’s most successful image processing softwares usually integrate multiple models, allowing users to analyze the image, select models, and interact with the processing.
This approach enables users to accomplish tasks far more complex than what any single model could achieve, even when using the most basic processing operations.
To advance beyond existing models designed for passive and structured IR tasks, we propose a systematic method capable of assuming a dynamic and agentic role in diverse and complex IR scenarios.
We aim for this system to function like a human, assessing the image, selecting, planning, and executing various existing IR models to address challenges that individual models alone struggle to solve.
We believe this could be one of the promising paths towards general intelligence in IR.

To achieve this goal, we first need to abstract and generalize how human users use tools to handle IR tasks in real-world scenarios.
\creffig:stages illustrates an example.
When faced with such an input image, human users may first analyze its quality and degradations (the Perception stage), realizing that it contains two types of degradation: noise and rain streaks.
Based on the analysis and professional experience, they may devise a plan for using tools (the Scheduling stage) – for instance, applying a denoising model before a deraining model to prevent noise from affecting the deraining effect.
Following the plan, they sequentially apply these models to the image step by step (the Execution stage).
However, the effects of the models can be dynamic and unpredictable; even models with the same function may behave differently, making it difficult to determine the optimal plan directly.
Typically, after an operation, human users may re-analyze the image to assess whether the tool was effective (the Reflection stage).
If the tool was ineffective or even worsened the image, they undo the operation and formulate a new plan to achieve better results (the Rescheduling stage).
After repeating this process multiple times as shown in \creffig:stages, they eventually achieve a satisfactory result and complete the task.
Thus, we abstract the human process of using tools for IR into five stages: Perception, Scheduling, Execution, Reflection, and Rescheduling.
In this work, we propose a system that operates in this manner to tackle tasks in complex IR scenarios.

Constructing such a system is challenging because each of the aforementioned stages requires non-trivial capabilities. We summarize these capabilities as follows:

- •
Ability to Analyze Image Quality: The system needs to understand and assess the quality of images generated at each stage of the processing pipeline. This involves not just outputting a quality score but also understanding the condition of the image quality and identifying the causes of degradation and quality issues. This capability forms the foundation for the perception and reflection stages and provides the necessary information for the (re)scheduling stage.
- •
Ability to Reason Based on Context: The system involves many conditional judgments with complex and ambiguous contexts. Rule-based systems struggle to perform such complex and dynamic reasoning. This ability is crucial for connecting the various stages within the system.
- •
Knowledge and Experience in IR: The system needs to propose a model usage plan based on the condition of the image, which requires the model to have prior knowledge of the behavior of IR models and experience in using them. The system even needs to know the impact of various IR models on each other when they are used in combination.

Fortunately, large language models (LLMs) (OpenAI, ; Touvron et al., ) and vision-language models (VLMs) (OpenAI, ; Yin et al., ; Liu et al., ) offer solutions to these challenges.
VLM-based image quality analysis methods provide excellent tools for recognizing complex image degradations, quality assessment, quality comparison, and reasoning about image quality (Wu et al., ; ; ; You et al., ; ).
LLMs can perform reasoning and judgment involving complex contexts through text generation.
Moreover, advanced pre-trained language models inherently contain foundational knowledge about image restoration, and it is also possible to inject additional prior information into them.
The combined and interactive use of LLMs and VLMs can build the IR intelligent agent system we envision.

In this work, we first construct a “toolbox” comprising several IR models and synthetic complex mixed degradation scenarios to emulate real-world problems, serving as the foundational platform for our research.
Building upon this, we propose AgenticIR, a system composed of LLMs and VLMs that interact and reason through text generation to dynamically operate the tools within the “toolbox” to solve complex IR tasks.
To enable AgenticIR to analyze image quality on demand, we extend a VLM-based image quality assessment method named DepictQA (You et al., ) by fine-tuning it to meet our complex requirement.
We also utilized advanced LLMs for reasoning, guiding the system to solve complex problems following the process described in \creffig:stages.
Although advanced LLMs typically contain some foundational knowledge of IR, they lack understanding of the complex behaviors of IR models and the intricate interactions between models.
Therefore, we designed a self-exploration and experience summarization method, allowing the LLM to observe and summarize a large number of restoration results, organizing the necessary experiential knowledge into referenceable documents.
During the decision-making process, AgenticIR retrieves relevant information and knowledge as concrete ground to make informed decisions.

Our experiments demonstrate the potential of AgenticIR in solving real-world problems.
Although our research was primarily conducted in a laboratory environment, we anticipate that this paradigm holds significant promise for applications in automated and intelligent image processing.
In practical scenarios, the agents’ operations are not limited to individual models; they may involve manipulating different parameters of complex large-scale models (Yu et al., ), as well as handling highly dynamic tasks that require judgment and action.
We hope that our work can serve as a stepping stone to inspire research toward truly general and intelligent visual processing AI systems.

## 2 Related Work

Image restoration (IR) aims to reconstruct high-quality images from their degraded counterparts.
Over the years, this field has witnessed lots of successful models for single-degradation restoration, e.g., denoising (Zhang et al., ; Chen et al., ), deblurring (Nah et al., ; Chen et al., ), deraining (Fu et al., ; Gu et al., ) and super-resolution (SR) (Dong et al., ; Ledig et al., ).
However, these models’ effectiveness is limited to a narrow range as they just focus on specific degradation.
There are also efforts devoted to unifying multiple IR tasks (Liu et al., ; Chen et al., ).
Most of them either mix different degradations into training data (Zhang et al., ; Valanarasu et al., ; Yu et al., ), or predict the degradation type to assist restoration (Li et al., ; Luo et al., ; Zhang et al., ; Gu et al., ; Zhang et al., ; Chen et al., ).
Despite remarkable achievements, they may suffer from limitations including optimization difficulty, parameter inefficiency, and poor extensibility.
Besides, Yu et al. () and Chen et al. () invoke multiple single-degradation restoration tools to address multiple degradations and thus are somewhat similar to our work.
But Yu et al. () employ reinforcement learning (RL)
while Chen et al. () fine-tune a VLM to directly give an execution plan, not so agentic as ours.
Besides, their preoccupation is not combining existing tools as our work.

LLMs and VLMs have been at the forefront of advancing general intelligence. Despite being trained on extensive text datasets, their remarkable problem-solving capabilities extend far beyond typical language processing tasks (OpenAI, ; ; Touvron et al., ; Liu et al., ). LLMs are now capable of handling complex challenges once thought to require human expertise or specialized algorithms, such as mathematical reasoning (Ahn et al., ), code generation (Jiang et al., ), and addressing complex legal queries (Cui et al., ). Recent research also suggests that LLMs can generate intricate plans for robotics (Driess et al., ) and game AI (Zhu et al., ), representing a significant step toward their role as general intelligent agents.

Agents can be described as systems interacting with environments to solve complicated problems under their agendas (Franklin & Graesser, ; Li et al., ; Xi et al., ; Sumers et al., ).
Early exploration mainly focuses on symbolic architectures (Franklin & Graesser, ) reminiscent of knowledge-based expert systems, rich in expressiveness but poor in flexibility.
In the past decade, RL agents have gained success in many tasks (Silver et al., ; Hwangbo et al., ), but they lack explicitly maintained long-term plans, just acting step by step.
LLMs bring new opportunities to this field.
With extensive general knowledge, LLMs can answer simple questions logically and flexibly (OpenAI, ).
Such responses can function as intermediate steps in a compound problem-solving system.
Yao et al. () compare LLMs’ response to humans’ automatic thinking mode “System 1”, while the compound system corresponds to the deliberate thinking mode “System 2” (Kahneman, ).
Along this trend, many works contextualize LLMs within compound systems to construct language agents.
To adapt them to specific tasks and fully harness LLMs’ potential, these works draw inspiration from how humans solve problems and intuitively design various mechanisms (Wang et al., ).
For instance, Tree of Thoughts (Yao et al., ) and Graph of Thoughts (Besta et al., ) prompt the LLM to provide heuristics for search;
HuggingGPT (Shen et al., ) and Visual ChatGPT (Wu et al., ) invokes various external tools for diverse tasks;
Ghost in the Minecraft (Zhu et al., ) and Generative Agents (Park et al., ) retrieves from memory to help decision-making;
Reflexion (Shinn et al., ) reflects on errors in exploration to accumulate experience.
As well as foundation models, such compound agent systems integrated with LLMs are widely deemed promising for artificial general intelligence (Zaharia et al., ).

## 3 Method

Figure: Figure 2: An example illustrating the framework of our AgenticIR. (a) presents the entire workflow, where bubble frames beside robots represent responses from LLMs and VLMs, and the numbers in circles correspond to those in (b). (b) points out the tree search nature of the system. (c) expounds how to execute a single-degradation restoration operation with a toolbox.\cref@constructprefixpage\cref@result
Refer to caption: x2.png

We first outline the abstraction of complex IR problems and the design of our research platform (\crefsec:playground).
We establish a simulated playground where our AgenticIR can experiment and demonstrate its capabilities.
Next, we provide an overview of the workflow design for our proposed method (\crefsec:workflow).
Finally, we introduce targeted enhancements and designs to equip our system with the specific abilities and knowledge required for effective performance (\crefsec:ca).

### 3.1 Research Platform Design

page\cref@result
We abstract real-world IR scenarios into a set of mixed scenes with well-defined degradations (Kong et al., ; Wang et al., ; Kong et al., ).
While this approach may not cover all possible scenarios, it provides a simple yet general and feasible playground for our research.
Specifically, we select eight types of degradation: low resolution, noise, motion blur, defocus blur, rain, haze, JPEG compression artifacts, and low light.
There are many specialized models for each type of degradation that can be used as tools.
For each degradation, we collect three to six advanced models to build the “toolbox” that the intelligent agent can use.
Although these models/tools are designed for single-degradation tasks, our goal is for the intelligent agent to leverage them to tackle more complex IR problems.
We further create complex restoration tasks of varying difficulty by combining different degradations (Kong et al., ), allowing us to evaluate the agent’s capability in solving complex problems and its generalization ability to unseen scenarios.
The degraded data used for testing are introduced in the experimental section.
More details can be found in the \crefsupp:sec:data and A.4.

Figure: (a) Dehazing after denoising works but dehazing first fails.\cref@constructprefixpage\cref@result
Refer to caption: extracted/5948715/figs/order/haze+noise/0-input.png

### 3.2 Workflow Design

page\cref@result
Overall Workflow. As illustrated in \creffig:framework(a), our agent restores images with complex degradations through a multi-stage process.
The agent begins with the Perception stage, dynamically analyzing the content and degradations of the input image using a multi-modal vision language model.
This information is then utilized in the subsequent Scheduling stage by the language model.
In the Scheduling stage, the language model leverages the identified degradation information, its inherent commonsense knowledge about degradation restoration, and additional domain-specific knowledge provided by us to formulate judgments and develop a possible overall plan to restore the image.
This plan consists of a sequence of operations.
Following this, in the Execution stage, the agent executes the first operation in the plan, as shown in \creffig:framework(c). The agent incrementally tries available tools in an attempt to achieve the goal of the plan.
During this process, the Reflection mechanism evaluates whether a tool has been successful and whether an operation has achieved its intended purpose.
In our approach, reflection is also implemented by a multi-modal model with degradation and quality awareness.
If the goal is met, the agent proceeds to the next operation, repeating this cycle until all operations are completed and the restoration is finalized.

The Importance of Scheduling. In our workflow, correct scheduling is crucial because the complexity of IR model behaviors means that the selection of models and their execution order can greatly impact the final outcome.
In real-world scenarios, complex degradations are mixtures of multiple degradations, and the mutual influence between different models leads to fundamental differences in results based on model selection and execution sequence.
\creffig:order illustrates two examples.
In the first example, due to the simultaneous presence of haze and noise, if we directly use a dehazing model, the noise interferes with the dehazing model’s ability, resulting in failure.
In the second example, the original rain streaks can be removed by a deraining model, but after super-resolution processing, the rain streaks exceed the deraining model’s processing capabilities.
The main purpose of the Scheduling stage is to develop better IR strategies for the current task and all the phenomena described above need to be taken into account.

Rollback Mechanism. However, the plan may encounter issues, as the scheduling stage is not guaranteed to give the optimal strategy.
If execution at any stage fails to meet its objective, we have the reflection mechanism to identify the failure.
Inspired by human interaction with image processing software, we design a rollback mechanism where the agent returns to the previous stage, learns from the failure, and creates a new plan (Rescheduling).
The agent then re-enters the execution stage with the updated plan.
This process essentially performs a depth-first tree search among all possible degradation plans, as shown in \creffig:framework(b).
Given the vast number of degradation possibilities and the complexity of the situations, it is impractical to exhaustively traverse all options in real-world applications.
However, our method provides an efficient mechanism to find feasible restoration paths. The language model’s reasoning capabilities, its knowledge of IR, and the experiential information we supply all contribute to improving the success rate and efficiency of this search and decision-making process.
For more details, please refer to \crefsupp:sec:workflow.

### 3.3 Capability Acquisition

page\cref@result
\crefsec:workflow outlined the workflow of the proposed AgenticIR.
To make this complex process both feasible and effective, the system requires several non-trivial capabilities.
In the following, we describe these essential capabilities and how we equip the system to acquire them.

Ability to Analyze Image Quality. For an agent system that processes images based on varying degradation conditions, it is crucial to provide it with the ability to recognize, analyze, and assess image quality.
While numerous models exist for image quality assessment, most simply output scores or similarities.
These approaches are insufficient for two key reasons: (1) they fail to provide the necessary information for the subsequent scheduling stage; and (2) they cannot establish a threshold to judge the success of an execution, as a mere score does not indicate whether a specific degradation has been effectively removed.
To address this limitation, we focus on a new form of image quality perception and assessment – one based on VLMs (You et al., ; ; Wu et al., ; ; ).
These models can describe both the content and quality issues of an image using natural language, offering critical information for the scheduling stage to develop a restoration plan.
Some VLMs can even perform reasoning and answer questions, enabling accurate judgment of the success or failure of specific executions.
Our system incorporates a VLM-based image quality assessment model to equip it with these advanced capabilities.

Specifically, we extend DepictQA (You et al., ) – a successful multi-modal image quality assessment model.
DepictQA, trained on a large dataset, can analyze image content and categorize types of quality issues, evaluate the quality of single images, and compare the quality of multiple images.
It already possesses substantial knowledge and capabilities in image quality assessment, meeting the requirements of our perception stage.
We further adapt DepictQA to the reflection stage of our system through LoRA fine-tuning (Hu et al., ).
We additionally train the model to evaluate severity of various types of degradations in images.
With this extension, the DepictQA method can evaluate each plan execution effectively.
Implementation details are described in \crefsupp:sec:ft.

Ability to Reason Based on Context. To effectively connect the various stages within the agent system – especially during the scheduling and rescheduling stages – the agent must have a thorough understanding of the tasks and workflow.
In such a dynamic and complex system, making judgments based on predefined rules is nearly impractical.
Moreover, the increasing number of possible states and heightened complexity make rule-based approaches exceedingly convoluted, and many of the judgment conditions are ambiguous.
This necessitates that the agent system possesses a considerable degree of intelligent “thinking” and “reasoning” capabilities.
We utilize advanced LLMs to perform this reasoning, serving as a bridge that links the different stages of the system and facilitates thought processes.
Trained on vast amounts of human language data, LLMs can mimic human reasoning through text generation.
The reasoning abilities of LLMs have been recognized across various tasks (OpenAI, ; Wei et al., ; Zhou et al., ).
In our work, we employ GPT-4 (OpenAI, ) for this reasoning because it possesses excellent comprehensive abilities and knowledge that can be conveniently accessed.
It is important to note that other language models with sufficiently strong capabilities can also be used to construct this agent system.

Figure: Figure 4: LLMs alone fail to grasp the intricate interactions among operations and thus cannot plan reliably. To equip LLMs with the ability to plan, we let the agent self-explore beforehand and then summarize the accumulated experience to distill knowledge. The knowledge will be a concrete ground for planning in inference.\cref@constructprefixpage\cref@result
Refer to caption: x3.png

**Table 1: Prompting GPT-4 (OpenAI, ) to plan.\cref@constructprefixpage\cref@result**
| ID | Prompt | Response |  |
| --- | --- | --- | --- |
| 1 | ... we will conduct these tasks: [’motion deblurring’, ’jpeg compression artifact removal’]. Please provide some insights into the correct order of these unordered tasks. | ... Motion deblurring should ideally be performed first because it works best on images that have<br>not been subjected to additional processing ... | ✗ |
| 2 | ... we will conduct these tasks: [’jpeg compression artifact removal’, ’motion deblurring’]. Please provide some insights into the correct order of these unordered tasks. | ... deblurring itself may exacerbate the appearance of compression artifacts, as it attempts to sharpen the image and restore detail ... | ✗ |
| 3 | ... we will conduct these tasks: [’motion deblurring’, ’jpeg compression artifact removal’] ... we have the following experience ... please give the correct order of the tasks. | ... the experience suggests that it is often better to remove artifacts before improving the image’s content or quality ... we should remove JPEG artifacts before addressing the motion blur. | ✓ |

Knowledge and Experience in IR. As previously discussed, the execution order of IR models significantly impacts the final results. We rely on the capabilities of LLMs to formulate model usage plans based on image conditions. While LLMs are extensively trained and possess some knowledge of IR, this foundational knowledge alone is insufficient to master the usage experience of the diverse tools in our toolbox.
For example, in Cases 1 and 2 of \creftab:sche-bias, we prompt GPT-4 (OpenAI, ) to arrange the sequence of motion deblurring and JPEG artifact removal.
When we change the presentation order of these two completely unrelated degradations, GPT-4’s answers also change.
This illustrates that GPT-4 cannot reliably resolve these issues because it is merely speculating. This echoes the viewpoint of Kambhampati et al. (): LLMs possess only general planning knowledge but cannot truly handle subtask interactions, thus failing to provide executable plans.

To address this limitation, we need to inform the LLM of experiential information about the usage of tools in the toolbox during reasoning, allowing it to base its reasoning on the prior knowledge we provide.
To obtain comprehensive prior knowledge and minimize human subjective judgment, we propose a method of self-exploration and summarization, as illustrated in \creffig:learn-sche.
The agent first actively explores actual scenarios to accumulate experience and summarize this experience into referenceable documents for future reasoning.
Specifically, for images with complex degradations, we apply all possible sequences of the corresponding tools and use the fine-tuned DepictQA to evaluate the restored images, determining whether the restoration was successful. We calculate the success rate of each operation sequence.
After collecting these statistics, we prompt GPT-4 to summarize them and store the results in a knowledge base, which will be retrieved during reasoning to assist in operation scheduling.
For example, when we prompt GPT-4 again to arrange the order of motion deblurring and JPEG artifact removal using this experiential knowledge, it can find references in the experience and consistently provide answers like Response 3 in \creftab:sche-bias.
In this way, GPT-4 can offer reliable heuristics for the execution order of operations.

## 4 Experiments

As described in \crefsec:playground, we construct our research data using mixed degradations. In the experimental section, we first build the test dataset. We designed 16 combinations of mixed degradations involving 2 or 3 types of degradation and divided them into three groups: A, B, and C.
Group A contains 8 combinations, while groups B and C each contain 4 combinations. The degradation combinations in groups A and B consist of 2 degradations, whereas those in group C consist of 3 degradations to simulate more complex situations.
During the exploration phase, the agent is exposed only to group A – that is, the agent is familiar with the degradations present in group A but is unaware of those in groups B and C. This setup helps us investigate the system’s generalization ability.
We applied each of the 16 degradation combinations to every one of the 100 images in the MiO100 (Kong et al., ; ) dataset. For each combination in group A, we allocated 20 images for exploration, totaling 160 images. The remaining 1,440 images are used for testing.
More detailed information can be found in \crefsupp:sec:data.

### 4.1 Effectiveness of Individual Designs

[Uncaptioned image]: extracted/5948715/figs/consistency/entropy_bit.png

| Degradations | As planned | PSNR | SSIM | LPIPS$\downarrow$ | MANIQA | CLIP-IQA | MUSIQ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Group A | ✓ | 21.14 | 0.6836 | 0.2753 | 0.3469 | 0.5091 | 60.77 |
| ✗ | 20.79 | 0.6652 | 0.3060 | 0.3385 | 0.4819 | 59.85 |  |
| Group B | ✓ | 21.14 | 0.7088 | 0.2683 | 0.3588 | 0.5275 | 61.92 |
| ✗ | 20.32 | 0.6811 | 0.2976 | 0.3623 | 0.5257 | 60.15 |  |
| Group C | ✓ | 18.78 | 0.5352 | 0.4239 | 0.3118 | 0.4876 | 51.08 |
| ✗ | 18.49 | 0.5277 | 0.4345 | 0.3058 | 0.4719 | 51.32 |  |

Figure: Figure 5: An example showing the correctness of scheduling with experience.\cref@constructprefixpage\cref@result
Refer to caption: x4.png

[Uncaptioned image]: x5.png

Figure: Figure 6: Examples of GPT-4’s responses for operation scheduling.\cref@constructprefixpage\cref@result
Refer to caption: x7.png

**Table 2: Ablation studies of the inference workflow, including reflection (Ref.) and rollback (Rb.).\cref@constructprefixpage\cref@result**
| Degradations | Method | PSNR | SSIM | LPIPS$\downarrow$ | MANIQA | CLIP-IQA | MUSIQ |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Ref. | Rb. |  |  |  |  |  |  |  |
| Group A | ✓ | ✗ | 21.12 | 0.6809 | 0.3079 | 0.3179 | 0.4617 | 57.52 |
| ✗ | ✗ | 20.47 | 0.6659 | 0.3282 | 0.2906 | 0.4387 | 55.56 |  |
| Group B | ✓ | ✗ | 20.74 | 0.6986 | 0.3084 | 0.3126 | 0.4567 | 56.66 |
| ✗ | ✗ | 20.46 | 0.6798 | 0.3412 | 0.2966 | 0.4359 | 54.81 |  |
| Group C | ✓ | ✗ | 18.85 | 0.5510 | 0.4559 | 0.2557 | 0.3771 | 47.38 |
| ✗ | ✗ | 18.93 | 0.5447 | 0.4764 | 0.2349 | 0.3595 | 43.77 |  |

#### Fine-tuning DepictQA for Image Quality Analysis.

For each image in the test set, we prompted the fine-tuned DepictQA to assess the severity of each degradation. By treating degradation assessment as a binary classification problem—that is, determining whether an image suffers from a particular degradation—we computed the precision, recall, and F1 scores, as shown in \creftab:finetune-res. Considering that we did not invest effort in designing special strategies and that the fine-tuning consumed only moderate computational resources (three hours on four V100 GPUs), the learning process proved to be quite efficient. The fine-tuned DepictQA effectively recognized almost all types of degradation, except that the recall rates for defocus blur and motion blur were somewhat limited. This limitation is understandable, as these two types of blur are very similar in many cases; even humans find it difficult to fully distinguish between them.

#### Self-exploration and Experience Summarization.

Informed of successful rates of each operation order for each degradation combination in group A, GPT-4 tries to summarize them and distill insights.
\creffig:sche-ex shows snippets of the distilled insights and how they help operation scheduling.
It can be seen that GPT-4 does conclude general rules that seem reasonable and apply them to infer the order of operations logically.
The scheduling of “deraining, super-resolution” and “denoising, dehazing” also echo our discussion about \creffig:order in \crefsec:ca.
The example in \creffig:ab-exp shows the correctness of the plans derived from experience: without experience, the agent decides to deblur first, only to distort the JPEG artifact and render it hard to remove; scheduling with experience can avoid this.
To quantitatively evaluate, we let the agent act as the plans to restore images in the test set; as the control group, we conduct another experiment wherein the agent does not act as the plans.
Six image quality assessment metrics are used for evaluation: three full-reference metrics PSNR, SSIM (Wang et al., ), LPIPS (Zhang et al., ), and three non-reference metrics MANIQA (Yang et al., ), CLIP-IQA (Wang et al., ), MUSIQ (Ke et al., ).
\creftab:fixed lists the results, which indicate the efficacy of the proposed method.
For more details, refer to \crefsupp:sec:learn and B.2.

We also investigate the consistency improvement to justify the motivation introduced in \crefsec:ca:
for a set of operations, GPT-4 fails to consistently give a scheduling result, especially sensitive to the presentation order of operations; providing experience should alleviate this problem.
We prompt GPT-4 to schedule each set 60 times with random presentation order, and measure the dispersion of the results.
Two metrics are adopted: (1) treating the results as a discrete distribution over all permutations of operations, entropy reflects the overall dispersion;
(2) variation ratio is the proportion of samples that are not mode, straightforwardly reflecting confidence in the dominant result.
\creffig:cons compares the dispersion of scheduling results of the 16 operation sets with and without experience averaged in groups.
In fact, for some operation sets, the experience improves the consistency from almost random to deterministic (refer to \crefsupp:sec:sche-cons for details).
These results support our claim.

#### Inference Workflow.

There are two mechanisms in the inference workflow to ablate:
reflection, i.e., whether to check the tool results,
and rollback, i.e., whether to roll back failed operations.
We compare the performances with and without the two mechanisms respectively(^1^11 Reflection is a prerequisite of rollback, so rollback is disabled when ablating reflection.),
and the quantitative results(^2^22When ablating rollback, to highlight the impact, the statistics are only calculated on cases wherein rollback is triggered (20% cases).) are listed in \creftab:ablation.
We can see the absence of any mechanism leads to a performance drop in most metrics.
Qualitative comparisons are shown in \creffig:ab-ref and \creffig:ab-rb.
In \creffig:ab-ref, without reflection, although the plans are the same, the agent fails to select the appropriate tool, resulting in remnant noise.
In \creffig:ab-rb, without rollback, the agent fails to correct the suboptimal operation order suggested by the experience:
although defocus deblurring before SR makes sense, the defocus blur in this case is slight enough to be addressed by SR tools(^3^33 The training data of many SR models also include blur, as proposed by Wang et al. ().),
but first conducting defocus deblurring over-smoothens the image.
These examples show that the mechanisms are necessary in various scenarios, working together to enable the agent to find the proper operation order and tools.

### 4.2 Comparison with Other Methods

**Table 3: Quantitative comparison with all-in-one models. The best and second best performances are marked in bold and underline respectively.\cref@constructprefixpage\cref@result**
| Degradations | Method | PSNR | SSIM | LPIPS$\downarrow$ | MANIQA | CLIP-IQA | MUSIQ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Group A | AirNet | 19.13 | 0.6019 | 0.4283 | 0.2581 | 0.3930 | 42.46 |
| PromptIR | 20.06 | 0.6088 | 0.4127 | 0.2633 | 0.4013 | 42.62 |  |
| MiOIR | 20.84 | 0.6558 | 0.3715 | 0.2451 | 0.3933 | 47.82 |  |
| DA-CLIP | 19.58 | 0.6032 | 0.4266 | 0.2418 | 0.4139 | 42.51 |  |
| InstructIR | 18.03 | 0.5751 | 0.4429 | 0.2660 | 0.3528 | 45.77 |  |
| AutoDIR | 19.64 | 0.6286 | 0.3967 | 0.2500 | 0.3767 | 47.01 |  |
| AgenticIR | 21.04 | 0.6818 | 0.3148 | 0.3071 | 0.4474 | 56.88 |  |
| Group B | AirNet | 19.31 | 0.6567 | 0.3670 | 0.2882 | 0.4274 | 47.88 |
| PromptIR | 20.47 | 0.6704 | 0.3370 | 0.2893 | 0.4289 | 48.10 |  |
| MiOIR | 20.56 | 0.6905 | 0.3243 | 0.2638 | 0.4330 | 51.87 |  |
| DA-CLIP | 18.56 | 0.5946 | 0.4405 | 0.2435 | 0.4154 | 43.70 |  |
| InstructIR | 18.34 | 0.6235 | 0.4072 | 0.3022 | 0.3790 | 50.94 |  |
| AutoDIR | 19.90 | 0.6643 | 0.3542 | 0.2534 | 0.3986 | 49.64 |  |
| AgenticIR | 20.55 | 0.7009 | 0.3072 | 0.3204 | 0.4648 | 57.57 |  |
| Group C | AirNet | 17.95 | 0.5145 | 0.5782 | 0.1854 | 0.3113 | 30.12 |
| PromptIR | 18.51 | 0.5166 | 0.5756 | 0.1906 | 0.3104 | 29.71 |  |
| MiOIR | 15.63 | 0.4896 | 0.5376 | 0.1717 | 0.2891 | 37.95 |  |
| DA-CLIP | 18.53 | 0.5320 | 0.5335 | 0.1916 | 0.3476 | 33.87 |  |
| InstructIR | 17.09 | 0.5135 | 0.5582 | 0.1732 | 0.2537 | 33.69 |  |
| AutoDIR | 18.61 | 0.5443 | 0.5019 | 0.2045 | 0.2939 | 37.86 |  |
| AgenticIR | 18.82 | 0.5474 | 0.4493 | 0.2698 | 0.3948 | 48.68 |  |

We test AgenticIR on our test set for the complex-degradation restoration task.
To our knowledge, there is no open-source work for IR tool ensemble yet, so we design a simple method to compare:
with the degradations predicted by our fine-tuned DepictQA, randomly selecting one tool for each degradation and invoking them in random order.
\creftab:random lists the quantitative results.
AgenticIR outperforms this method in almost all metrics.

Figure: Figure 7: Qualitative comparison with other methods.\cref@constructprefixpage\cref@result
Refer to caption: extracted/5948715/figs/main/A-rain+lr-6/0-input.png

As for the IR performance, we compare AgenticIR with several all-in-one models:
AirNet (Li et al., ), PromptIR (Potlapalli et al., ), MiOIR (Kong et al., ), DA-CLIP (Luo et al., ), InstructIR (Conde et al., ), and AutoDIR (Jiang et al., ).
\creftab:all-in-one lists the quantitative comparison.
AgenticIR wins in almost all metrics.
Note that by this comparison we do not intend to argue the superiority of AgenticIR over all-in-one models.
Instead, we hope to verify that combining single-degradation IR tools can achieve comparable performance with all-in-one models, so this approach is feasible and meaningful.
After all, the intrinsic purpose of this paper is not IR but IR tool ensemble.

Qualitative comparison with the baseline and all-in-one models is shown in \creffig:main.
We can see AgenticIR effectively addresses all degradations and yields visually pleasing results.
In contrast, random invocations may fail to find the appropriate operation order and tools, and all-in-one models may struggle to address all degradations due to their diverse and even conflicting characteristics.

#### Real-world Applications.

We also verify AgenticIR’s ability on real-world complexly degraded images.
\creffig:real shows some examples:
the first one is from an under-display camera dataset (Zhou et al., ), and AgenticIR restores it by motion deblurring, defocus deblurring, and brightening;
the second one is from an underwater dataset (Zhou et al., ), and AgenticIR restores it by defocus deblurring, dehazing, and motion deblurring (similar decomposition is also observed by Jiang et al. ());
the last one is an image with heavy rain downloaded from the Internet, and AgenticIR restores it by deraining and dehazing.
These examples illustrated that AgenticIR can really restore some real-world low-quality images by decomposing the complex-degradation IR task into several tractable single-degradation operations, verifying the application value of AgenticIR.
\creffig:real-comp compares AgenticIR with individual models.
In this example, the deraining model (Restormer (Zamir et al., )), dehazing model (DehazeFormer (Song et al., )), and all-in-one model (AutoDIR (Jiang et al., )) all behave poorly,
while AgenticIR, sequentially conducting deraining by Restormer and dehazing by DehazeFormer, manages to yield a clean image.
This illustrates that AgenticIR as a compound system does have an advantage over individual models on some occasions.

[Uncaptioned image]: extracted/5948715/figs/real/udc/udc10_input_cropped.jpg

## 5 Conclusion

In conclusion, we introduced AgenticIR, a system that combines LLMs and VLMs to emulate human-like strategies in complex IR tasks by dynamically utilizing a toolbox of IR models through the stages of perception, scheduling, execution, reflection, and rescheduling. Our experiments demonstrate that AgenticIR effectively tackles complex IR problems beyond the capability of individual models, showing significant potential for applications in automated and intelligent image processing. We hope this work can lay a promising foundation for future research toward truly general and intelligent visual processing AI systems.

## Appendix A More Implementation Details

page\cref@result

### A.1 Inference workflow

page\cref@result

Figure: Algorithm 1 Inference workflow\cref@constructprefixpage\cref@result

#### LLM-Implemented Functions.

alg:workflow and 2 describe the entire inference workflow of AgenticIR, where the functions Evaluate, Schedule, Reflect, Reschedule and PickBest are implemented by LLMs.
In the function Evaluate, fine-tuned DepictQA (You et al., ) evaluates the severities of all types of degradation to recognize degradations present in the image;
if DepictQA thinks the severity is medium, high, or very high, then the degradation is deemed existent.
In Reflect, DepictQA evaluates the degradation corresponding to the single-degradation restoration subtask at hand to judge whether the execution is successful.
In Schedule and Reschedule, GPT-4 (OpenAI, ) references accumulated experience to infer the order of given subtasks;
specially, in Reschedule, GPT-4 is informed the previous failed attempts.
In PickBest, the candidates are linearly scanned and compared in pairs by DepictQA, resulting in the highest quality candidate.
Prompts are listed in \crefsupp:tab:prompts.

#### Subtask Execution.

To select the appropriate tool for the subtask, we iteratively invoke and reflect.
After invoking a tool, if the fine-tuned DepictQA thinks the severity of the degradation is very low, then we accept the tool result immediately; otherwise, we continue to try other tools.
If no tool gives very low degradation severity but some give low severity, then we run PickBest to select the best one of them as a successful subtask result.
If no tool gives low or very low severity, then the subtask is reported as failed;
in this case, we still run PickBest to find the best one, which is useful if all subtask orders from the input image fail.

### A.2 Fine-Tuning VLM

page\cref@result
DIV2K (Agustsson & Timofte, ) and Flickr2K (Timofte et al., ) datasets are used for fine-tuning DepictQA (You et al., ) on degradation evaluation.
We randomly add two to four degradations on each image to obtain 15,000 complexly degraded images.
For each of them, we iterate the seven degradations(^4^44 The degradation “low resolution” is excluded since we can directly recognize it by checking the resolution.) to synthesize seven question-answer pairs:
question: “What’s the severity of degradation in this image?”; answer: “very low / low / medium / high / very high”.
We fine-tune DepictQA for one epoch with batch size 64 on 4 NVIDIA Tesla V100 GPUs,
using learning rate 0.0005, weight decay 0.001, and Adam optimizer ($\beta_{1}=0.9,\beta_{2}=0.95$).

### A.3 Data

page\cref@result
This section details how we synthesize degraded images. We first introduce how the eight degradations are implemented respectively and then list the combinations.

#### Single degradation.

For low resolution, following most SR works (Dong et al., ; Liang et al., ), we downsample the images by a factor of 4 using bicubic interpolation.
For noise, we add Gaussian or Poisson noise with random scale.
For motion blur, following Michaelis et al. (), we filter images with linear kernels with random direction and radius.
For defocus blur, following Michaelis et al. (), we filter images with circular kernels with random radius.
For rain, following Kong et al. (), we first add noise and then filter the noise with linear kernels with random direction.
For haze, following most dehazing works (He et al., ; Li et al., ), we adopt the atmospheric scattering model with random global atmospheric light and scattering coefficient.
For JPEG compression artifact, we compress images with random quality factor.
For low light, we randomly reduce the V channel value of HSV color space with one of the following approaches: subtracting a constant, Gamma correction, and linear mapping.

#### Degradation combination.

supp:tab:degra-combs lists the 16 degradation combinations for exploration and testing.
Note that we consider real-world scenarios deliberately when designing the combinations.
The combinations should be common in reality, e.g., haze and rain, dark and noise.
Besides, the degradation order should conform to physical limitations, e.g., blur should be added before noise since they follow a chronological order in imaging.

### A.4 Tools

page\cref@result
\crefsupp:tab:irmodel lists the single-degradation restoration tools adopted in our implementation.
Except for brightening, all tools are cutting-edge deep models.
For brightening, deep models are not employed since the low-light conditions considered here are not so severe as those considered by deep models (Wei et al., ) for practical purposes.

**Table 4: Prompts for VLM and LLM. {$\cdot$} represents slot to fill.\cref@constructprefixpage\cref@result**
| Prompt for DepictQA to evaluate degradation |
| --- |
| What’s the severity of {degradation} in this image? Answer the question using a single word or phrase in the followings: very low, low, medium, high, very high. |
| Prompt for GPT-4 to (re)schedule subtasks |
| There’s an image suffering from degradations {degradations}. We will invoke dedicated tools to address these degradations, i.e., we will conduct these tasks: {agenda}. Now we need to determine the order of these unordered tasks. For your information, based on past trials, we have the following experience:<br>{experience}<br>Based on this experience, please give the correct order of the tasks. Your output must be a JSON object with two fields: "thought" and "order", where "order" must be a permutation of {agenda} in the order you determine.<br>(Besides, in attempts just now, we found the result is unsatisfactory if {failed_tries} is conducted first. Remember not to arrange {failed_tries} in the first place.) |
| Prompt for DepictQA to compare two images |
| Which of the two images, Image A or Image B, do you consider to be of better quality? Answer the question using a single word or phrase. |

**Table 5: Degradation combinations for exploration and testing.\cref@constructprefixpage\cref@result**
| Group | # Degradations | Seen in exploration? | Degradation combination |
| --- | --- | --- | --- |
| A | 2 | Yes | rain, haze<br>motion blur, low resolution<br>low light, noise<br>defocus blur, JPEG compression artifact<br>noise, JPEG compression artifact<br>rain, low resolution<br>motion blur, low light<br>defocus blur, haze |
| B | 2 | No | motion blur, JPEG compression artifact<br>haze, noise<br>defocus blur, low resolution<br>rain, low light |
| C | 3 | No | haze, motion blur, low resolution<br>rain, noise, low resolution<br>low light, defocus blur, JPEG compression artifact<br>motion blur, defocus blur, noise |

Table: Table 6: Single-degradation restoration tools.\cref@constructprefixpage\cref@result

## Appendix B More Results

page\cref@result

### B.1 Learning from Exploration

page\cref@result
After collecting the statistics in exploration, we prompt GPT-4 by:

We are studying image restoration with multiple degradations. The degradation types we are focusing on are: low-resolution, noise, motion blur, defocus blur, rain, haze, dark, and jpeg compression artifact. We have tools to address these degradations, that is, we can conduct these tasks: super-resolution, denoising, motion deblurring, defocus deblurring, deraining, dehazing, brightening, and jpeg compression artifact removal. The problem is, given the tasks to conduct, we need to determine the order of them. This is very complicated because different tasks may have special requirements and side-effects, and the correct order of tasks can significantly affect the final result. We have conducted some trials and collected the following experience:

To address dark+noise in the image, when conducting first denoising and then brightening, the fail rates of addressing [’dark’, ’noise’] are [’22%’, ’43%’] respectively, and the total fail rate is 32%; when conducting first brightening and then denoising, the fail rates of addressing [’dark’, ’noise’] are [’28%’, ’42%’] respectively, and the total fail rate is 35%.

To address defocus blur+haze in the image, when conducting first defocus deblurring and then dehazing, the fail rates of addressing [’defocus blur’, ’haze’] are [’0%’, ’36%’] respectively, and the total fail rate is 18%; when conducting first dehazing and then defocus deblurring, the fail rates of addressing [’defocus blur’, ’haze’] are [’0%’, ’40%’] respectively, and the total fail rate is 20%.

To address defocus blur+jpeg compression artifact in the image, when conducting first jpeg compression artifact removal and then defocus deblurring, the fail rates of addressing [’defocus blur’, ’jpeg compression artifact’] are [’10%’, ’31%’] respectively, and the total fail rate is 20%; when conducting first defocus deblurring and then jpeg compression artifact removal, the fail rates of addressing [’defocus blur’, ’jpeg compression artifact’] are [’8%’, ’48%’] respectively, and the total fail rate is 28%.

To address motion blur+dark in the image, when conducting first motion deblurring and then brightening, the fail rates of addressing [’motion blur’, ’dark’] are [’22%’, ’25%’] respectively, and the total fail rate is 23%; when conducting first brightening and then motion deblurring, the fail rates of addressing [’motion blur’, ’dark’] are [’28%’, ’25%’] respectively, and the total fail rate is 26%.

To address motion blur+low resolution in the image, when conducting first motion deblurring and then super-resolution, the fail rates of addressing [’motion blur’, ’low resolution’] are [’23%’, ’9%’] respectively, and the total fail rate is 16%; when conducting first super-resolution and then motion deblurring, the fail rates of addressing [’motion blur’, ’low resolution’] are [’31%’, ’6%’] respectively, and the total fail rate is 19%.

To address noise+jpeg compression artifact in the image, when conducting first denoising and then jpeg compression artifact removal, the fail rates of addressing [’noise’, ’jpeg compression artifact’] are [’38%’, ’13%’] respectively, and the total fail rate is 26%; when conducting first jpeg compression artifact removal and then denoising, the fail rates of addressing [’noise’, ’jpeg compression artifact’] are [’38%’, ’14%’] respectively, and the total fail rate is 26%.

To address rain+haze in the image, when conducting first deraining and then dehazing, the fail rates of addressing [’rain’, ’haze’] are [’5%’, ’37%’] respectively, and the total fail rate is 21%; when conducting first dehazing and then deraining, the fail rates of addressing [’rain’, ’haze’] are [’25%’, ’24%’] respectively, and the total fail rate is 24%.

To address rain+low resolution in the image, when conducting first deraining and then super-resolution, the fail rates of addressing [’rain’, ’low resolution’] are [’26%’, ’2%’] respectively, and the total fail rate is 14%; when conducting first super-resolution and then deraining, the fail rates of addressing [’rain’, ’low resolution’] are [’63%’, ’0%’] respectively, and the total fail rate is 32%.

The response, which directly serves as experience in \crefsupp:tab:prompts, is

Based on the collected experience, we can distill the following knowledge to guide the order of tasks for image restoration:

1. **Dark + Noise**: It is slightly better to denoise before brightening (32% total fail rate) than to brighten before denoising (35% total fail rate).

2. **Defocus Blur + Haze**: Defocus deblurring should be done before dehazing, as it results in a lower total fail rate (18% vs. 20%).

3. **Defocus Blur + JPEG Compression Artifact**: Removing JPEG compression artifacts before defocus deblurring leads to a lower total fail rate (20% vs. 28%).

4. **Motion Blur + Dark**: Motion deblurring should be done before brightening, as it results in a lower total fail rate (23% vs. 26%).

5. **Motion Blur + Low Resolution**: Motion deblurring should be done before super-resolution, as it results in a lower total fail rate (16% vs. 19%).

6. **Noise + JPEG Compression Artifact**: The order of denoising and JPEG compression artifact removal does not significantly affect the total fail rate (both 26%).

7. **Rain + Haze**: Deraining should be done before dehazing, as it results in a lower total fail rate (21% vs. 24%).

8. **Rain + Low Resolution**: Deraining should be done before super-resolution, as it results in a significantly lower total fail rate (14% vs. 32%).

From these observations, we can infer that generally, it is more effective to address blurring issues (motion blur, defocus blur) before addressing issues related to image content enhancement (super-resolution, brightening, dehazing). Additionally, it is often better to remove artifacts (rain, JPEG compression) before improving the image’s content or quality. However, the impact of the order may vary, and in some cases, such as noise and JPEG compression artifact, the order does not significantly affect the outcome.

When dealing with more complex degradations, these guidelines can serve as a starting point, but it may be necessary to conduct additional trials to determine the most effective order of tasks.

### B.2 Scheduling Result

page\cref@result
\crefsupp:tab:sche-res comprehensively lists the reasoning and results of scheduling the 16 degradation combinations,
and \crefsupp:tab:fixed lists the performance of the resultant plans over each combination.
We can see that the scheduling is logical and profitable.

### B.3 Scheduling Consistency

page\cref@result
\crefsupp:fig:cons compares the dispersion of scheduling results with and without experience for each degradation combination.
It can be seen that in some extreme cases, scheduling with experience can always yield the same result, while scheduling without experience yields nearly random results.
This strongly supports our motivation that providing experience is a must for the sake of consistency.

Besides, as introduced in \crefsec:ca, we find that the scheduling result may be sensitive to the presentation order of subtasks.
Therefore, the dispersion of scheduling results may arise from two aspects:
one is the intrinsic uncertainty of the LLM, i.e., the response may differ even though the prompt remains the same;
another is the bias to presentation order, i.e., the order significantly affects the result.
To investigate the bias, the first aspect should be ruled out.
We design an intuitive approach:
for a metric that measures dispersion, compute the metric values for all presentation orders respectively and then subtract their average from the metric value for the overall results.
Formally, denote the metric by $M$ and the distribution of scheduling results by $p$.
Also, when the presentation order is fixed, denote the distributions of scheduling results by $p_{1},p_{2},\cdots,p_{N}$, respectively, where $N$ is the number of subtask permutations.
The difference

$$ $M(p)-\frac{1}{N}\sum_{i=1}^{N}M(p_{i})$ $$

can reflect the sensitivity to presentation order.
For entropy and variation ratio, \crefsupp:fig:cons-diff compares this difference for scheduling results with and without experience.
Notably, for the degradation combination in group A and B, the differences are all zero for scheduling with experience, which means the bias to presentation order is totally eliminated in a sense.
These results again justify our design of exploration.

**Table 7: Scheduling of degradation combinations.**
| Group A |
| --- |
| Degradations: rain, haze.<br>Scheduling result: deraining, dehazing.<br>Reasoning:<br>According to the collected experience, specifically point 7, it is more effective to perform deraining before dehazing as it results in a lower total fail rate. Therefore, the correct order of tasks should be to first apply the deraining tool and then the dehazing tool. |
| Degradations: motion blur, low resolution.<br>Scheduling result: motion deblurring, SR.<br>Reasoning:<br>According to the collected experience, specifically point 5, it is more effective to address motion blur before dealing with low resolution. Therefore, we should perform motion deblurring before super-resolution to achieve a lower total fail rate. |
| Degradations: low light, noise.<br>Scheduling result: denoising, brightening.<br>Reasoning:<br>According to the collected experience, it is slightly better to denoise before brightening when dealing with an image that is both dark and noisy. This is supported by the lower total fail rate observed in past trials for denoising before brightening (32%) compared to brightening before denoising (35%). |
| Degradations: defocus blur, JPEG compression artifact.<br>Scheduling result: JPEG compression artifact removal, defocus deblurring.<br>Reasoning:<br>According to the collected experience, it is more effective to address image artifacts like JPEG compression before dealing with blurring issues. Specifically, for the combination of defocus blur and JPEG compression artifact, the experience indicates that removing JPEG compression artifacts before defocus deblurring leads to a lower total fail rate (20% vs. 28%). Therefore, the correct order of tasks should start with JPEG compression artifact removal followed by defocus deblurring. |
| Degradations: noise, JPEG compression artifact.<br>Scheduling result: JPEG compression artifact removal, denoising.<br>Reasoning:<br>Since the experience indicates that the order of denoising and JPEG compression artifact removal does not significantly affect the total fail rate, we can choose either order for the tasks. However, to maintain consistency with the general guideline that suggests removing artifacts before improving the image’s content or quality, we will opt to remove JPEG compression artifacts before denoising. |
| Degradations: rain, low resolution.<br>Scheduling result: deraining, SR.<br>Reasoning:<br>According to the collected experience, specifically point 8, it is more effective to address the artifact caused by rain before enhancing the image’s resolution. Therefore, we should perform deraining before super-resolution to achieve a lower total fail rate. |
| Degradations: motion blur, low light.<br>Scheduling result: motion deblurring, brightening.<br>Reasoning:<br>According to the collected experience, specifically point 4, it is more effective to address motion blur before brightening. This results in a lower total fail rate. |
| Degradations: defocus blur, haze.<br>Scheduling result: defocus deblurring, dehazing.<br>Reasoning:<br>According to the collected experience, it is advised to address defocus blur before haze. This is because defocus deblurring should be done before dehazing, as it results in a lower total fail rate. |
| Group B |
| Degradations: motion blur, JPEG compression artifact.<br>Scheduling result: JPEG compression artifact removal, motion deblurring.<br>Reasoning:<br>Since the experience suggests that it is often better to remove artifacts before improving the image’s content or quality, and specifically that removing JPEG compression artifacts before defocus deblurring leads to a lower total fail rate, it would be logical to apply the same principle to motion blur. Therefore, we should remove JPEG compression artifacts before addressing the motion blur. |
| Degradations: haze, noise.<br>Scheduling result: denoising, dehazing.<br>Reasoning:<br>While the provided experience does not directly address the combination of haze and noise, we can infer from the guidelines that issues related to image content enhancement should be addressed after dealing with artifacts. Since noise can be considered an artifact and dehazing is related to enhancing image content by removing haze, it would be logical to denoise the image before dehazing it. |
| Degradations: defocus blur, low resolution.<br>Scheduling result: defocus deblurring, SR.<br>Reasoning:<br>Following the guidelines, it is generally more effective to address blurring issues before image content enhancement. Therefore, we should perform defocus deblurring before super-resolution. |
| Degradations: rain, low light.<br>Scheduling result: deraining, brightening.<br>Reasoning:<br>Since there is no direct experience related to the combination of rain and dark, we can draw parallels from similar situations. The guidelines suggest addressing artifacts like rain before enhancing the image’s content. This is supported by point 7, which recommends deraining before dehazing, and point 8, which advises deraining before super-resolution. Both dehazing and super-resolution are content enhancement tasks, similar to brightening. Therefore, it would be logical to apply the deraining tool before the brightening tool. |
| Group C |
| Degradations: low light, defocus blur, JPEG compression artifact.<br>Scheduling result: JPEG compression artifact removal, defocus deblurring, brightening.<br>Reasoning:<br>According to the collected experience, it is generally more effective to address blurring issues before image content enhancement. Additionally, removing artifacts like JPEG compression should be done before addressing blurring issues. Therefore, the order should start with JPEG compression artifact removal, followed by defocus deblurring, and finally brightening the image. |
| Degradations: haze, motion blur, low resolution.<br>Scheduling result: motion deblurring, dehazing, SR.<br>Reasoning:<br>According to the guidelines, it is more effective to address blurring issues before image content enhancement. Therefore, we should start with motion deblurring. There is no direct guideline for the order of dehazing and super-resolution in relation to motion blur, but since dehazing is an image content enhancement and super-resolution is also an enhancement but more towards image quality, we should perform dehazing before super-resolution. This aligns with the general principle of removing artifacts before improving the image’s content or quality. |
| Degradations: motion blur, defocus blur, noise.<br>Scheduling result: motion deblurring, defocus deblurring, denoising.<br>Reasoning:<br>Since both motion blur and defocus blur are types of blurring issues, and based on the experience that blurring issues should be addressed before image content enhancement, which in this case is denoising, we should first deal with the blurring. Between motion blur and defocus blur, there is no direct experience suggesting which should be addressed first. However, considering that motion blur might affect the entire image while defocus blur could be more localized, it might be more effective to correct the motion blur first to ensure that any subsequent defocus deblurring operates on a more stable image base. Therefore, the order should start with motion deblurring, followed by defocus deblurring, and finally denoising. |
| Degradations: rain, noise, low resolution.<br>Scheduling result: deraining, denoising, SR.<br>Reasoning:<br>According to the guidelines, it is generally more effective to remove artifacts like rain before addressing image content enhancement like super-resolution. Additionally, there is no specific mention of the interaction between noise and low resolution, but since noise is an artifact, it would be logical to address it before enhancing the image content. Therefore, the order should start with deraining, followed by denoising, and finally, super-resolution. |

**Table 8: Detailed comparison between acting as the agent’s plan and the opposite. “Not as planned” means randomly shuffling the plan (guaranteed to be different).\cref@constructprefixpage\cref@result**
| Degradations | As planned | PSNR | SSIM | LPIPS$\downarrow$ | MANIQA | CLIP-IQA | MUSIQ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| rain,<br>haze | ✓ | 19.00 | 0.8075 | 0.1655 | 0.4309 | 0.6015 | 68.25 |
| ✗ | 16.35 | 0.7324 | 0.2250 | 0.4296 | 0.5900 | 68.03 |  |
| motion blur,<br>low resolution | ✓ | 20.57 | 0.5532 | 0.2575 | 0.4192 | 0.6575 | 66.38 |
| ✗ | 20.28 | 0.5427 | 0.2637 | 0.4374 | 0.6568 | 67.19 |  |
| low light,<br>noise | ✓ | 20.13 | 0.7454 | 0.3042 | 0.3516 | 0.4679 | 59.88 |
| ✗ | 18.97 | 0.7175 | 0.2997 | 0.3962 | 0.5022 | 61.63 |  |
| defocus blur,<br>jpeg compression artifact | ✓ | 24.34 | 0.6846 | 0.3885 | 0.2163 | 0.2859 | 45.83 |
| ✗ | 23.78 | 0.6580 | 0.4614 | 0.1938 | 0.2743 | 41.53 |  |
| noise,<br>jpeg compression artifact | ✓ | 28.08 | 0.8129 | 0.2491 | 0.3537 | 0.5343 | 60.03 |
| ✗ | 27.50 | 0.7997 | 0.2714 | 0.3484 | 0.5182 | 58.95 |  |
| rain,<br>low resolution | ✓ | 20.42 | 0.6005 | 0.3045 | 0.4216 | 0.6710 | 66.42 |
| ✗ | 21.59 | 0.5796 | 0.3801 | 0.3449 | 0.5292 | 64.35 |  |
| motion blur,<br>low light | ✓ | 17.06 | 0.5350 | 0.2849 | 0.2785 | 0.4151 | 60.03 |
| ✗ | 17.05 | 0.5442 | 0.2903 | 0.2826 | 0.3935 | 60.41 |  |
| defocus blur,<br>haze | ✓ | 19.52 | 0.7300 | 0.2484 | 0.3033 | 0.4399 | 59.33 |
| ✗ | 20.81 | 0.7477 | 0.2563 | 0.2751 | 0.3908 | 56.73 |  |
| motion blur,<br>jpeg compression artifact | ✓ | 22.12 | 0.6638 | 0.3385 | 0.2355 | 0.3784 | 51.24 |
| ✗ | 21.61 | 0.6464 | 0.3831 | 0.1943 | 0.3215 | 45.64 |  |
| haze,<br>noise | ✓ | 20.20 | 0.7395 | 0.3130 | 0.3480 | 0.4645 | 59.81 |
| ✗ | 17.45 | 0.6654 | 0.3523 | 0.3785 | 0.5519 | 58.38 |  |
| defocus blur,<br>low resolution | ✓ | 22.42 | 0.6118 | 0.2475 | 0.4283 | 0.6681 | 67.92 |
| ✗ | 23.22 | 0.6232 | 0.2491 | 0.4432 | 0.6777 | 67.74 |  |
| rain,<br>low light | ✓ | 19.82 | 0.8202 | 0.1743 | 0.4234 | 0.5991 | 68.70 |
| ✗ | 18.99 | 0.7892 | 0.2058 | 0.4334 | 0.5517 | 68.86 |  |
| haze, motion blur,<br>low resolution | ✓ | 17.20 | 0.4923 | 0.2951 | 0.4193 | 0.6640 | 65.97 |
| ✗ | 15.98 | 0.4730 | 0.3215 | 0.4084 | 0.6698 | 65.32 |  |
| rain, noise,<br>low resolution | ✓ | 20.73 | 0.5602 | 0.3785 | 0.4100 | 0.6736 | 62.34 |
| ✗ | 21.14 | 0.5868 | 0.3908 | 0.3683 | 0.5610 | 59.89 |  |
| low light, defocus blur,<br>jpeg compression artifact | ✓ | 18.57 | 0.6194 | 0.4223 | 0.1978 | 0.2726 | 47.04 |
| ✗ | 18.40 | 0.6108 | 0.4251 | 0.1951 | 0.2685 | 46.45 |  |
| motion blur, defocus blur,<br>noise | ✓ | 18.64 | 0.4690 | 0.5997 | 0.2202 | 0.3401 | 28.96 |
| ✗ | 18.42 | 0.4402 | 0.6006 | 0.2514 | 0.3883 | 33.63 |  |

Figure: Figure 8: Detailed comparison between dispersion of scheduling results with and without experience.\cref@constructprefixpage\cref@result
Refer to caption: extracted/5948715/supp/figs/consistency/entropy_bit_detail.png

Figure: Figure 9: Detailed comparison between sensitivity to presentation order when scheduling with and without experience.\cref@constructprefixpage\cref@result
Refer to caption: extracted/5948715/supp/figs/consistency/entropy_difference_bit_detail.png

## Appendix C Limitation

Our primary goal is to realize an intelligent agent for IR tool ensemble, but this paper only considers single-degradation restoration tools, limiting the application to the so-called complex-degradation restoration problem.
It is worth exploring whether our paradigm can integrate more general and heterogeneous tools,
which will require higher flexibility and decision-making capabilities.
A generally capable agent is supposed to surmount these challenges.

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 6
presentation: 4
questions: 1. How are the discovered workflows similar to the original motivation of following the human workflow? For instance, is the subtask scheduling by GPT-4 (w. experience) match the best practices performed by a human? It would be better if the authors could provide more insights or discussions.

2. How does the proposed model perform when there is only a single type of degradation? Does it also perform competitively?

3. What is the criterion for deciding whether Execution step is Failed or Successful? (I might have missed)
soundness: 4
strengths: 1. Clear presentation of the benefits of the proposed methodologies. I especially liked Figure 3, 6, 7, and 8, where the authors show dramatic improvements on why choosing a good scheduler for the image restoration components is important, as well as reflection and rollback.

2. Novelty in connecting the human process of IR with LLM-based agentic systems. Though the idea of mimicking the human workflow is being widely adopted more recently, application to image restoration tasks and showing effectiveness is not demonstrated before, to my knowledge.

3. Thorough justification of the design choices and careful experiment designs. Reasons for the proposed workflow and the capabilities the authors are trying to give to the LLM/VLMs are well described, and the evaluations seem to be fairly performed.
summary: This paper presents an agentic workflow based on LLM/VLMs for image restoration. The agentic system follows how actual humans would process images, consisting of five stages: Perception, Scheduling, Execution, Reflection, and Rescheduling. Since the existing VLMs are not sufficiently capable of analyzing image quality or reasoning about the order of image processing tasks, the VLMs are finetuned and allowed for (self-)exploration to understand the effects of scheduling the image restoration components. Experimental results clearly demonstrate the effects of scheduling with learned experience and the other proposed components.
weaknesses: 1. No cost analysis. Using such agentic systems require numerous requests to the LLM/VLM APIs; if the system chooses to perform "Reflection" with the tree search, the worst case scenario would be extremely costly. Compared to the existing image restoration models, the proposed model uses significantly more compute. In this sense, given that many previous works (roughly) match the FLOPs when comparing the restoration quality, one might argue that the comparisons are unfair.

2. Relatively subtle improvements for quantitative metrics (though qualitative improvements look quite significant). I would suggest also adding quantitative measures on the figures so that the readers can compare both aspects with a single glance.

---

confidence: 4
contribution: 1
initial_rating: 5
presentation: 3
questions: NA
soundness: 2
strengths: 1. The proposed system offers a comprehensive approach to image restoration, addressing a broad range of degradation types through a structured, adaptable methodology.
2. It incorporates human-interaction-inspired insights into the image restoration process, potentially enhancing adaptability and effectiveness in handling complex restoration tasks.
summary: The proposed AgenticIR system addresses the inherent complexity of real-world image restoration (IR) tasks by emulating a human-like, multi-stage processing workflow. The system operates through distinct phases: Perception, Scheduling, Execution, Reflection, and Rescheduling. It integrates Large Language Models (LLMs) and Vision Language Models (VLMs) into a collaborative framework, allowing text-based interactions to direct the application of specialized IR models. This agentic approach builds on existing multi-modal IR frameworks by dynamically adapting its restoration strategy, actively reassessing and adjusting to handle various complex degradations.
weaknesses: 1. **Comparison fairness**: The comparative experiments appear to lack fairness, as the baselines (e.g., InstructIR) are designed as unified IR models trained to handle multiple degradation types in a single framework. In contrast, AgenticIR leverages specialized off-the-shelf models for each type of degradation. Therefore, it would be more appropriate to compare AgenticIR to the state-of-the-art models for each specific degradation task rather than to unified restoration models.
  
2. **Efficiency Concerns**: While the system is comprehensive, its workflow is notably complex and lengthy. Compared to regular image restoration models, how efficient is AgenticIR in processing images? This is a critical aspect for real-world applications of image restoration and should be addressed with precise comparative evaluations.

3. **Toolbox Ablation Study**: Lines 159-160 state, "For each degradation, we collect three to six advanced models to build the ‘toolbox’ that the intelligent agent can use." There is no ablation study analyzing the impact of selecting these advanced models on the system’s effectiveness. Understanding the influence of each selected model in the toolbox could provide valuable insights.

4. **GPT-4 Usage and Reproducibility**: AgenticIR uses GPT-4, but GPT-4 lacks a fixed version, which raises concerns about the reproducibility of experimental results. Additionally, there is no ablation study on the effects of using alternative LLMs, particularly open-source options, on performance outcomes.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Please see the Weaknesses.
soundness: 3
strengths: -	This paper is well-written and easy to follow.
-	The experimental setup is comprehensive, with sufficient ablation and comparative experiments demonstrating the effectiveness of their proposed methods
-	The discovery that execution order is key to restoring complex degraded images is compelling.
summary: To address the complex image restoration(IR) problems in real-world scenarios, the authors propose AgenticIR, which is an agent-based system that comprises five stages: perception, scheduling, execution, reflection, and rescheduling. The system leverages a Vision-Language Model (VLM) and a Large Language Model (LLM) to reason and connect these five stages, utilizing an IR toolbox during the execution phase to perform actual image reconstruction.

The three main elements in this process are the LLM, VLM, and the IR toolbox. The VLM primarily analyzes the image quality and summarizes the degradation issues, fine-tuning its capabilities based on existing work. The LLM is responsible for planning and strategizing based on the VLM's results, utilizing GPT-4 and employing a self-exploration method to gain experience with IR problems. The IR toolbox consists of a combination of 3-6 existing models tailored to each type of image degradation.
weaknesses: -	The main concern is that this work resembles several widely-used frameworks (e.g., large language models (LLMs), vision-language models (VLMs) and image restoration models), giving it a predominantly engineering-focused approach. 
-	Additionally, this work appears complex, so providing statistics on the time and complexity involved in a single inference would enhance clarity.
-	Given that LLMs and VLMs often struggle with the issue of 'hallucination,' does this work encounter a similar challenge? If so, how does it address this problem?
-	What are the limitations of the proposed framework?
-	Regarding the point that 'execution order is crucial', are the documents (knowledge) remain consistent during inference across different test samples?

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: In addition to the weaknesses mentioned, I have two further questions:

1. Why Not Use VLMs Exclusively Throughout the Pipeline? Given VLMs' strong capabilities in image quality assessment and reasoning, could a VLM-only approach be more efficient or effective for the entire pipeline?
2. Would Online Updates to the Reference Data Benefit the Pipeline? Could implementing real-time updates to the experiential knowledge base further enhance the pipeline’s adaptability and performance?

I hope the authors can make up for the weaknesses mentioned and address these questions.
soundness: 3
strengths: 1. Human-Centric Design: AgenticIR provides an image restoration approach that mirrors human actions, incorporating processes like reflection and iterative rescheduling into its pipeline. This design enhances action interpretability and facilitates meaningful human interaction with the system.
2. Clear and Concise Expression: The paper presents complex ideas with clarity, accompanied by detailed images and diagrams that enhance comprehension and support the technical explanations.
3. Comprehensive Experiments and Ablation Studies: Thorough experimental evaluations are provided, with well-structured ablation studies for each module. This approach effectively validates the system's design and performance.
4. Illustrative Pipeline Examples: The pipeline is illustrated with specific cases, offering a clear understanding of how each component functions within real-world scenarios.
summary: This paper introduces AgenticIR, an intelligent system designed to handle complex image restoration tasks by emulating human-like problem-solving methods. The system operates through five stages: Perception, Scheduling, Execution, Reflection, and Rescheduling. AgenticIR leverages LLM and VLM, using their text generation capabilities to operate a set of IR tools dynamically. It relies on VLMs for image quality assessment and LLMs for step-by-step reasoning, enhancing its adaptability to various IR challenges.

It also incorporates a self-exploration mechanism that generates referenceable summaries from past restoration attempts, which improves its decision-making. Experimental results show AgenticIR’s effectiveness in complex restoration scenarios, highlighting its potential for real-world automated image processing and broader AI applications in visual processing.
weaknesses: 1. Limitations Compared to Optimal Solutions: AgenticIR tends to encounter an early stopping issue, where it may settle on a satisfactory solution prematurely, halting further exploration and potentially missing the optimal outcome. Addressing this limitation is important, and it might be beneficial to add an additional row to Table 4 to reflect this aspect.
2. Insufficient Reporting on Iteration Count and Processing Time: Although the paper emphasizes the role of experiential information and provides illustrative examples, it lacks concrete data on the actual reduction in iterations or time consumption. Including specific metrics on these improvements would strengthen the evaluation of AgenticIR’s efficiency and practical advantages.

Meta-review: The paper addresses the problem of image restoration and proposes an agential system where a VLM and LLM can mimic a human workflow for image restoration (e.g. using specific image restoration tools, make decisions as to what tools should be used, reflecting on the current performance, etc.). The paper specifically proposes a 5 stage process composed of perception, scheduling, execution, reflection, and rescheduling in which the VLM attempts to: perceive specific degradation, make a plan as to how to address them (as well as the order in which to address them), apply the tools and then enter a loop of rescheduling, execution, and reflection where the VLM can decide whether a step succeeded or not and how to proceed from there.

The main strength of the paper is its interesting proposed method and strong qualitative results as well as decent quantitative results. While agentic workflows are becoming more commonplace, a strong demonstration of how they can apply to vision (e.g. image restoration) is very interesting and useful for the field. The main weakness seems to be inference time. However as this is, to my and the reviewers' knowledge, the first attempt at an agential workflow, I value the interestingness of the approach and the performance more.

I advocate for acceptance.

Decision: Accept (Poster)