AdaIR: Adaptive All-in-One Image Restoration via Frequency Mining and Modulation

## Abstract

Abstract In the image acquisition process, various forms of degradation, including noise, blur, haze, and rain, are frequently introduced. These degradations typically arise from the inherent limitations of cameras or unfavorable ambient conditions. To recover clean images from their degraded versions, numerous specialized restoration methods have been developed, each targeting a specific type of degradation.
Recently, all-in-one algorithms have garnered significant attention by addressing different types of degradations within a single model without requiring the prior information of the input degradation type.
However, these methods purely operate in the spatial domain and do not delve into
the distinct frequency variations inherent to different degradation types.
To address this gap, we propose an adaptive all-in-one image restoration network based on frequency mining and modulation.
Our approach is motivated by the observation that different degradation types impact the image content on different frequency subbands, thereby requiring different treatments for each restoration task.
Specifically, we first mine low- and high-frequency information from the input features, guided by the adaptively decoupled spectra of the degraded image.
The extracted features are then modulated by a bidirectional operator to facilitate interactions between different frequency components.
Finally, the modulated features are merged into the original input for a progressively guided restoration.
With this approach, the model achieves adaptive reconstruction by accentuating the informative frequency subbands according to different input degradations.
Extensive experiments demonstrate that the proposed method, named AdaIR, achieves state-of-the-art performance on different image restoration tasks, including image denoising, dehazing, deraining, motion deblurring, and low-light image enhancement. Our code and pre-trained models are available at \href https://github.com/c-yn/AdaIRhttps://github.com/c-yn/AdaIR.

###### Keywords:

## 1 Introduction

Image restoration is the task of generating a high-quality clean image by removing degradations (*e.g*., noise, haze, blur, rain) from the original input image. It serves as a vital component in numerous downstream applications across diverse domains, including image/video content creation, surveillance, medical imaging, and remote sensing. Given its inherently ill-posed nature, effective image restoration demands learning strong image priors from large-scale data. To this end, deep neural network-based image restoration approaches [, , , , , , , ] have emerged as preferable choices over the conventional handcrafted algorithms [, , , , , , ]. Deep-learning methods learn image priors either implicitly from data [, , , , , , ], or explicitly by incorporating task-specific knowledge into the network architectures [, , , , , ]. Despite promising results on individual restoration tasks, these approaches are either not generalizable beyond the specific degradation types and levels which hinders their broader application, or require training separate copies of the same network on different degradation types, which is computationally expensive and tedious procedure, and maybe infeasible solution for deployment on resource-constraint edge-devices. Therefore, there is a need to develop an all-in-one image restoration method that can handle images with different degradation types, without requiring prior information regarding the corruption present in the input images.

Figure: Figure 1: *Left*, from top to bottom: degraded images, ground-truth images, and the Fourier spectra of residual images obtained by subtracting the degraded images from the ground-truth images. The images are obtained from LOL-v1 [], SOTS [], Rain100L [], GoPro [], and BSD68 [], respectively. *Right*, the sub-graph illustrates the mean values of Fourier spectra on the square of length shown on the x-axis, across five tasks. The spectra are all resized to $320\times 320$ for comparisons. As seen, different tasks pay different attention to different frequency subbands. For example, there are larger discrepancies in low frequency between degraded and target image pairs of the low-light image enhancement and dehazing datasets. In contrast, the frequency differences are generally evenly distributed for image denoising.
Refer to caption: extracted/5487039/main_fig/teaser/final_use/lol-v1/79_low.png

Recently, an increasing number of attempts have been made [, , , ] to address multiple degradations with a single model. These include employing a degradation-aware encoder in the restoration network learned via contrastive learning paradigm []; designing a two-stage framework IDR [], where the first stage is dedicated to task-oriented knowledge collection based on underlying physics characteristics of degradation types, and the second stage is responsible for ingredients-oriented knowledge integration that progressively restores the image; or developing prompt-learning strategies [, ] inspired from their success in the natural language processing domain [, , ]. Nonetheless, all these approaches purely operate in the spatial domain, and do not consider frequency domain information. However, as illustrated in Fig. 1, we observe that different types of degradations may impact the image content on different frequency subbands. For instance, on the one hand, noisy and rainy images are contaminated with high-frequency content, while on the other hand, low-light and hazy images are dominated by low-frequency degraded content, thus indicating the need to treat each restoration task on its own merits.

In this paper, we propose an adaptive all-in-one image restoration framework based on frequency mining and modulation. Specifically, the frequency mining module extracts different frequency signals from the input features, guided by an adaptive spectra decomposition of the degraded input image. The extracted features are then refined using a bidirectional module, which facilitates the interactions between different frequency components by exchanging complementary information. Finally, these modulated features are used to transform the original input features via an efficient transposed cross-attention mechanism. With the proposed key design choices, our method can learn discriminative degradation context more effectively than other competing approaches, as shown in Fig. 2. Overall, the following are the main contributions of our work.

Figure: Figure 2: The t-SNE results of intermediate features produced by the three-task all-in-one models. Our model is better at learning discriminative degradation contexts.
Refer to caption: x7.png

- •
We propose an adaptive all-in-one image restoration framework that leverages both spatial and frequency domain information to effectively decouple degradations from the desired clean image content.
- •
We introduce the Adaptive Frequency Learning Block (AFLB), which is a plugin block specifically designed for easy integration into existing image restoration architectures. The AFLB performs two sequential tasks: firstly, through its Frequency Mining Module (FMiM), it generates low- and high-frequency feature maps via guidance obtained from the spectra decomposition of the original degraded image; secondly, the Frequency Modulation Module (FMoM) within the AFLB calibrates these features by enabling the exchange of information across different frequency bands to effectively handle diverse types of image degradations.
- •
Extensive experiments demonstrate that our AdaIR algorithm sets new state-of-the-art performance on several all-in-one image restoration tasks, including image denoising, dehazing, deraining, motion deblurring, and low-light image enhancement.

## 2 Related Work

Single-Task Image Restoration. Image restoration is a fundamental task in computer vision that aims to reconstruct a clean image from its degraded counterpart. Since it is a highly ill-posed problem, many conventional methods have been proposed that utilize hand-crafted features and assumptions to reduce the solution space [, ]. Such solutions, though perform well on some datasets, may not generalize well to complicated real-world images []. Recently, with the rapid advancements in deep learning, a great number of convolutional neural network (CNN) based methods have been proposed and attained superior performance over traditional methods on various image restoration tasks, such as image denoising [, ], dehazing [, ], deraining [, ], and motion deblurring [, ]. To model long-range dependencies, Transformer models have been introduced to low-level tasks and significantly advanced state-of-the-art performance [, , ]. Despite the obtained promising performance, these task-specific methods lack generalization beyond certain degradation types and levels. For general image restoration, several network design-based approaches are proposed, which perform favorably on different restoration tasks  [, , , ]. Although these networks demonstrate robust performance on various restoration tasks, they require training separate copies on different datasets and tasks. Furthermore, applying a separate model for each possible degradation is resource-intensive, and often impractical for deployment, especially on edge devices.

All-in-One Image Restoration. All-in-one image restoration methods can address numerous degradations within a single model [, , , ]. Early unified models  [, ] employ distinct encoder and decoder heads to attend to different restoration tasks. However, these non-blind methods need prior knowledge about the degradation involved in the corrupted image in order to channelize it to the relevant restoration head. To achieve blind all-in-one image restoration, AirNet [] learns the degradation representation from the corrupted images using the contrastive learning strategy, and the learned representation is then used to restore the clean image. The subsequent method, IDR [], models different degradations depending on the underlying physics principles and achieves all-in-one image restoration in two stages. Recently, several prompt-learning-based schemes have been proposed [, , , ]. For instance, PromptIR [] presets a series of tunable prompts to encode discriminative information about degradation types, which involve a large number of parameters. Different from the above-mentioned methods, which operate only in the spatial domain, this paper presents an all-in-one image restoration algorithm that exploits information both in spatial and frequency domains.

Figure: Figure 3: (a) The overall pipeline of the proposed AdaIR framework. It is a Transformer-based encoder-decoder architecture, employing novel Adaptive Frequency Learning Blocks (AFLB). Each AFLB contains (b) Frequency Mining Module (FMiM) that extracts different frequency components from input features guided by the adaptively decoupled spectra of the degraded input image, and (c) Frequency Modulation Module (FMoM) that exchanges the complementary information between different frequency features. (d) Cross Attention (CA). (e) Mask Generation Block (MGB) that yields a learnable frequency boundary for spectra decomposition. (f) H-L unit delivers high-frequency attention maps to enrich Low-frequency features. (g) L-H unit enhances high-frequency features by complementing it with low-frequency features. FFT and IFFT denote the Fast Fourier Transform and its inverse operator, respectively.
Refer to caption: x11.png

## 3 Method

Overall Pipeline.
Figure 3 presents the pipeline of AdaIR. The overall goal of our AdaIR framework is to learn a unified model M that can recover a clean image $\hat{\textbf{I}}$ from a given degraded image I, without any prior information of degradation type D present in the input image I.
Formally, given a degraded image I $\in\mathbb{R}^{H\times W\times 3}$, AdaIR first extracts shallow features $\mathbf{Y_{0}}$ $\in$ $\mathbb{R}^{H\times W\times C}$ using a ${3\times 3}$ convolution layer; where ${H\times W}$ denotes the spatial size and $C$ represents the number of channels. Next, these features $\mathbf{Y_{0}}$ are processed through a 4-level encoder-decoder network. Each level of the encoder employs multiple Transformer blocks (TBs) [], where the number of blocks gradually increases from the top level to the bottom level, facilitating a computationally efficient design. The encoder takes high-resolution features $\mathbf{Y_{0}}$ as input, and progressively transforms them into a lower-resolution latent representation $\mathbf{Y}_{l}$ $\in$ $\mathbb{R}^{\frac{H}{8}\times\frac{W}{8}\times 8C}$.
On the decoder side, the latent features $\mathbf{Y}_{l}$ are processed with interleaved Adaptive Frequency Learning Block (AFLB) and TBs to progressively reconstruct high-resolution clean output.
Particularly, between every two levels of the decoder, we insert the AFLB that adaptively segregates the degradation content from the clean image content in the frequency domain, and subsequently assists in refining features in the spatial domain for effective image restoration.

Since different types of degradations affect image content at different frequency bands (as shown in Fig. 1), we specifically design the Adaptive Frequency Learning Block (AFLB) that extracts low- and high-frequency components from the input features and then modulate them to accentuate the corresponding informative subbands for each degradation.
Next, we describe the two key components of AFLB: (1) Frequency Mining Module (FMiM) and Frequency Modulation Module (FMoM).

### 3.1 Frequency Mining Module (FMiM)

As shown in Fig. 3(b), given as inputs both the degraded image I and the intermediate features $\textbf{X}\in\mathbb{R}^{H\times W\times C}$, FMiM mines different frequency representations from X with the guidance of adaptively decoupled spectra of I.
Primarily, FMiM consists of three steps, *i.e*., domain transformation, mask generation, and feature extraction.

For the domain transformation, FMiM applies a $3\times 3$ convolution layer on the degraded image I to expand the channel capacity to align with that of the input features X. These output features are then transformed into spectral domain representation $\textbf{F}\in\mathbb{R}^{H\times W\times C}$ via the Fast Fourier Transform (FFT).

Since we want to adaptively extract different frequency parts from the input features X, we design a lightweight Mask Generation Block (MGB) to generate a 2D mask that serves as a frequency boundary to separate the spectra of input image I. The cutoff frequency boundary adaptively changes according to the type of degradation present in the image. As illustrated in Fig. 3(e), the projected feature map P is first mapped into a vector using a global average pooling operator and then passes through two $1\times 1$ convolution layers with the GELU activation function in between to produce two factors ranging from 0 to 1, which define the mask size by multiplying with the width and height of the spectra. The mask generation process can be formally expressed as:

$$ $[\alpha,\beta]=\delta\left(W^{1\times 1}_{2}\left(\sigma\left(W^{1\times 1}_{1 }\left(\textrm{GAP}_{\textrm{s}}\left(\textbf{P}\right)\right)\right)\right)\right)$ (1) $$

where $\textrm{GAP}_{\textrm{s}}$ denotes spatial global average pooling, $\sigma$ represents the GELU activation function, and $\delta$ indicates the sigmoid function. The convolution weights ${W}_{1}$ and ${W}_{2}$ have the reduction ratios of $r_{1}$ and $\frac{C}{2r_{1}}$, respectively, progressively downsampling the channel dimensions to 2. Subsequently, the binary mask $\textbf{M}_{l}\in\{0,1\}^{H\times W}$ for extracting low frequency can be obtained by setting $\textbf{M}_{l}[\frac{H}{2}-\alpha\frac{H}{k}:\frac{H}{2}+\alpha\frac{H}{k},
\frac{W}{2}-\beta\frac{W}{k}:\frac{W}{2}+\beta\frac{W}{k}]=1$, where $k$ is set to a small value of 128, as the curve junction is relatively small in Fig. 1. Accordingly, the mask for high frequency $\textbf{M}_{h}$ can be obtained by setting the values within the remaining region as 1.
Subsequently, we can obtain the adaptively decoupled features by applying the learned masks to the spectra via element-wise multiplication and using the inverse Fourier transform.

Next, we adapt the multi-dconv head transposed cross attention (Fig. 3(d)) [, ] to mine the different feature parts from the input features with the guidance of $\textbf{F}_{l}$ and $\textbf{F}_{h}$. Overall, the feature extraction process is defined as:
\linenomathAMS

$$ $\displaystyle\textbf{X}_{*}=\textrm{softmax}\left(\textbf{Q}\textbf{K}^{\top}/ \alpha\right)\textbf{V},\quad\quad\textrm{where,}$ (2) $\displaystyle\textbf{Q}={DW}_{1}\left({W}_{3}^{1\times 1}(\textbf{F}_{*}) \right),\textbf{K}={DW}_{2}\left({W}_{4}^{1\times 1}(\textbf{X})\right), \textbf{V}={DW}_{3}\left({W}_{5}^{1\times 1}(\textbf{X})\right),\textrm{where,}$ (3) $\displaystyle\textbf{F}_{*}=\mathcal{F}^{-1}\left(\textbf{M}_{*}\odot\textbf{F }\right),$ (4) $$

where $*\in\{l,h\}$ is an indicator for low/high frequency, ${DW}$ represents a $3\times 3$ depth-wise convolution, $\odot$ is element-wise multiplication, $\mathcal{F}^{-1}$ indicates the inverse fast Fourier transform,
Q, K and V are query, key and value projections, respectively, which are separately generated with a sequential application of ${1\times 1}$ convolution and $3\times 3$ depth-wise convolution, and $\alpha$ is a learnable scaling factor to control the magnitude of the dot product result of Q and K before using the softmax function.

### 3.2 Frequency Modulation Module (FMoM)

We devise FMoM to facilitate the cross interaction between the low-frequency mined features and high-frequency mined features, shown in Fig. 3(c). The goal is to cross complement one type of mined features with the other. For instance, high-frequency features contain edges and fine texture details, and thus we use this information to enrich low-frequency mined features via a super-lightweight spatial attention unit (H-L), depicted in Fig. 3(f). Similarly, the global information present in low-frequency features is passed to the high-frequency branch through the channel attention unit (L-H), illustrated in Fig. 3(g).

H-L Unit: This unit computes the spatial attention map from high-frequency mined features that are then used to complement features of the low-frequency branch. The H-L unit leverages two different channel-wise pooling techniques in parallel to produce two single-channel spatial feature maps, each of size $H\times W\times 1$. These maps are then concatenated along the channel dimension. The concatenated features are further refined with a $7\times 7$ convolution, followed by a sigmoid operation to generate the final spatial attention map, \addedwhich is then used to obtain the modulated low-frequency features via element-wise multiplication. Overall, the process of the H-L Unit is given by:
\linenomathAMS

$$ $\displaystyle\hat{\textbf{X}}_{l}=\textbf{X}_{l}\odot\textbf{A}_{H-L},\quad \quad\textrm{where,}$ (5) $\displaystyle\textbf{A}_{H-L}=\delta\left(W^{7\times 7}_{6}([\textrm{GAP}_{c}( \textbf{X}_{h}),\textrm{GMP}_{c}(\textbf{X}_{h})])\right),$ (6) $$

where $\textbf{W}_{6}$ has a channel reduction ratio of 2. $\delta$ is the sigmoid function. $\textrm{GAP}_{c}$ and $\textrm{GMP}_{c}$ are the channel-wise global average pooling and max pooling, respectively. $[\cdot,\cdot]$ indicates the concatenation operation.

L-H Unit: It is a dual branch module that processes incoming low-frequency mined features, yielding a feature descriptor that is subsequently used to attend to the high-frequency mined features. Specifically, given the mined low-frequency features $\textbf{X}_{l}\in\mathbb{R}^{H\times W\times C}$, the top branch of the L-H unit applies global average pooling along spatial dimension to obtain a feature vector of size $1\times 1\times C$, followed by two convolutional layers with the ReLU activation function in between. The bottom branch of the L-H unit employs the same structure, with the only difference of Max pooling at the head. The results of the two branches are added together, on which the sigmoid function is applied to produce the final attention descriptor $\textbf{A}_{L-H}\in\mathbb{R}^{1\times 1\times C}$, \addedwhich is used to modulate the mined high-frequency features $\textbf{X}_{h}$. The process of the L-H Unit is expressed by:
\linenomathAMS

$$ $\displaystyle\hat{\textbf{X}}_{h}=\textbf{X}_{h}\odot\textbf{A}_{L-H},\quad \quad\textrm{where,}$ (7) $\displaystyle\textbf{A}_{L-H}=\delta\left(W_{8}^{1\times 1}\left(\gamma\left(W _{7}^{1\times 1}(\textrm{GAP}_{s}(\textbf{X}_{l})))\right)+W_{10}^{1\times 1} \left(\gamma(W_{9}^{1\times 1}(\textrm{GMP}_{s}(\textbf{X}_{l}))\right)\right) \right),$ (8) $$

where $\delta$ is the sigmoid function, $\hat{\textbf{X}}_{h}$ is the modulated high-frequency features, $\textrm{GAP}_{s}$ and $\textrm{GMP}_{s}$ represent the global average pooling and max pooling along the spatial dimensions, respectively. $\gamma$ indicates the ReLU activation function. $\textbf{W}_{7}$ and $\textbf{W}_{9}$ have a reduction ratio of $r_{2}$ for the channel adjustment, while $\textbf{W}_{8}$ and $\textbf{W}_{10}$ have an increasing ratio of $r_{2}$. The parameters are shared among $\textbf{W}_{7}$ and $\textbf{W}_{9}$, $\textbf{W}_{8}$ and $\textbf{W}_{10}$ for computational efficiency.

Subsequently, the modulated high-frequency features $\hat{\textbf{X}}_{h}$ and low-frequency features $\hat{\textbf{X}}_{l}$ are aggregated and processed via a $1\times 1$ convolution to obtain $\textbf{X}_{m}$, which is merged into the original input features X using the cross-attention unit, where the query Q tensor is produced from X while $\textbf{X}_{m}$ yields the key K and value V tensors.
By using FMiM and FMoM, the high-frequency and low-frequency contents of the input features are separately and adaptively modulated according to the degradation type present in the corrupted input image, leading to adaptive all-in-one image restoration.

## 4 Experiments

To validate the efficacy of the proposed AdaIR, we conduct experiments by strictly following previous state-of-the-art works [, ] under two different settings: (1) All-in-One, and (2) Single-task.
In the All-in-One setting, a unified model is trained to perform image restoration across multiple degradation types. Whereas, within the Single-task setting, separate models are trained for each specific restoration task.
We provide additional ablation experiments, visual examples, and more details on the architecture in the supplementary material.
In tables, the best and second-best image fidelity scores (PSNR and SSIM []) are highlighted in red and blue, respectively.

Implementation Details.
Our AdaIR presents an end-to-end trainable solution without the necessity for pretraining any individual component. The architecture of AdaIR employs a 4-level encoder-decoder structure, with varying numbers of Transformer blocks (TB) at each level, specifically [4, 6, 6, 8] from level-1 to level-4. We integrate one AFLB block between every two consecutive decoder levels, amounting to a total of three AFLBs in the overall network.

For training, we adopt a batch size of 32 in the all-in-one setting, and a batch size of 8 in the single-task setting. The network optimization is achieved through an L1 loss function, employing the Adam optimizer ($\beta 1=0.9$ and $\beta 2=0.999$), with a learning rate of $2e^{-4}$, over the course of 150 epochs. During the training process, cropped patches sized at ${128\times 128}$ pixels are provided as input, with additional augmentation applied via random horizontal and vertical flips.

Datasets.
In preparing datasets for training and testing, we closely follow prior works [, ]. For single-task image dehazing, we use SOTS [] dataset that comprises 72,135 training images and 500 testing images. For single-task image deraining, we utilize the Rain100L [] dataset, which contains 200 clean-rainy image pairs for training and 100 pairs for testing. For single-task image denoising, we combine images of BSD400 [] and WED [] datasets for model training; the BSD400 encompasses 400 training images, while the WED dataset consists of 4,744 images. Starting from these clean images of BSD400 [] and WED [], we generate their corresponding noisy versions by adding Gaussian noise with varying levels ($\sigma\in\{15,25,50\}$). Denoising task evaluation is performed on the BSD68 [] and Urban100 [] datasets. Finally, under the all-in-one setting, we train a single model on the combined set of the aforementioned training datasets, and directly test it across multiple restoration tasks.

**Table 1: Comparisons under the three-degradation all-in-one setting: a unified model is trained on a combined set of images obtained from all degradation types and levels. On Rain100L [] for image deraining, AdaIR yields 2.27 dB gain over PromptIR [].**
|  | Dehazing | Deraining | Denoising on BSD68 [] |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Method | on SOTS [] | on Rain100L [] | $\sigma=15$ | $\sigma=25$ | $\sigma=50$ | Average |
| BRDNet [] | 23.23/0.895 | 27.42/0.895 | 32.26/0.898 | 29.76/0.836 | 26.34/0.693 | 27.80/0.843 |
| LPNet [] | 20.84/0.828 | 24.88/0.784 | 26.47/0.778 | 24.77/0.748 | 21.26/0.552 | 23.64/0.738 |
| FDGAN [] | 24.71/0.929 | 29.89/0.933 | 30.25/0.910 | 28.81/0.868 | 26.43/0.776 | 28.02/0.883 |
| MPRNet  [] | 25.28/0.955 | 33.57/0.954 | 33.54/0.927 | 30.89/0.880 | 27.56/0.779 | 30.17/0.899 |
| DL [] | 26.92/0.931 | 32.62/0.931 | 33.05/0.914 | 30.41/0.861 | 26.90/0.740 | 29.98/0.876 |
| AirNet [] | 27.94/0.962 | 34.90/0.968 | 33.92/0.933 | 31.26/0.888 | 28.00/0.797 | 31.20/0.910 |
| PromptIR [] | 30.58/0.974 | 36.37/0.972 | 33.98/0.933 | 31.31/0.888 | 28.06/0.799 | 32.06/0.913 |
| AdaIR (Ours) | 31.06/0.980 | 38.64/0.983 | 34.12/0.935 | 31.45/0.892 | 28.19/0.802 | 32.69/0.918 |

Figure: Figure 4: Image dehazing comparisons on SOTS [] between all-in-one methods. Compared to other algorithms, our method is more effective in haze removal.
Refer to caption: extracted/5487039/main_fig/3d-haze/154_c/input_full.png

Figure: Figure 5: Image deraining results on Rain100L [] between all-in-one methods. AdaIR yields high-fidelity rain-free images with structural fidelity and without streak artifacts.
Refer to caption: extracted/5487039/main_fig/3d-rain/air_rain_14_65_c/input_full.png

### 4.1 All-in-One Results: Three Distinct Degradations

We evaluate the performance of our *all-in-one* AdaIR model on three different restoration tasks, including image dehazing, deraining, and denoising. We compare AdaIR against various general image restoration methods (BRDNet [], LPNet [], FDGAN [], and MPRNet []), as well as specialized all-in-one approaches (DL [], AirNet [], and PromptIR []).
Table 1 shows that the proposed AdaIR provides consistent performance gains over the other competing approaches.
When averaged across various restoration tasks and settings, our AdaIR obtains $0.63$ dB PSNR gain over the recent best method PromptIR [], and $1.49$ dB improvement over the second best algorithm AirNet [].
Specifically, compared to PromptIR [], AdaIR yields a substantial boost of $2.27$ dB on the deraining task, and $0.48$ dB on the dehazing task.
We provide visual examples in Fig. 4 for dehazing, Fig. 5 for deraining, and Fig. 6 for denoising. These examples show that our AdaIR is effective in removing degradations, and generates images that are visually closer to the ground truth than those of the other approaches [, ]. Particularly, in the restored images, our method preserves better structural fidelity and fine textures.

### 4.2 Single Degradation One-by-One Results

Consistent with previous works [, ], we further evaluate AdaIR under the *single-task* experimental protocol. To this end, we train separate copies of AdaIR model for each distinct restoration task. Table 2 reports dehazing results; compared to the previous all-in-one approaches PromptIR [] and AirNet [], our method obtains PSNR gains of $0.49$ dB and $8.62$ dB, respectively. Similarly, on the deraining task, our AdaIR advances the state-of-the-art [] by $1.86$ dB as shown in Table 3. A similar performance trend can be observed in image quality scores provided in Table 4 for denoising.

Figure: Figure 6: Image denoising comparisons on BSD68 [] between all-in-one methods. The image reproduction quality of our AdaIR is more visually faithful to the ground truth.
Refer to caption: extracted/5487039/main_fig/3d-noise/1_c/input_full.png

**Table 2: Dehazing results in the single-task setting on the SOTS-Outdoor [] dataset. Compared to PromptIR [], our method generates a 0.49 dB PSNR improvement.**
|  | DehazeNet | MSCNN | AODNet | EPDN | FDGAN | AirNet | Restormer | PromptIR | AdaIR |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | [] | [] | [] | [] | [] | [] | [] | [] | (Ours) |
| PSNR | 22.46 | 22.06 | 20.29 | 22.57 | 23.15 | 23.18 | 30.87 | 31.31 | 31.80 |
| SSIM | 0.851 | 0.908 | 0.877 | 0.863 | 0.921 | 0.900 | 0.969 | 0.973 | 0.981 |

**Table 3: Deraining results in the single-task setting on the Rain100L [] dataset. Our AdaIR obtains a significant performance boost of 1.86 dB PSNR over PromptIR [].**
|  | DIDMDN | UMR | SIRR | MSPFN | LPNet | AirNet | Restormer | PromptIR | AdaIR |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | [] | [] | [] | [] | [] | [] | [] | [] | (Ours) |
| PSNR | 23.79 | 32.39 | 32.37 | 33.50 | 33.61 | 34.90 | 36.74 | 37.04 | 38.90 |
| SSIM | 0.773 | 0.921 | 0.926 | 0.948 | 0.958 | 0.977 | 0.978 | 0.979 | 0.985 |

**Table 4: Denoising results in the single-task setting on Urban100 [] and BSD68 []. On Urban100 [] for the noise level 50, AdaIR yields a 0.31 dB gain over PromptIR [].**
|  | Urban100 | BSD68 |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Method | $\sigma=15$ | $\sigma=25$ | $\sigma=50$ | $\sigma=15$ | $\sigma=25$ | $\sigma=50$ | Average |
| CBM3D [] | 33.93/0.941 | 31.36/0.909 | 27.93/0.840 | 33.50/0.922 | 30.69/0.868 | 27.36/0.763 | 30.80/0.874 |
| DnCNN [] | 32.98/0.931 | 30.81/0.902 | 27.59/0.833 | 33.89/0.930 | 31.23/0.883 | 27.92/0.789 | 30.74/0.878 |
| IRCNN [] | 27.59/0.833 | 31.20/0.909 | 27.70/0.840 | 33.87/0.929 | 31.18/0.882 | 27.88/0.790 | 29.90/0.864 |
| FFDNet [] | 33.83/0.942 | 31.40/0.912 | 28.05/0.848 | 33.87/0.929 | 31.21/0.882 | 27.96/0.789 | 31.05/0.884 |
| BRDNet [] | 34.42/0.946 | 31.99/0.919 | 28.56/0.858 | 34.10/0.929 | 31.43/0.885 | 28.16/0.794 | 31.44/0.889 |
| AirNet [] | 34.40/0.949 | 32.10/0.924 | 28.88/0.871 | 34.14/0.936 | 31.48/0.893 | 28.23/0.806 | 31.54/0.897 |
| PromptIR [] | 34.77/0.952 | 32.49/0.929 | 29.39/0.881 | 34.34/0.938 | 31.71/0.897 | 28.49/0.813 | 31.87/0.902 |
| AdaIR (Ours) | 34.96/0.953 | 32.74/0.931 | 29.70/0.885 | 34.36/0.938 | 31.72/0.897 | 28.49/0.813 | 32.00/0.903 |

**Table 5: Comparisons for five-degradation all-in-one restoration. Denoising results are reported for the noise level $\sigma=25$. The top super-row methods denote the general image restoration approaches, and the rest are specialized all-in-one approaches. On SOTS [] for dehazing, AdaIR attains a remarkable gain of 5.29 dB over IDR [].**
|  | Dehazing | Deraining | Denoising | Deblurring | Low-Light |  |
| --- | --- | --- | --- | --- | --- | --- |
| Method | on SOTS | on Rain100L | on BSD68 | on GoPro | on LOL | Average |
| NAFNet [] | 25.23/0.939 | 35.56/0.967 | 31.02/0.883 | 26.53/0.808 | 20.49/0.809 | 27.76/0.881 |
| HINet [] | 24.74/0.937 | 35.67/0.969 | 31.00/0.881 | 26.12/0.788 | 19.47/0.800 | 27.40/0.875 |
| MPRNet [] | 24.27/0.937 | 38.16/0.981 | 31.35/0.889 | 26.87/0.823 | 20.84/0.824 | 28.27/0.890 |
| DGUNet [] | 24.78/0.940 | 36.62/0.971 | 31.10/0.883 | 27.25/0.837 | 21.87/0.823 | 28.32/0.891 |
| MIRNetV2 [] | 24.03/0.927 | 33.89/0.954 | 30.97/0.881 | 26.30/0.799 | 21.52/0.815 | 27.34/0.875 |
| SwinIR [] | 21.50/0.891 | 30.78/0.923 | 30.59/0.868 | 24.52/0.773 | 17.81/0.723 | 25.04/0.835 |
| Restormer [] | 24.09/0.927 | 34.81/0.962 | 31.49/0.884 | 27.22/0.829 | 20.41/0.806 | 27.60/0.881 |
| DL [] | 20.54/0.826 | 21.96/0.762 | 23.09/0.745 | 19.86/0.672 | 19.83/0.712 | 21.05/0.743 |
| Transweather [] | 21.32/0.885 | 29.43/0.905 | 29.00/0.841 | 25.12/0.757 | 21.21/0.792 | 25.22/0.836 |
| TAPE [] | 22.16/0.861 | 29.67/0.904 | 30.18/0.855 | 24.47/0.763 | 18.97/0.621 | 25.09/0.801 |
| AirNet [] | 21.04/0.884 | 32.98/0.951 | 30.91/0.882 | 24.35/0.781 | 18.18/0.735 | 25.49/0.846 |
| IDR [] | 25.24/0.943 | 35.63/0.965 | 31.60/0.887 | 27.87/0.846 | 21.34/0.826 | 28.34/0.893 |
| AdaIR (Ours) | 30.53/0.978 | 38.02/0.981 | 31.35/0.889 | 28.12/0.858 | 23.00/0.845 | 30.20/0.910 |

**Table 6: Image denoising results of directly applying the pre-trained model under the five-degradation setting to the Urban100 [], Kodak24 [] and BSD68 [] datasets. The results are PSNR scores. On Urban100 [] for the noise level $\sigma=25$, AdaIR produces a significant performance gain of 0.39 dB PSNR over IDR [].**
|  | Urban100 | Kodak24 | BSD68 |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | $\sigma=15$ | $\sigma=25$ | $\sigma=50$ | $\sigma=15$ | $\sigma=25$ | $\sigma=50$ | $\sigma=15$ | $\sigma=25$ | $\sigma=50$ | Average |
| DL [] | 21.10 | 21.28 | 20.42 | 22.63 | 22.66 | 21.95 | 23.16 | 23.09 | 22.09 | 22.04 |
| Transweather [] | 29.64 | 27.97 | 26.08 | 31.67 | 29.64 | 26.74 | 31.16 | 29.00 | 26.08 | 28.66 |
| TAPE [] | 32.19 | 29.65 | 25.87 | 33.24 | 30.70 | 27.19 | 32.86 | 30.18 | 26.63 | 29.83 |
| AirNet [] | 33.16 | 30.83 | 27.45 | 34.14 | 31.74 | 28.59 | 33.49 | 30.91 | 27.66 | 30.89 |
| IDR [] | 33.82 | 31.29 | 28.07 | 34.78 | 32.42 | 29.13 | 34.11 | 31.60 | 28.14 | 31.48 |
| AdaIR (Ours) | 34.10 | 31.68 | 28.29 | 34.89 | 32.38 | 29.21 | 34.01 | 31.35 | 28.06 | 31.55 |

### 4.3 Additional All-in-One Results: Five Distinct Degradations

Following the recent work of IDR [], we further verify the effectiveness of AdaIR by performing experiments on five restoration tasks: dehazing, deraining, denoising, deblurring, and low-light image enhancement. For this, we train an all-in-one AdaIR model on combined datasets gathered for five different tasks. These include datasets from the aforementioned three-task setting as well as additional datasets: GoPro [] for motion deblurring, and LOL-v1 [] for low-light image enhancement.

Table 5 shows that AdaIR achieves a $1.86$ dB gain compared to the recent best method IDR [], when averaged across five restoration tasks. Particularly, the performance improvement is over $5$ dB on dehazing. Table 6 reports denoising results on three different datasets with various noise levels. It can be seen that our method performs favorably well compared to the other competing approaches.

**Table 7: Ablation studies for the proposed components. Fixed uses a fixed square mask with sides of 10. FLOPs are measured on the patch size of $256\times 256\times 3$.**
|  |  | FMiM | FMoM |  |  | Overhead |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Net | Baseline | Fixed | MGB | L-H | H-L | PSNR | SSIM | Params. | FLOPs |
| (a) | ✓ |  |  |  |  | 28.21 | 0.966 | 26.13M | 141.24G |
| (b) | ✓ | ✓ |  |  |  | 29.79 | 0.969 | 27.73M | 145.09G |
| (c) | ✓ | ✓ |  | ✓ |  | 30.37 | 0.975 | 28.74M | 147.44G |
| (d) | ✓ | ✓ |  | ✓ | ✓ | 30.52 | 0.976 | 28.74M | 147.44G |
| (e) | ✓ |  | ✓ | ✓ | ✓ | 31.24 | 0.978 | 28.77M | 147.45G |

### 4.4 Ablation Studies

In this section, we conduct ablation studies to test the impact of various individual components to the overall performance of AdaIR.
All ablation experiments are performed on the image dehazing task by training models for 20 epochs.

Impact of individual architecture modules.
Table 7 summarizes the performance benefits of individual architectural contributions. Table 7(b) demonstrates that the proposed frequency mining mechanism (FMiM) brings gains of $1.58$ dB PSNR over the baseline model, using only a fixed mask to decompose the spectra of input images.
Furthermore, the L-H unit boosts the performance to $30.37$ dB PSNR; see Table 7(c). It can be seen in Table 7(d) that we use both L-H and H-L units, and the performance reaches $30.52$ dB PSNR. Finally, Table 7(e) shows that the overall AdaIR brings $3.03$ dB improvement over the baseline, while incurring a small computational overhead of 2.64M parameters and 6.21 GFlops. These results corroborate the effectiveness of our design.

Strategies for spectral decomposition.
We carry out this ablation to test different strategies to segregate low- and high-frequency representations from the degraded input images. We compare the proposed mask-guided adaptive frequency decomposition approach with the Average pooling and Gaussian filtering strategies. Results are provided in Table 9. Following  [], we use average pooling to obtain the low-frequency features which are then subtracted from the input features to obtain the high-frequency features. This strategy provides PSNR of $30.59$ (see column 1 in Table 9), which is $0.65$ dB lower than our method. Similarly, when we switch to the Gaussian filter of size ${5\times 5}$, the model achieves only $30.22$ dB PSNR (second column). In contrast, our method of applying a flexible mask for Fourier spectra decomposition performs the best, yielding $31.24$ dB.

**Table 8: Spectra decomposition methods.**
| Methods | Average Pooling | Gaussian Filter | Ours |
| --- | --- | --- | --- |
| PSNR | 30.59 | 30.22 | 31.24 |
| SSIM | 0.976 | 0.976 | 0.978 |

**Table 10: Results on the unseen desnowing task with the CSD [] dataset.**
| Method | AirNet [] | PromptIR [] | Ours |
| --- | --- | --- | --- |
| PSNR | 19.32 | 20.47 | 20.54 |
| SSIM | 0.733 | 0.7638 | 0.7643 |

Frequency representation mining at image-level vs. feature-level.
Each AFLB block in AdaIR decoder receives the original degraded image as input, on which FMiM applies the procedure of spectra decomposition. To verify the efficacy of this design, we switch to using the input embedding features $\mathbf{X}$ (rather than degraded image) for frequency representation. This ablation result in Table 9 shows a performance drop from $30.52$ dB to $29.29$ dB, indicating that the raw input image offers better discriminative information about the degradation for effective spectra separation.

Generalization to out-of-distribution degradations. To show the generalization ability of our AdaIR, we take the all-in-one model trained on the three-task setting, and directly test it under two different scenarios: (1) unseen degradation type, and (2) multi-degraded images.
Table 11 shows that, on the unseen task of image desnowing, AdaIR provides more favorable results than other approaches.
We create a mixed degradation dataset by adding Gaussian noise (level $\sigma=50$) to the rainy images of Rain100L []. Table 11 depicts that our method is more robust in the mixed degradation scenes than PromptIR [] and AirNet [].

## 5 Conclusion

This paper introduces AdaIR, an all-in-one image restoration model capable of adaptively removing different kinds of image degradations.
Motivated by the observation that different degradations affect distinct frequency bands, we have developed two novel components: a frequency mining module and a frequency modulation module.
These modules are designed to identify and enhance the relevant frequency components based on the degradation patterns present in the input image.
Specifically, the frequency mining module extracts specific frequency elements from the image’s intermediate features, guided by an adaptive decomposition of the input’s spectral characteristics that reflect the underlying degradation.
Subsequently, the frequency modulation module further refines these elements by facilitating the exchange of complementary information across different frequency features.
Incorporating the proposed modules into a U-shaped Transformer backbone, the proposed network achieves state-of-the-art performance on a range of image restoration tasks.

## Appendix 0.A Additional Ablation Studies

AFLBs in encoder and decoder? We run an experiment to assess the feasibility of employing AFLB modules on either the encoder side, decoder side, or both. Table 12 shows that utilizing AFLBs in both the encoder and decoder leads to notable performance degradation compared to AFLBs solely integrated into the decoder.

**Table 12: Comparisons of image dehazing under the single-task setting: between the use of AFLBs on either the encoder-side, decoder-side, or both.**
|  | Dehazing on SOTS [] |  |
| --- | --- | --- |
| Method | PSNR | SSIM |
| Encoder+Decoder+AFLB | 29.70 | 0.973 |
| AdaIR (Ours) | 30.52 | 0.976 |

Placement of AFLB in the network. Next, we conduct an ablation experiment to study where to place AFLBs in our hierarchical network. Table 13 demonstrates that employing only one AFLB (between level 1 and level 2) leads to a deterioration in the network’s performance (29.58 dB in top row). Conversely, integrating AFLBs between every consecutive level of the decoder yields the best performance.

**Table 13: AFLB position. Results are reported on the SOTS [] dataset.**
| Method | PSNR | SSIM |
| --- | --- | --- |
| Level 2 | 28.58 | 0.973 |
| Level 2+3 | 29.83 | 0.975 |
| Level 2+3+4 | 30.52 | 0.976 |

Figure: Figure 7: Different choices for FMoM. (a) Using widely adopted spatial attention [] to modulate different frequency features, where the attention map is generated without discriminating different frequency inputs. (b) Using specially designed attention units to exchange complementary information across different frequency features. GAP and GMP denote the global average pooling and global max pooling, respectively. The experiments are conducted on image dehazing under the single-task setting.
Refer to caption: x12.png

Design choices of FMoM.
We investigate different choices for the frequency modulation module (FMoM). As shown in Fig. 7(a), we leverage the commonly used spatial attention [] to modulate different frequency features without discriminating different inputs. Overall, the process is formally given by:
\linenomathAMS

$$ $\displaystyle\hat{\textbf{X}}=\textbf{X}_{h}\odot\textbf{A}_{h}+\textbf{X}_{l} \odot\textbf{A}_{l},\quad\quad\textrm{where},$ (9) $\displaystyle\textbf{A}_{h},\textbf{A}_{l}=\textrm{Split}\left(\delta( \widetilde{\textbf{A}})\right),\quad\quad\textrm{where},$ (10) $\displaystyle\widetilde{\textbf{A}}=W^{7\times 7}\left([\textrm{GAP}([\textbf{ X}_{h},\textbf{X}_{l}]),\textrm{GMP}([\textbf{X}_{h},\textbf{X}_{l}])]\right)$ (11) $$

where $\odot$ represents element-wise multiplication, Split indicates splitting the features among the channel dimension, $\delta$ is the Sigmoid function, $W^{7\times 7}$ is a $7\times 7$ convolution, and $[\cdot,\cdot]$ is a concatenation operator. GAP and GMP are global average pooling and global max pooling among the channel dimensions, respectively. The experiments are performed on the image dehazing task under the single-task setting.
This variant achieves only 29.67 dB PSNR, which is 0.85 dB lower than our FMoM, shown in Fig. 7(b), indicating the effectiveness of our design.

Combinations of different degradations.
We investigate the influence of various combinations of degradation types on model performance, as presented in Table 14. As expected, including more degradation types make it increasingly difficult for the model to perform restoration. Notable, hazy images in a combined dataset lead to a larger performance drop than rainy or noisy images. One reason could be that the aim of the restoration model in deraining and denoising tasks is to focus more on restoring high-frequency content (noise, rain), whereas, in the dehazing task the goal is to focus on removing low-frequency (hazy) content.

**Table 14: Ablation studies on the combinations of degradations for the three-task setting. Results are presented in the form of PSNR (dB)/SSIM.**
| Degradation | Denoising on BSD68 | Deraining on | Dehazing |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Noise | Rain | Haze | $\sigma=15$ | $\sigma=25$ | $\sigma=50$ | on Rain100L | on SOTS |
| ✓ |  |  | 34.36/0.938 | 31.72/0.897 | 28.49/0.813 | - | - |
|  | ✓ |  | - | - | - | 38.90/0.985 | - |
|  |  | ✓ | - | - | - | - | 31.80/0.981 |
| ✓ | ✓ |  | 34.31/0.938 | 31.67/0.896 | 28.42/0.811 | 38.22/0.983 | - |
| ✓ |  | ✓ | 34.11/0.935 | 31.48/0.892 | 28.19/0.802 | - | 30.89/0.980 |
|  | ✓ | ✓ | - | - | - | 38.44/0.983 | 30.54/0.978 |
| ✓ | ✓ | ✓ | 34.12/0.935 | 31.45/0.892 | 28.19/0.802 | 38.64/0.983 | 31.06/0.980 |

Figure: Figure 8: Architectural details of the Transformer Block (TB) used in the AdaIR framework. TB involves two elements: multi-dconv head transposed attention (MDTA) and gated-dconv feed-forward network (GDFN).
Refer to caption: x14.png

## Appendix 0.B Computational Comparisons

Table 15 shows that the proposed AdaIR strikes a better tradeoff between accuracy and complexity than other all-in-one competing methods.

**Table 15: Computational comparisons of all-in-one methods under the three-degradation setting. Average PSNR across three tasks is reported here (see Table 1 of the main paper for more detailed results). FLOPs are measured on the patch size of $256\times 256\times 3$.**
|  | Params. | FLOPs | PSNR |
| --- | --- | --- | --- |
| Method | (M) | (G) |  |
| AirNet [] | 8.93 | 311 | 31.20 |
| PromptIR [] | 35.59 | 158.4 | 32.06 |
| AdaIR | 28.77 | 147.45 | 32.69 |

## Appendix 0.C Transformer Block in the AdaIR Framework

In the AdaIR framework, we use Transformer Blocks (TB) based on the design proposed in []. Fig. 8 presents its architectural details. It consists of two successive components, multi-dconv head transposed attention (MDTA) and gated-dconv feed-forward network (GDFN).

MDTA first normalizes the input $\textbf{X}\in\mathbb{R}^{H\times W\times C}$ using a layer normalization operator [], and then generates the query (Q$\in\mathbb{R}^{H\times W\times C}$), key (K$\in\mathbb{R}^{H\times W\times C}$), and value (V$\in\mathbb{R}^{H\times W\times C}$) projections using combinations of $1\times 1$ convolution and $3\times 3$ depth-wise convolution layers. The transposed-attention map of size $C\times C$ is yielded by applying the Softmax function to the dot-product results of the reshaped query and key projections. Overall, the process of MDTA is given by:
\linenomathAMS

$$ $\displaystyle\hat{\textbf{X}}=W^{1\times 1}_{1}\textrm{Attention}\left(\textbf {Q}^{\prime},\textbf{K}^{\prime},\textbf{V}^{\prime}\right)+\textbf{X},\quad \quad\textrm{where,}$ (12) $\displaystyle\textrm{Attention}\left(\textbf{Q}^{\prime},\textbf{K}^{\prime}, \textbf{V}^{\prime}\right)=\textbf{V}^{\prime}\cdot\textrm{Softmax}\left( \textbf{K}^{\prime}\cdot\textbf{Q}^{\prime}/\alpha\right),$ (13) $$

where $\hat{\textbf{X}}$ is the output of MDTA. $W_{1}^{1\times 1}$ denotes a $1\times 1$ convolution. $\alpha$ is a learnable factor to control the magnitude of the dot product result of K and Q. $\textbf{Q}^{\prime}$, $\textbf{K}^{\prime}$ and $\textbf{V}^{\prime}$ are obtained by reshaping tensors from the original size $\mathbb{R}^{H\times W\times C}$.

Similarly, GDFN first applies a layer normalization operator to normalize the input $\textbf{X}\in\mathbb{R}^{H\times W\times C}$. The result then passes through two branches, each including a $1\times 1$ convolution with a factor $\gamma$ to expand channels, followed by a $3\times 3$ depth-wise convolution layer. Two branches converge using element-wise multiplication after activating one branch via a GELU function. Overall, the GDFN process is formally expressed as:
\linenomathAMS

$$ $\displaystyle\hat{\textbf{X}}=W_{2}^{1\times 1}\textrm{Gating}(\textbf{X})+ \textbf{X},\quad\quad\textrm{where,}$ (14) $\displaystyle\textrm{Gating}(\textbf{X})=\phi\left(DW^{3\times 3}_{1}\left(W_{ 3}^{1\times 1}(\textrm{LN}(\textbf{X}))\right)\right)\odot DW^{3\times 3}_{2} \left(W_{4}^{1\times 1}(\textrm{LN}({\textbf{X}}))\right),$ (15) $$

where LN is the layer normalization, $\odot$ denotes element-wise multiplication, $DW^{3\times 3}$ represents a $3\times 3$ depth-wise convolution, and $\phi$ indicates the GELU non-linearity.

## Appendix 0.D Additional Visual Results

In this section, we first provide the t-SNE result of our method under the five-degradation setting in Fig. 9. It can be seen that our method is capable of discriminating degradation contexts for five different degradation types. It is worth noting that the cluster for low-light image enhancement is closer to the dehazing cluster than others, suggesting the effectiveness of our model, since these two degradation types mainly impact the image content on low-frequency components.

Figure: Figure 9: The t-SNE result of our model under the five-degradation setting.
Refer to caption: x15.png

Finally, we provide more qualitative results of the all-in-one setting and single-task setting for three image restoration tasks, including image deraining, dehazing, and denoising.

Figure: Figure 10: Image deraining comparisons on Rain100L [] under the three-degradation setting.
Refer to caption: extracted/5487039/supp_fig/3d-derain/7_34_c/input.png

Figure: Figure 11: Image dehazing comparisons on SOTS [] under the three-degradation setting.
Refer to caption: extracted/5487039/supp_fig/3d-haze/16_c/input.png

Figure: Figure 12: Image denoising comparisons on BSD68 [] with $\sigma=50$ under the three-degradation setting.
Refer to caption: extracted/5487039/supp_fig/3d-denoise/7_c/input_full.png

Figure: Figure 13: Image draining comparisons under the single task setting on Rain100L [].
Refer to caption: extracted/5487039/main_fig/single-rain/0_24/24.png

Figure: Figure 14: Image dehazing comparisons under the single task setting on SOTS [].
Refer to caption: extracted/5487039/main_fig/single-haze/0_0330_0.8_0.08/input.png

Figure: Figure 15: Image denoising results under single task setting on BSD68 [] with $\sigma=50$.
Refer to caption: extracted/5487039/main_fig/single-denoise/0_223061_c/input_full.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Address weaknesses above.
soundness: 3
strengths: The paper highlights a key insight for any image restoration: Noisy and rainy images are contaminated with high-frequency content while the low-frequency content degrades more in Low-light and hazy images.

Exploiting frequency domain is a key differentiation, as previous methods like AirNet, IDR, and PromptIR operate solely on spatial domain information

Frequency Mining Module (FMiM) extracts relevant frequency signals from degraded images through adaptive spectral decomposition, targeting degradation-specific frequency content. The use of a dynamic, learnable Mask Generation Block (MGB) to separate low- and high-frequency representations based on the input image's degradation type allows AdaIR to adapt to different degradations effectively

Frequency Modulation Module (FMoM) refines these frequency features by enabling interaction between different frequency components, enhancing the model’s effectiveness on various degradation types.

Experiments on a wide set of benchmarks demonstrates consistent performance gains over competing approaches in both three-degradation and five-degradation settings

The improved quality of discriminative feature learning (shown in Fig. 2) is impressive.
summary: The paper presents AdaIR, which aims to address multiple forms of degradation, such as noise, blur, haze, and rain, within a single model. AdaIR is motivated by the observation that different types of degradation impact image content on different frequency subbands.  AdaIR leverages both spatial and frequency domain information to effectively separate and remove these degradations using customised modules.
weaknesses: While the paper demonstrates the generalisation ability of AdaIR by testing it on an unseen desnowing task and synthetic images with mixed degradations (rain and noise) in Table 7 and 8, there are no experiments on real out-of-distribution images. Extensive qualitative comparisons on real images should be included. Evaluation on a wider range of out-of-distribution degradations would strengthen the claims of the model's generalisation capabilities. 

The quantitative tables primarily compare AdaIR with other all-in-one methods but not the scores of single-task trained state-of-the-art models. Including the highest scores achieved by SotA single task models on each individual task would provide a more comprehensive assessment of AdaIR's performance relative to specialised approaches.

While Table 4 shows the overhead of AFLB in terms of parameters and FLOPs, a more in-depth analysis of its computational cost and potential impact on real-time applications would be beneficial.
FLOPs are only reported for ablation studies, but there are no FLOPs comparisons with competing methods.
In addition to FLOPs, there should also be comparison between inference time of different models.
using frequency domain processing inherently introduces additional processing for transformations eg IFFT. 
Although they are beneficial for accuracy, a potential drawbacks of such approaches is the computation cost and incompatibility with edge platforms. 

The paper doesn't cite or compare with any existing models that use frequency domain processing. There have been many such studies [refs 1-7 below]. Some of these papers are quite related as they propose transformer modules that focus on high/low frequencies.

Furthermore, comparing with alternative transform-domain methods, such as wavelet transforms or other adaptive filtering techniques, commonly used in related papers, could enhance the performance and adaptability of AdaIR.

The MGB dynamically generates a mask to separate low and high frequencies based on the input image. It would be interesting to see the difference in generated masks for different tasks. This vanilla approach could be prone to overfitting, especially since the training dataset does not adequately cover the diversity of real-world degradations. A more robust approach might involve specifically incorporating prior knowledge about frequency characteristics of different degradations or exploring regularization techniques to prevent overfitting

Paper lacks deeper analysis on how the interaction between different frequency bands contributes to the restoration process. Such analysis could lead to more informed design choices and potentially improve the interpretability of the model.

The graph in Fig. 1 right doesn’t clearly demonstrate the stark frequency-domain differences between different tasks.


[1] Seeing the Unseen: A Frequency Prompt Guided Transformer for Image Restoration (ECCV 2024)

[2] Selective Frequency Network for Image Restoration (ICLR 2023)

[3] Image Restoration via Frequency Selection, TPAMI 2024

[4] Intriguing Findings of Frequency Selection for Image Deblurring, AAAI 2023

[5] Efficient Frequency Domain-based Transformer for High-Quality Image Deblurring, CVPR 2023 

[6] LoFormer: Local Frequency Transformer for Image Deblurring [ACM MM 2024]

[7] Learning Frequency Domain Priors for Image Demoireing, TPAMI 2022

---

confidence: 5
contribution: 3
initial_rating: 8
presentation: 3
questions: Please refer to the weaknesses.
soundness: 4
strengths: 1) The paper presents a novel adaptive all-in-one image restoration architecture that can effectively decouple the input into different frequencies and modulate them for high-quality reconstruction. The paper is well-motivated, and Figure 1 makes sense.
2) The proposed blocks are reasonable and can be easily injected into other algorithms.
3) The visualizations well demonstrate the efficacy of the proposed model and motivation.
4) The model is computationally efficient. It achieves sota, and the experiments on two kinds of all-in-one settings are thorough, while other algorithms only experiment on a single setting.
summary: This paper presents an all-in-one image restoration framework that can generate clean images from multiple degradation patterns, such as noise, blur, haze, and rain. Different from existing all-in-one algorithms that perform only in the spatial domain, the proposed network deals with all-in-one tasks from the perspectives of frequency mining and modulation. The network extracts different frequencies from the degradation images and modulates features for high-quality reconstruction. The adaptive strategy enhances the learning of informative frequency signals based on input degradation. Comprehensive experiments on two kinds of all-in-one settings demonstrate the model's effectiveness.
weaknesses: 1) The comparisons using only PSNR/SSIM seem insufficient to measure the quality of the resulting images. LPIPs can be used to further verify the proposed model's superiority.
2) The architecture is mainly based on Restormer. However, Tab. 1 does not include Restormer's performance.
3) The paper lacks comparisons with recent methods, such as Harmony in Diversity: Improving All-in-One Image Restoration via Multi-Task Collaboration.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: - Could the authors provide both the original and enlarged images in the figures? If this is not possible, assessing the performance of the proposed methods becomes challenging. For instance, while AdaIR appears to excel in rain removal, other models might perform better in different regions.
- The total number of parameters appears large (28.77M). Can authors add a column to their comparison tables (e.g. Tables 1 and 2) showing the number of parameters for each model? This would allow for a more comprehensive comparison of model efficiency alongside performance metrics.
- Can the authors clarify the term "Fixed" in Table 4? It is unclear whether the mask value is also fixed or not. The current caption in Table 4 only mentions that the mask shape is fixed at $10 \times 10$.
- Could the authors demonstrate that AdaIR, as an all-in-one model, performs well on unseen and mixed degradation using the UDC-SIT dataset [8], which features complex degradations?
- It would be helpful if the authors could explain more clearly the significance of the channel-wise features and the corresponding attention scores depicted in Figure 7.
- Can the authors elaborate on why AdaIR outperforms the Encoder + Decoder + AFLB configuration presented in Table 12? The explanation should include the advantages of AdaIR's design compared to the alternative.
- The additional ablation study regarding the design choices of FMoM in Appendix B may not be entirely fair. For example, the proposed design in Figure 8(b) does not initially concatenate $X_h$​ and $X_l$. Instead, each feature passes through the H-L or L-H unit before being cross-multiplied. The reviewer believes that the spatial attention comparison in Figure 8(a) should also avoid concatenation initially; instead, $X_h$​ and $X_l$​ should go through channel attention separately before their respective spatial attention maps are cross-multiplied. Could the authors provide a comparison based on this suggestion?

[8] Ahn, K., Ko, B., Lee, H., Park, C., & Lee, J. (2024). UDC-SIT: a real-world dataset for under-display cameras. Advances in Neural Information Processing Systems, 36.
soundness: 2
strengths: - The motivation for incorporating frequency information in an all-in-one image restoration model is well-founded. The authors substantiate their claim that various degradation types influence image content across frequency subbands. For instance, the t-SNE visualizations in Figures 2 and 11 illustrate how degradations can be clustered based on their types.
- The authors successfully demonstrate the function of each module within the model through extensive ablation studies. For example, in Figure 9, the visualizations of intermediate features and spectra effectively highlight the roles of the AFLB module, which is composed of the FiMM and FMoM components.
summary: This paper proposes an adaptive all-in-one image restoration model called AdaIR using frequency information. A key motivation is that different degradation types impact the image’s different frequency sub-bands. AdaIR mines low- and high-frequency information from input features (FMiM) and modulates the extracted features by facilitating the cross-interaction between low-frequency mined features and high-frequency mined features (FMoM).
weaknesses: - The authors claim that all previous approaches operate solely in the spatial domain and do not consider frequency information (L086-L087). However, several studies [1-2] have already explored the frequency domain. The authors should clarify how their approach differs from or advances these existing frequency-based methods. Given the existence of prior frequency domain exploration, the primary motivation for this work may need to be revised.
- The proposed AdaIR appears to be a straightforward combination of existing techniques, such as Restormer [3] and CBAM [4]. Specifically, all encoders and decoders utilize multiple Transformer blocks from Restormer, and the cross attention (CA) mechanisms in both FMiM and FMoM leverage the MDTA approach from Restormer. Additionally, the H-L unit in FMoM seems to replicate the channel attention mechanism found in CBAM. 
- The authors should appropriately cite the sources in both the figures and the main text. For example, in Figure 3, Restormer should be referenced for the TB in (a) and the CA in (d). Furthermore, CBAM should be cited since its channel attention mechanism is employed in the H-L unit of FMoM, both in Figure 3 and the main text. The authors should add citations to Restormer and CBAM in the caption of Figure 3, as well as in the relevant sections of the main text where these components are described.
- The roles of the H-L and L-H units appear to be similar in facilitating cross-interaction between low and high-frequency mined features. However, the authors have implemented different designs for these two units. It would be helpful for the authors to clarify the rationale behind using distinct methods for each.
- The metrics in the tables require clarification. For instance, Table 1 should include column headings that explicitly indicate the inclusion of PSNR and SSIM. The authors should add column headers in Table 1 (and any other relevant tables) to clearly indicate which metrics are being reported (e.g. "PSNR / SSIM").
- The benchmark models referenced are somewhat outdated. The authors should consider updating the comparison models to more recent ones [5-7]. Additionally, they should include comparisons with all-in-one restoration models that account for frequency information [1-2].


[1] Wen Y, Zhang K, Zhang J, Chen T, Liu L, Luo W. Frequency-oriented efficient transformer for all-in-one weather-degraded image restoration. IEEE Transactions on Circuits and Systems for Video Technology. 2023 Jul 27.

[2] Shi Z, Su T, Liu P, Wu Y, Zhang L, Wang M. Learning Frequency-Aware Dynamic Transformers for All-In-One Image Restoration. arXiv preprint arXiv:2407.01636. 2024 Jun 30.

[3] Zamir SW, Arora A, Khan S, Hayat M, Khan FS, Yang MH. Restormer: Efficient transformer for high-resolution image restoration. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition 2022 (pp. 5728-5739).

[4] Woo S, Park J, Lee JY, Kweon IS. Cbam: Convolutional block attention module. InProceedings of the European conference on computer vision (ECCV) 2018 (pp. 3-19).

[5] Ai Y, Huang H, Zhou X, Wang J, He R. Multimodal Prompt Perceiver: Empower Adaptiveness Generalizability and Fidelity for All-in-One Image Restoration. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2024 (pp. 25432-25444).

[6] Park D, Lee BH, Chun SY. All-in-one image restoration for unknown degradations using adaptive discriminative filters for specific degradations. In2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2023 Jun 17 (pp. 5815-5824). IEEE.

[7] Yang H, Pan L, Yang Y, Liang W. Language-driven All-in-one Adverse Weather Removal. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2024 (pp. 24902-24912).

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: Some works as following also have discussed using the frequency domain perspective to solve various degradation problems. [1] and [2] observe a specific degradation from the perspective of frequency domain.  [3] and [4] also adopt frequency mining and modulation to solve various degradations condition. It is suggested  to provide a point-by-point comparison of the proposed method with these works, highlighting key similarities and differences in approach and performance.

[1] Efficient Frequency Domain-Based Transformers for High-Quality Image Deblurring, CVPR, 2023
[2] Wavelet-based Fourier Information Interaction with Frequency Diffusion Adjustment for Underwater Image Restoration, CVPR, 2024
[3] Hybrid Frequency Modulation Network for Image Restoration, IJCAI, 2024
[4] Selective Frequency Network for Image Restoration, ICLR, 2023
soundness: 2
strengths: The AdaIR model offers significant advantages by adaptively addressing various types of image degradations through innovative frequency analysis. Its frequency mining module precisely extracts relevant frequency components that reflect underlying degradations, while the frequency modulation module optimizes these components by allowing the exchange of complementary information across different features. Integrated into a U-shaped Transformer backbone, these components enable AdaIR to achieve state-of-the-art performance across diverse image restoration tasks, highlighting its robust adaptability and effectiveness.
summary: This work solves the all-in-one problem in image restoration from a frequency perspective and demonstrates its effectiveness. But my main problem is that I think the insights and designed modules are fallen behind and there are already many homogenized work. Therefore, the novelty of this paper does not appeal to me.
weaknesses: 1. In the t-sne clustering in Figure 2, the model is better at learning discriminative degradation contexts. First, the comparison method is relatively old, so some more recent comparison methods is suggested in the t-SNE analysis.

2. Moreover, I do not think that this completely separated state can help solve the all-in-one problem more effectively. In fact, the correlation between various degradations may need to be learned to solve it better. For example, an image of rainy scene should also have heavy haze, so these two degradations should have something in common. And the rain lines can be regarded as a kind of noise, so they may also be correlated in AirNet and PromptIR of Fig 2, rather than completely decoupled.  In other word, I doubt that the reason for the decoupling of the t-sne analysis may be the overfitting of the model on each dataset.

3. As the core motivation of this work, the idea in Figure 1 has been discussed extensively, and I think it is not innovative. Please see the question section for detailed discussion.

4. The method design is also based on the combination of the very traditional U-Net network and the attention mechanism. This design has obviously fallen behind. Diffusion model may be a more proper framework to solve this problem.

Meta-review: All reviewers agree on the innovation and novelty of the proposed work. Reviewers had individual questions regarding the presentation of the results, which the authors responded rigorously. This led to reviewers increasing their initial score after the rebuttal to the average score of 6.
Therefore the paper is recommended for acceptance.

Decision: Accept (Poster)