Parameter Efficient Adaptation for Image Restoration with Heterogeneous Mixture-of-Experts
Authors: Hang Guo
Tao Dai
Yuanchao Bai
Bin Chen
Xudong Ren
Zexuan Zhu
Shu-Tao Xia
Venue: NeurIPS 2024 poster (2024)
Keywords: Image Restoration
Parameter-efficient Transfer Learning
Primary area: machine_vision
TL;DR: This paper proposes the first parameter efficient adaptation framework, which tunes only 0.6% parameters to adapt pre-trained restoration models to various tasks.

Abstract: Designing single-task image restoration models for specific degradation has seen great success in recent years. To achieve generalized image restoration, all-in-one methods have recently been proposed and shown potential for multiple restoration tasks using one single model. Despite the promising results, the existing all-in-one paradigm still suffers from high computational costs as well as limited generalization on unseen degradations. In this work, we introduce an alternative solution to improve the generalization of image restoration models. Drawing inspiration from recent advancements in Parameter Efficient Transfer Learning (PETL), we aim to tune only a small number of parameters to adapt pre-trained restoration models to various tasks. However, current PETL methods fail to generalize across varied restoration tasks due to their homogeneous representation nature. To this end, we propose AdaptIR, a Mixture-of-Experts (MoE) with orthogonal multi-branch design to capture local spatial, global spatial, and channel representation bases, followed by adaptive base combination to obtain heterogeneous representation for different degradations. Extensive experiments demonstrate that our AdaptIR achieves stable performance on single-degradation tasks, and excels in hybrid-degradation tasks, with training only 0.6% parameters for 8 hours.

Parameter Efficient Adaptation for Image Restoration with Heterogeneous Mixture-of-Experts
Hang Guo!
Tao Dai*2
Yuanchao Bai
Bin Chen
Xudong Ren
Zexuan Zhu
Shu-Tao Xia
TTinnhua University
SSenzzen University
3 Harbin Institute of Technology
*Peng Cheng Larboratory
com/ htt.://ww..oo//ccc///////aaaaa/a csguoh/Adapt
Abstract
Designing single-task image restoration models for specific degradation has seen great success in recent years. To achieve generalized image restoration, all-in-one methods have recently been proposed and shown potential for multiple restoration tasks using one single model. Despite the promising results, the existing all-in-one paradigm still suffers from high computational costs as well as limited generaliza- tion on unseen degradations. In this work, we introduce an alternative solution to improve the generalization of image restoration models. Drawing inspiration from recent ad advancements in Parameter Efficient Transfer Learning (PETL). we aim to tune only a small number of parameters to adapt pre-trained restoration models to various tasks. However, current PETL methods fail to generalize across varied restoration tasks due their homogeneous representation nature. To this end, we propose AdaptIR, Mixture-of Experts (MoE) with orthogonal multi-branch design to capture local spatial, global spatial, and channel representation bases, followed by adaptive base combination to obtain heterogeneous representation for different degradations. Extensive experiments demonstrate that our AdaptIR achieves stable performance on degleddegraoation tasks, and excels in hybrid-degrraattinn tasks, with fine-tuning only 0.6% parameters for hours.
Introduction
Image restoration, aiming to restore high-quality images from their degraded counterparts, is fundamental computer vision problem and has been studied for many years. Due to its ill-posed nature, early research efforts [1, 2, 3] typically focus on developing single-task models, with each model handling only one specific degradation. Consequently, these methods often exhibit limited generalization across different image restoration tasks.
To improve generalization ability, all-in-one image restoration methods [4, 5, 6] have recently been proposed and have attracted great research interest. By training one model with multiple degradation data, these methods enable the single model handle various degradations. Despite the promising results, the existing all-in-one paradigm still faces several challenges. Firstly, the all-in-one model can only restore degradations encountered during training; once training is complete, the model cannot handle new degradations. Secondly, since the knowledge of restoring multiple degradations is learned by single model, it incurs a significant cost to train and store these all-in-one models.
In this work, we propose an alternative solution to improve the generalization ability of restoration models in handling multiple degradations. Drawing inspiration from Parameter Efficient Transfer Learning (PETL) [7. 8, 9. 10], we aim to insert a small number of trainable modules into frozen pre-trained restoration backbones. By training only these newly added modules on downstream tasks, the preitrrined restoration backbone can be adapted to unseen restoration tasks. Since only small
Corresponding author: Tao Dai (daitao.edu gomil.com).
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

51 Low www.gggiiagggggg... Entrcemert
D2 Hyrr begsddriinnnaaaaa
Touree www.soonnligee
fraffor anales Dars
undaate axteor
02:0
144
insorppratty
20:00
2013
0us
Figure 1: (a)&(b) We find directly applying the current PETL methods to image restoration leads (c) The current PETL method suffers sub-optimal
to unstable performance on single degradation. results on hybrid degradation which requires heterogeneous representation. (d)&(e) We use Fourier analysis to visualize Adapter and our AdaptIR and find that Adapter exhibits homogeneous frequency representations even when faced with different degradations, while our AdaptIR can adaptively learn eegradationnspeciffe heterogeneous representations. We provide more evidence Appendix I.
number of parameters need to be trained, the training cost is very small and the training process can converge quickly when new tasks are added. When the training is completed, only the newly added parameters need to be stored, thus greatly reducing the storage cost.
Despite the potential of applying PETL techniques image restoration, our experiments reveal that existing PETL methods can work normally on specific degradation, but fail to generalize across multiple degradations, exhibiting unstable performance when adapted to different restoration tasks. As shown in Fig. I(a). the most widely used PETL method, Adapter [8], performs well on the draining task. However, when applying Adapter to the low-light image enhancement task, the adapted model shows significant performance degradation. This phenomenon also occurs with other methods, such as the recent offte-of-thh-arr PETL method FacT [10] 1(b)). This confusing phenomenon motivates us to discover possible reasons.
To this end, we design preliminary experiments, in which we fine-tune the pre-trained restoration model [1]] using existing PETL schemes, and then use Fourier analysis [12] to observe the frequency characteristics of features from these methods. It is observed that the features from current PETL methods exhibit homogeneous representation across different restoration tasks (see Fig. 1(d)) As demonstrated in previous work [6], different restoration tasks prefer certain representations for optimal results, we thus hypothesize that the performance drop occurs when the representation needed to address one specific degradation does not match the homogeneous representation of existing PETL methods. To verify this hypothesis, we further test current PETL methods using the hybrid degradation task (Fig. I()),, which requires heterogeneous representations to handle diverse degradations, and we find all existing approaches suffer severe performance drops. Based on the above experiments, we argue that the homogeneous representation of existing PETL methods hinders stable performance on single degradation tasks and advanced performance on hybrid degradation tasks.
In order to learn heterogeneous representations across tasks, one possible solution the multi- branch structures, where each branch is designed to learn orthogonal representation bases, and then adaptively combine these bases for specific degradation. Following this idea, we propose AdapUR, a heterogencous Mixture-of-Expert (MoE) to adapt pre-trained restoration models with heterogeneous representations across tasks. Our AdaptIR adopts orthogonal multi-branch design to learn local spatial, global spatial, and channel representation bases. Specifically, The Local Interaction Module (LIM) employs depth-separane exploit local spatial representation. We then employ the Frequency Affine Module (FAM), which performs frequency affine transformation to introduce global spatial modeling ability. Additionally, the Channel Gating Module (CGM) is adopted to capture channel interactions. Finally, we utilize the Adaptive Feature Ensemble to dynamically fuse these three representation bases for specific degradation. Thanks to the heterogeneous representation modeling, our AdaptR achieves stable performance on single-degradation tasks and advanced performance on -egrddddgroittion tasks.
convolution with kernel weight decomposition to
The contributions of our work are as follows: (i) We propose novel PETL paradigm to improve the generalization for image restoration models, and further investigate the specific challenges when applying existing PETL methods to low-level restoration. (ii) We introduce AdaptIR. custom PETL method that employs a heterogeneous MoE for orthogonal representation modeling. To the best of our knowledge, this is the first work to explore parameter-efficient adaptation for image restoration. (iii) Experiments on various downstream tasks demonstrate that AdaptIR achieves robust performance on single degradation tasks and advanced results on hybrid degradation tasks.

---

2 Related Work
2.1 Generalized Image Restoration
Image restoration has attracted lot of research interest in recent years. Due to the challenging ill-posed nature, some early research paradigms typically study each sub-task in image restoration independently and have recently achieved favorable progress in their respective fields [13, 14, 15, 2, 16]. However, designing such a single-task model is cumbersome. and does not consider the similarities among different tasks. Recently, all-in-one image restoration [4. 5, 6] has offered a way to improve the generalization of image restoration models. By training one single model on multiple degradations, allows the model to have the ability to handle multiple degradations. For example, AirNet [4] proposes two-stage training scheme to first learn the degradation representation, which then used in the following restoration stage. PromptIR [5] utilizes prompt learning to obtain degradation-specinn prompts to train the model in end-to-end manner. Despite the progress, the current all-in-one restoration paradigm can only deal with degradation seen during training and is inevitable to re-train the model when needing to add new degradations. In addition, incorporating the knowledge of handling multiple degradations into one model has to increase the model size, which causes large training and storage costs.
2.2 Parameter-Efficien Transfer Learning
Parameter efficient transfer learning, which initially came from with NLP [17, 18, 19, 8, 20, 21, 22, 231, aims catch up with full fine-tuning by training a small number of parameters. Recently, this technique has emerged in the field of computer vision with promising results [9, 7, 24, 10, 25, 26, 27, 28, 29, 30, 31]. For example, VPT [9] adds learnable tokens, also called prompts, to the input sequence of one frozen transformer layer. Adapter [17] employs bottleneck structure to adapt the pre-trained model. Some attempts also introduce parameterized hypercomplex multiplication layers [22] and re-parameterisattoon [32] to adapter-based methods. Moreover, LoRA [21] utilizes the low-rank nature of the incremental weight attention and performs matrix decomposition for parameter efficiency. He et al. [23] go further identify all the above three approaches from a unified perspective. In addition, NOAH [26] and GLoRA [25] introduce Neural Architecture Search (NAS) to combine different methods. SSF [24] performs learnable affine transformation features of the pre-trained model. FacT [10] tensorizes ViT and decomposes the increments into lightweight factors. Although applying PETL methods to pre-trained image restoration models to improve the generalization seems promising, we find that current PETL methods suffer from homogeneous representations when facing different degradations, hindering stable performance across tasks.
3 Method
3.1 Preliminary
In this work, we aim adapt pre-trained restoration models multiple downstream tasks by fine-tuning a small number of parameters. Following existing PETL works, we mainly focus on transformer-based restoration models since transformer has been shown to be suitable for pre- training [33] and there is no CNN-based pre-trained model available. As shown in Fig. 2, a typical pre-trained restoration model [11, 34] usually contains one large transformer body as well as task- specific heads and tails. Given the pre-assigned task type, the low-quality image !LQ will first go through the corresponding head get the shallow feature X After that, Xhead is flattened into ID sequence on the spatial dimension and is input to the transformer body which contains several stacked transformer blocks with each block containing multiple transformer layers [35]. Finally, skip connection is adopted followed by the task-specific tail to reconstruct the high-quality image THQ- During the pre-training stage, gradients from multiple tasks are used to update the shared body as well as the corresponding task-specific head and tail. After pre-training the restoration model, previous common practice fine-tunes all parameters of the pre-trained model for specific downstream tasks, which burdens training and storage due to the per-task model weights.
3

---

Toaslama Layse
Thunsfann Bock
Therffrre Book
Tholk moll-posiie
CNN
Mailts 2ood
Au A BETARRRNNNN
aalesponffic Taal
Transforme Block
Large
AdI
Ro leeaatiee Nedal
(MPCIMERR Nistae
Copocaaion
Ieleradion
Lovelll NAME:
fifiee
Fovele
Characl
Staft
comprete
Allae
Affica
poolin
NLP
H-t tdlethee Fostare Froeeddy
An illustration of the proposed AdaptIR. Our AdaptR is placed parallel to the frozen MLP in one transformer layer and thus can seamlessly inserted into various transformer- based pre-trained restoration models.
Figure 2:
3.2 Heterogeneous Representation Learning
To obtain stable performance across multiple restoration tasks, it is crucial to allow the learning of heterogeneous representation for different degradations. To this end, we formalize AdaptR as a multi-branch MoE structure, where each branch learns representations orthogonal to each other form the representation bases, and then these bases are adaptively combined achieve degradation-speccie
representation. Formally, as shown in Fig. 2(a), since the transformer body flattens the I-th layer feature into 1D token sequence, we first restore the 2D image structure to After that, we apply the 1 x convolution with channel reduction rate Y to transfer X; to low-dimension space for parameter efficiency and obtain the intrinsie feature Xintrin E R xW Then three parallel branches are orthogonally designed to learn local spatial, global spatial, and channel bases. Next, bases from these three branches are adaptively ensembled to obtain representation X X-adapt for specific degradation. Finally, X- added output of the frozen MLP to adapt the pre-trale pre-trained restoration models. Details of the three branches given below.
obtain X; E IR CB XV
Local Interaction Module. We first introduce the Local Interaction Module (LIM) to model the local spatial representation. As shown in Fig. 2(b). the proposed LIM implemented by the depth-wiss wise convolution with weight factorization for parameter efficiency. Specifically, given the convolution weight W E C x Cout
group x x where Cin. Cout are input and output channel, is the kernel size and group is the number of convolution groups, first reshape W into 2D weight matrix W' E IR Cin x Caut X²
group
and then decompose W' into multiplication of two low-rank weight matrices:
= T
where U Xr V IR group K²x and is the rank trade-off performance and efficiency. Then we reshape W' to the original kernel size and use it to convolve X Xintrin to get X.IM,
X Reshape(W") Xintrin
where @ denotes convolution operator, and Reshape(-) transforms 2D matrices into 4D convolution kernel weights.
Frequency Affine Module. We then consider modeling global spatial to achieve orthogonal spatial modeling to LIM. A possible solution to introduce attention mechanism [35] which a global receptive field. However, the attention comes at the cost of high complexity. which goes against principle of parameter efficiency. In this work, we resort to the frequency domain for a solution. Specifically, apply the Fast Fourier Transform (FFT) on X Xintrin to obtain the corresponding

---

frequency feature map X C x
H
W-
(f( HW
X
xintrin (h, w)e 2 "0 W.)
h=0
w=0
As can be seen from Eq.3, good property of FFT is that each position of the frequency feature map the weighted sum of all features in the spatial domain. Therefore, performing pixel-wise projection on X is equivalent to performing global operator in the spatial domain.
Motivated by this observation, we propose the Frequency Affine Module (FAM) to take advantage of the inherent global representation in X (see Fig. 2(c)). Concretely, we perform the affine transformation on amplitude map Magi and phase map Pha, respectively with septh-separable 1 x 1 convolution. To ensure numerical stability during the early training stages, we initialize the transformation layers as all-one weights and zero bias. Subsequently, the inverse Fast Fourier Transform (iFFT) is applied to convert the affined feature back to the spatial domain. Finally, another Septh-separable x convolution used as a scale layer for subsequent feature ensemble. short, the whole process can be formalized as:
[Mag Phai] = FFT(X
(4)
XFAM Conv (ttp///FFTT(ooooooe(((eeooo complex(01 o2((h)))
where (.). and (2(-) are the frequency projection function and to co_omllee((.))) converts the magnitude and phase to complex numbers.
Channel Gating Module. The above LIM and FAM both adopt the depth-separable strategy for further develop the Channel Gating Module (CGM) for salient channel selection. As shown in Fig. 2(d), we first obtain the spatial weight mask Mi E 1xx by employing 1 convolution which compresses the channel dimension of Xintrin to 1, followed by the Softmax on the spatial dimension:
parameter efficiency. To allow for another orthogonal representation,
Mi = (ofttaax(oovv((("t (Ximtrin)).
(5)
We then apply M₁ on each channel of X;intrin to perform spatially weighted summation to obtain the channel vector which will go through a Feed Forward Network (FFN) to generate the channel gating factor X ER- x1x1.
XCGM = FFNN Σ Mi @ Xintrin),
h,w
where & denotes the Hadamard product.
Adaptive Feature Ensemble. The orthogonal modeling from local spatial, global spatial, and channel serve favorable representation bases, and we further introduce the Adaptive Feature Ensemble to learn their combinations to obtain degradation-specifee representations. As shown in Fig. 2(e), we convolution to compress the channel of X to 1 to extract its spatial details, while applying global average pooling and FFN X,FM to preserve the global information. Next, the sigmoid activation is employed to generate dynamic weights for multiplication and produces the adaptive spatial features X, xppatiil After that, we use XCGM to perform channel selection on x; obtain the degradation-speciff ensemble feature Xensem E
x-spatinl to
C. H
At last, a 1 x 1 convolution is employed to up-dimension the Xensem to generate X-ddapt E RBCxHX
For stability, we zero to initialize the convolution weights. Then, the x,ddppt is added to the output of frozen MLP as residual to adapt pre-trained models to downstream tasks.
3.3 Parameter Efficient Training
During training, we freeze all parameters in the pre-trained model, including task-specific heads and tails as well the transformer body except for the proposed AdaptRR, simple L1 loss is employed to provide pixel-level supervision:
Cpis = | TIQlh,
where ||
denotes L1 norm.
5

---

4 Experiment
We first employ single-degradation restoration tasks to assess the performance stability of differ- ent PETL methods, including image SR, color image denoising, image deraining, and low-light enhancement. Subsequently, we introduce hybrid degradation to further evaluate the ability to learn heterogeneous representations. In addition, we compare with recent all-in-one methods in both effectiveness and efficiency to demonstrate the advantages of applying PETL for generalized image restoration. Finally, we conduct ablation studies to reveal the working mechanism of the proposed method as well as different design choices. Since evaluating the performance stability requires experiments multiple single-degradation tasks, due to the page limit, the related experiments can be seen in the Appendix E.I.
4.1 Experimental Settings
Datasets, For image SR, we choose DIV2K [36] and Flickr2K [37] as the training set, and we
evaluate on Set5 [38], Set14 [39], ESDDS100 [40]. Ur [41], and Manga109 [42]. For color
image denoising, training sets consist of DIV2K [36], Flickr2K [37], BSD400 [40], and WED [43], and we have two testing sets: CBSD68 [44] and 1rban100 [41]. For image deraining, we evaluate using the Rain 100L [15] and Rain 100H [15] benchmarks, corresponding to light/heavy rain streaks. For evaluation on hybrid degradation, where one image contains multiple degradation types, choose two representatives, consisting of low-resolution and noise as well as low-resolution and JPEG artifact compression, and we add noise or apply JEPG compression on the low-resolution images to synthesize second-order degraded images. For low-light image enhancement, we utilize the training and testing set of LOLvI [45].
Evaluation Metrics. We use the PSNR and SSIM to evaluate the effectiveness. The PSNR/SSIM of image SR, deraining, and second-order degradation are computed the Y channel from the YCbCr space, and we evaluate the RGB channel for denoising and low-light image enhancement. Moreover, we use trainable #param to measure efficiency.
Baseline Setup. This work focuses on transferring pre-trained restoration models to downstream tasks under low parameter budgets. Since there is little work studying PETL on image restoration, we reproduce existing PETL approaches and compare them with the proposed AdapUR. Specifically, we include the following representative PETL methods: i) VPT [9], where the learnable prompts are inserted as the input token of transformer layers, and we compare Deep [9] in experiments because of its better performance. ii) Adapter [17], which introduces bottleneck structure placed after Attention and MLP. iii) LoRA [21], which adds parallel sub-networks to learn low-rank incremental matrices of query and value. iv) AdaptFormer [7], which inserts tunable module parallel to MLP. v) SSF [24], where learnable scale and shift factors are used to modulate the frozen features. vi) FacT [10], which tensorises ViT and then decomposes the incremental weights. We also present results of vii) full line-tuning (Full-ft), and viii) directly applying pre-trained models to downstream tasks (Pretrain), to provide more insights. For readers unfamiliar with PETL, we have also provided basic background introduction in Appendix A.
Implementation Details. We use two pre-trained transformer-basee restoration models, f.e., IPT [11] and EDT [34], as the base models to evaluate different PETL methods. We control tunable parameters by adjusting channel reduction rate 7. We use AdamW W [46] as the optimizer and train for 500 epochs. The learning rate initialized to le-4 and decayed by half at (250,400,450,477) epochs. All experiments are conducted on four NVIDIA 3080Ti GPUs.
4.2 Comparison on Hybrid Degradation Tasks
In order to obtain convincing evaluation results, it is tedious and time-consuming to observe the stability of one particular PETL method multiple single-degradation tasks. Here, we introduce dybrid-degradation restoration. Since restoring hybrid degraded images requires a heterogeneous representation of the PETL methods, and thus the hybrid degradation is more suitable for evaluation.
In this work, we consider the second-order degradation as a representative of hybrid degradation. Specifically, we employ two different types of second-order degradations, i.e., the x4 low-resolution and noise with o=30 (denoted as 1.44& Noise30) as well as the x4 low-resolution and JEPG com- pression with quality factor q=30 (denoted as LR4&JPEG30), Moreover, we also include the classic

---

Table 1: Quantitative comparison for hybrid-degradation restoration tasks. The best and the second best results are in red and blue.
Set5
Set14
BSDS100
Urban
Manga 109
Method
Degradation
#paaam
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
Full-ft
I No
119M
27.24 0.7859
25.56 0.6686
25.02
0.6166
24.02
0.6967
26.31 0.8245
Pretrain
3R4&Noisc30
19.74 0.3569 19.27 0.3114
19.09
0.2783
18.54 0.3254
19.75 0.3832
SSF [24]
3R4&Noisc30
373K
25.41 0.6720
24.02 0.5761
24.06
0.5411
21.89 0.5514
23.33 0.6736
VPT [9]
LR4&Noisc30
884K
24.11 0.5570 22.97 0.4722
22.91
0.4336
21.20 0.4527
22.61 0.5570
Adapter [8]
R4
691K
25.60 0.6862 24.16 0.5856
24.17
0.5498
22.05 0.5640
23,61 0.6914
LoRA [21]
R44&Noisc30
995K
25.19 0.6371 23.82 0.5405
23,82
0,5026
21.81 0.5193
23,30 0.6396
Adaptfor 77]
R4&NNoicc3
677K
26.10 0.7138 24.58 0.6095
24.44
0,5686
22.52 0.5976
24,38 0.7296
FacT [10]
LR4&Noisc30
537K
25.70 0.6963
24.24 0.5944
24.25
0,5586
21.10 0.5727
23.63 0.6993
MoE
LR4&Noisc30
667K
26.35 0.7335 24.80
0.6254
24.59
0,5835
22.77 0.6188
24.73 0.7517
Ours
LRR4&Noisc30
697K
26.48
0.7441
24.88
0.6345
24.67
0.6279
22.88
0.5932
24.96 0.7625
Full-ft
LR4&JPEG30
119M
27.21
0.7778
25.49
0.6563|
25.08
0.6076
23.54 0.6687
25.48 0.7971
Pretrain
LR4&JPEG30
25.23 0.6702
24.12
0.5917
24.19
0,5627 21,74 0,5654
22.93 0.6732
SSF [24]
LR4&JPEG30
373K
26.26 0.7321
24.81
0.6285
24.71
0,5882 22.44 0.6085
23,92 0.7350
VPT
LR&JJPEG30
884K
26.63 0.7497
25.14 0.6414
24.89
0.5974 22.96 0.6377
24.53 0.7591
Adapter [8]
LR4&JPEG30
691K
26.73 0.7554
25.22
0.6448
24.92 0.5999 23.09 0.6447
24.74 0.7677
LoRA [21]
LR4&JPEG30
995K
26.64
0.7501 25.17
0.6424
24.91 0,5983 23.02 0.6405
24.64 0.7619
Adaptfor. [7]
LR4&JPEG30
677K
26.74 0.7562 23.08
0.6441
25.22
0.6447 24.92 0.5996
24.72 0.7669
FacT [10]
LR4&JPEG30
537K
26.71 0.7557 25.22 0.6450)
24.93
0.5998 23.08 0.6446
24.74 0.7681
MoF.
LR4&JPEGG3
667K
26.80
0.7590
25.26
0.6465
24.04
0.6009
23.14 0.6477
24.81
0.7708
Ours
LR4&/PEG30
697K
26.91
0.7646
25.34
0.6502
24.98
0.6032
23.25
0.654]
25.02 0.7791
HR
Bicuboc
VPT 20 20L5408
HR
Hicuti
VPT 21501B
UHha1000 x4): img. 091
Adapter 21. 2dn LoRA 21.01dB Ours 215MB
10hon100(x4):: img. 060
A daapter 23.034B LoRA 22,52:0 Ours ZI9HB
Figure 3: Visual comparison on hybrid degradation with LR4&Noise30. We provide more visualiza- tion in Appendix E.I.
MoE 48,,48,44 49], which also employs the multi- structure but the design of each branch is the same, to give the impact of the multi- structure on the performance.
Tab. gives the results. Consistent with the previous analysis in Fig. 1, existing PETL methods suffer severe performance drops hybrid degradation tasks due to the difficulty of learning heterogeneous representations. Interestingly, even the simple MoE baseline which only uses the multi- multi-branch struc- ture outperforms the current state-of-the-an art PETL methods, suggesting that multi-branch structures are promising for heterogeneity across tasks. However, since each branch of the classical MoE employs the same structure, it struggles to capture orthogonal representation bases from different branches. In contrast, our method achieves consistent state-of-ther the-art performance across all tasks and on all datasets. For example, AdaptIR outperforms the sfate-of-the-art PETL method FacT [10] by 1.78dB on Urban 100 with LR4&NNise300 and 0.28dB on Manga109 with LR4&JEPG30. By orthogonally designing branches to obtain representation bases and then adaptively combining them, our AdaptIR allows for heterogeneous representations across different tasks. We also give several visual results in Fig. and our AdaptIR can well handle complex degradation.
4.3 Comparison with All-in-One Methods
Recently, all-in-one image restoration methods [5, 4], which learn a single restoration model for various degradations, have shown to be a promising paradigm in achieving generalized image restoration. Here, we compare our AdaptR with these methods on both single-task and multi-task setups in Tab. 2. For the single-task setting, our method achieves better PSNR results, e.g. 0.31 higher than PromptIR on denoising a=50. In addition, the performance advantage of our AdaptIR still preserves the multi-task setup. For instance, our AdaptIR outperforms PromptIR by even 4.9dB PSNR and 0.016 SSIM on light rain streak removal. This is because all-in-one methods need to learn multiple degradation restoration within one model, resulting in learning difficulties, and the problem of negative transfer among different tasks [50] can also lead to performance degradation. By contrast, the heterogencous representation from the orthogonal design facilitates the stable performance of our
7

---

Table 2: Comparison with all one image restoration methods under single-task setting. The
"training time' of AdaptRR refers to the downstream fine-tuning time excluding the pre- pre-training stage.
Method
task
datasct
#param
training time
GPU memory
PSNR
SSIM
AirNet [4]
light derain
Rain100L
8.75M
488h
-1GG
34.90
0,977
PromptR [5]
light deruin
Rain
97M
84h
1288
37.04
0.979
Ours
light derain
Rain100L.
000
697K
~8h
~8G
37.81
0.981
AirNet [4]
denoise ==50
Urban 100
8.75M
4-8h
G1G
28.88
0.871
PromptRR [5]
denoise 0=50
Urban100
97M
8-4
128G
29.39
0.881
Ours
denoise o=50
Urban100
697K
~8h
88
29.70
0.881
Table 3: Comparison with all-in-one image restoration methods under multi-task setting.
Method
#param
GPU
training time
light
denoise r=25
denoise
memory
derain
e=30
AirNet [4]
8.7M
11G
448h
967
31.90/09114 28.68/0.86
PrompiR [5]
97.1M
~ 128G
488h 36.37/0.972
32.09/0.919 28.99/0.871
Ours
697K
8G
10h
41.27/0.988 32.64/0.926 29.16/0.875
Figure 4: Fouriur analysis on outputs from LIM and FAM.
Figure 5: Channel activation visualization
outputs from CGM.
46-
40.
Ohere Actiot on maaes
Feature from Local Inter Mediula
www
Feature from Frequency Affine Module
Channd Indes
35
AdaptIR across different degradations. As for efficiency, our AdaptIR only trains 0.7% parameters than that of PromptIR with a fast fine-tuning process. We provide a detailed summarization and discussion about the existing multi-task restoration paradigm in Appendix C.
4.4 Discussion
Why does the Proposed Methods Work? Our proposed AdaptIR adopts the heterogeneous MoE structure to allow diverse representation learning. Here, we delve deep to verify whether the model design can indeed influence the learned features. For LIM and FAM, we visualize the frequency characteristics of their outputs in Fig. 4. It can be seen that LIM's relative log amplitude higher than FAM, suggesting it has learned to capture high-frequency local textures. Meanwhile, more than 95% of energy is centralized within 0.05 for FAM, indicating it can well model low-frequency global structure. For CGM, we visualize the channel activation in Fig. 5, and find large activation differences across channels, with large variance of 96.10, indicating that the CGM learns to select degradation-spect speciiic channels.
is 11.02
Scaling Trainable Parameters. We compare the performance of different PETL methods under varying parameter budgets. We use the hybrid degradation LR4& Noise30 in this setup. Fig. 6 shows the results. It can be seen that the proposed method surpasses other strong baselines across various parameter settings, demonstrating the strong scalability of the proposed method.
How About the Performance on Other Pre-trained Models? The above experiments employ IPT [11] as the base model. In order to verify the generalization of the proposed method, we further adopt another pre-trained image restoration model EDT [34] as the frozen base model. Tab. 4 represents the results. It can be seen that the proposed method maintains state-of-the-art performance by tuning only 1.5% parameters. More experiments with EDT can be seen in Appendix E.2.
4.5 Ablation Study
Parameter Efficient Designs. In this work, we introduce several techniques to achieve orthogonal representation learning. Here, we ablate to study the impact of these choices. The results, presented in Tab. 5, indicate that (1) the adaptive feature ensemble can assemble representations according to specific degradation, without which will cause performance drop. In addition, (2)&(3) removing the
8

---

Table 4: Comparison on generalization ability with more pretrained base model.
Figure 6: Scalability comparison with dif- ferent PETL methods.
Method
#param
Set5
Set14
BSDS
Urban
Manga 109
Deg
Degracacion Manga109
100
100
22-
Full-ft
11.6
27.32
25 60
25.03
24.10
26.42
Pretrain
19.29
18.45
18.27
17.92
19.25
SSF[24]
117K
26.92 25.24
24.83
23,41
25.77
POMRURN
VPT [9]
311K
24.19 22.91
22.81
21.12
22.49
Adapter [17]
26.92 25.27
23,48
25.84
Cu's
Ours
LoRA [21]
259K
26.91 25.25
24,80
23,46
25.81
Adaptar
Adapiter
AdaptFor. [7]
162K
26.99 25.31
24.85
23.59
25.95
LoftA
LoRA
[IcT[10]
174K
26 89
25.25
24.81
23,43
25.78
Presrain
FacT
Durs
173K
27.04 25.34
24.87
23.60
25.97
Thiratie ParsmebenjMi
Tranab ParamaterssM)
Table 5: Ablation of different design choices on PSNR(dB), 'Baseline' refers to the setting of depth- separable projection LIM and FAM, as well as the channel-spatial orthogonal modeling.
Settings
#param
Set5
Set14 Urban
(0)Baseline
697K
26.48
24,88
22,88
(1) adaptive feature ensemble
692K
24.26 24.88
22.82
(2) depth-separable LIM
718K
25.67
24.22
22.10
(3)+w/o depth-separable FAM
728K
25.73
24.28
22.17
(4)+ CGM&w/o depth-separable
743K
24.26 23.15
21.36
Table 6: Ablation experiments of different components on PSNR(dB).
Table 7: Ablation for different insertion posi- tions and forms on PSNR(dB).
LIM FAM CGMM]Hpaam Set5 Set14 U0
position
form
Sett
Sec1d Urhan 100
680K
23.64 22,66
20.97
MLP parallel
26.48 24.88
22.88
682K
25.52
24.17
22.05
Attention
parallel
26.28 24.70
22.59
678K
26.11 24.60
22.52
MLP
sequential
26.35 24.77
22.67
697K
26.48 24.88
22.88
Atlention sequential
25.60 24.19
22.07
depth-seppe sepahaleperable design in LIM or FAM will conflict with the channel modeling in CGM, and lead to sub-optimal results. Further, (4) removing the CGM branch while allowing full-channel interaction in other branches results in poor performance, which we attribute to the learning difficulty of modeling channel and spatial simultaneously.
Ablation for Components. In the proposed AdaptR, three parallel branches are developed to learn orthogonal bases. We ablate to discern the roles of different branches. As shown Tab. separate utilization of one or two branches only yields sub-optimal results owing to the insufficient representation. And the combination of the three branches achieves the best results.
Insertion Position and Form. There are various options for both the insertion location and form of our AdapIR. The impact of these choices is shown in Tab. 7. It can be seen that inserting AdaptIR into MLP achieves better performance under both parallel and sequential forms. This is because there a certain dependency between the well-trained MLP and attention, and insertion into the middle of them will damage this relationship. Moreover, the parallel insertion form performs better than its sequential counterpart. We argue that parallel form can preserve the knowledge of frozen features through summation, thus reducing the learning difficulty.
5 Conclusion
In this work, we explore for the first time the potential of parameter-efffcient adaptation to improve the generalization of image restoration models. We observe that current PETL methods struggle to generalize to multiple single-degradation tasks and suffer from performance degradation on aybrid-degradation tasks. We identify that this issue arises from the misalignment between the degradation-requiie representation and the homogeneity in current PETL methods. Based on this observation, we propose AdapuR,, a heterogeneous
coaturre-of-Experee (MoE) to learn local spatial, global spatial, and channel orthogonal bases under low parameter budgets, followed by the adaptive feature ensemble dynamically fuse these bases for degradation-speeifie representation. Extensive experiments validate our AdaptIR as a versatile and powerful adaptation solution.

---

Acknowledgements
This work is supported in part by the National Natural Science Foundation of China, under Grant (623023399,62171248)) Shenzhen Science and Technology Program (JCY)20220818100) 8101011430 JCY1202208111010122033 and the PCNL KEY project (PCL2023AS6-1),
Bibliography
[1] Chao Dong. Chen Change Loy, Kaiming He, Xiaoou Tang. Learning deep convolutional network for image super-resolution. ECCV, pages 184-199. Springer, 2014.
Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond gaussian denoiser: Residual learning of deep CNN for image denoising. IEEE TIP, 26(7):3142-3155 2017. 1,3
[3] Dongwei Ren, Wangmeng Zuo, Qinghua Hu, Pengfei Zhu, and Deyu Meng- Progressive image deraining networks: better and simpler baseline. In CVPR, pages 3937-3946, 2019.
[4] Boyun Li, Xiao Liu, Peng Hu. Zhongqin Wu, Jiancheng Lv, and Xi Peng. All-in-one image restoration for unknown corruption. In CVPR. pages 17452-17462, 2022. 1, 7, 8, 14
[5] Vaishnav Potlapalli, Syed Waqas Zamir, Salman Khan, and Fahad Shahbaz Khan. Promptir: Prompting for all-in-one blind image restoration. arXiv preprint arXiv:2306. 13090, 2023. 3, 7, 13, 14
[6] Dongwon Park, Byung Hyun Lee, and Se Young Chun. All-in-one image restoration for unknown degradations using adaptive discriminative filters for specific degradations. In CVPR, pages 5815-5824. IEEE, 2023. 2,3
[7] Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang. Yibing Song, Jue Wang, and Ping Luo. Adapt- Former: Adapting vision transformers for scalable visual recognition. NeurlPS, 35:16644-16677 2022. 1, 3, 6, 9. 13, 15, 16
[8] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint 2rXiv:2104.0800 2021. 3, 18
[9] Menglin Jia, Luming Tang, Bor- Cor-Chun Chen, Claire Cardie, Serge Belongic, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In ECCV, pages 709 -27. Springer, 2022. 1, 3, 6, 7, 9, 13, 15, 16
[10] Shibo Jie Zhi-Hong Deng. FacT: Factor-tuning lightweight adaptation on vision transformer. In AAAI, volume 37, pages 1060-1068, 2023. 1, 2, 3, 6, 7, 9, 13, 15, 16, 17, 18
[11] Hanting Chen. Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma. Chunjing Xu. Chao Xu, and Wen Gao. Pre-trained image processing transformer. In CVPR, pages 12299-12310, 2021.
2, 3, 6, 8, 15, 16
[12] Namuk Park and Songkuk Kim. How vision transformers work? ICLR, 2021.
[13] Kai Zhang, Wangmeng Zuo, Shuhang Gu, and Lei Zhang. Learning deep CNN denoiser prior for image restoration. In CVPR, pages 3929-3938, 2017. 3
[14] Tao Dai, Jianrui Cai, Yongbing Zhang, Shu-Tao Xia. and Lei Zhang, Second-order attention network for single image super-resolution. In CVPR, pages 11065-11074, 2019. 3, 15
[15] Wenhan Yang, Robby T Tan, Jiashi Feng, Zongming Guo, Shuicheng Yan, and Jiaying Liu. Joint rain detection and removal from single image with contextualized deep networks. IEEE TPAMI, 43(6):1377- 1393, 2019. 19
[16] Hang Guo, Jinmin Li, Tao Dai, Zhihao Ouyang, Xudong Ren, and Shu-Tao Xia. MambalR: A simple baseline for image restoration with state space model. arXiv preprint 4rXhv:2402.16645 2024.
[17] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan. and Sylvain Gelly. Parameter-efficienl transfer learning for NLP. pages 2790-2799. PMLR, 2019. 6, 13, 15, 16
[18] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. BitFit: Simple paramcter-cffcient fine-tuning for transformer- masked language-models. arXiv preprint arXiv:2106. 10199, 2021. 3
[19] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang. Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35, 2023.
10

---

[20] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprini arXiv:2101.00190 2021. 3
[21] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. arXiv preprint (rXiv:2106.096:: 2021. 3, 7, 9, 13, 15, 16, 17, 18
[22] Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. NeurlPS, 34:1022-1035, 2021. 3
[23] Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Kirk-aiika, and Graham Neubig. Towards a unified view of parameter-effiient transfer learning. arXiv preprint arXiv:21 10.04366, 2021. 3
[24] Dongze Lian, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Scaling & shifting your features: A new baseline for efficient model tuning. Neur/PS, 35:109 123, 2022. 3, 6, 7, 9, 13, 15, 16
[25] Arnav Chavan, Zhuang Liu, Deepak Gupta, Eric Xing, and Zhiqiang Shen. One-for-All: Generalized LoRA for parameter-efficient fine-tuning, arXiv preprint arXiv:2306.07966 2023. 3
[26] Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. Neural prompt search. arXiv preprint arXiv:2206.04673, 2022. 3
[27] Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. arXiv preprint arXiv:2205.083355 2022. 3
[28] Yen-Cheng Liu, Chih-Yar Ma, Junjiao Tian, Zijian He, and 7solt Kira. Polyhistor: Paramcter-effiiient multi-task adaptation for dense vision tasks. NeurlPS, 45:36889-36901, 2022. 3
[29] Shibo Jie and Chi-Hong Deng. Convolutional bypasses are better vision transformer adapters. arXiv preprint Xxiv:2207.070399 2022. 3
[30] Shibo Jie, Haoqing Wang, and Zhi-Hong Deng. Revisiting the parameter efficiency of adapters from the perspective of precision redundancy. In ICCV, pages 17217-17226, 2023. 3
[31] Taolin Zhang, Jinpeng Wang, Hang Guo, Tao Dai, Bin Chen, and Shu-Tao Xia. BoostAdapter: Improving test-time adaptation via regional bootstrapping. arXiv preprint arXiv:2410.15430, 2024. 3
[32] Gen Luo, Minglang Huang, Yiyi Zhou, Xiaoshuai Sun, Guannan Jiang. Zhiyu Wang, and Rongrong Ji. Towards efficient visual adaption via structural re-parameterization. arXiv preprint a6Xi:22302.08106, 2023. 3
[33] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Tlge Akkaya, Florencia Leoni Aleman, Diego Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. GPT-4 technical report. arXiv preprint arXiv:2303.0877. 2023. 3
[34] Wenbo Li. Xin Lu, Shengju Qian, Jiangbo Lu, Xiangyu Zhang, and Jiaya Jia. On efficient transformer- image pre-training for low-level vision, arXiv preprint arXiv:21 10175, 2021. 3, 6, 8, 15,
[35] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurlPS, 30, 2017, 3, 4, 13
[36] Eirikur Agustsson and Radu Timofte. NTIRE 2017 challenge on single image super- resolution: Dataset and study. In CVPRW, pages 126-135, 135, 2017. 6
[37] Radu Timofte, Eirikur Agustsson, Lue Van Gool, Ming-Hsuan Yang, and Lei Zhang, NTIRE 2017 challenge on single image super-resolution: Methods and results. In CVPRW, pages 1 144-225, 2017. 6
[38] Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie Line Alheri-Morel. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. 2012. 6
[39] Roman Zeyde, Michael Elad, and Matan Protter. On single image scale-up using reasserreprssnntations. In Curves and Surfaces: 7th International Conference, Avignon, France, June 24-30, 2010, Revised Selected Papers 7, pages 711-730. Springer, 2012. 6
[40] Pablo Arbelacz, Michael Maire, Charless Fowlkes, and Jitendra Malik. Contour detection and hicrarchical image segmentation. IEEE TPAMI, 33(5):898 916, 2010. 6
[41] Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super-resolution from transformed self-exemplars. In CVPR, pages 5197-5206, 2015. 6

---

[42] Yusuke Matsui, Kota It. Yuji Aramaki, Azuma Fujimoto, Toru Ogawa, Toshihiko Yamasaki. and Kiyoharu Aizawa. Sketch-based manga retrieval using Mangal09 dataset. Multimedia Tools and Applications. 76:21811-21838, 2017.
[43] Kede Ma, Zhengfang Duanmu, Qingbo Wu, Zhou Wang. Hongwei Yong. Hongliang Li, and Lei Zhang. Wa- terloo exploration database: New challenges for image quality assessment models. IEEE TIP, 26(2):1004- 1016, 2016. 6
[44] David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In ICCV, volume 2. pages 416 423. IEEE, 2001. 6
[45] Chen Wei, Wenjing Wang, Wenhan Yang. and Jiaying Liu. Deep retinex decomposition for low-light enhancement. arXiv preprint arXiv: 04560, 2018. 6, 19
[46] Ilya Loshchilov and Frank Hutter.
Decoupled
weight decay regularization. arXiv preprint
arXiv:1711..0535 2017. 6, 16
[47] Sneha Kudugunta, Yanping Huang, Ankur Bapna, Maxim Krikun, Dmitry Lepikhin, Minh-Thang Luong, and Orhan Firat. Beyond distillation: Task-level mixture-of of-exerss for efficient inference. arXiv preprint arXiv:2110.03722, 2021.7 7
[48] Jinguo Zhu, Xizhou Zhu, Wenhai Wang. Xiaohua Wang, Hongsheng Li, Xiaogang Wang, and Jifeng Dai. Uni noi-pereciver-mo:: Learning sparse generalist models with conditional moes. NeurlPS, 45:2664-2678 2022.7 7
[49] Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Su- sano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. NeurIPS, 34:8583-8595, 2021.
[50] Wenlong Zhang, Xiaohui Li, Guangyuan Shi, Xiangyu Chen, Yu Qiao, Xiaoyun Zhang, Xiao-Ming Wu, and Chao Dong. world image rusouttcno as multi-task learning. NewrIPS, 36, 2024. 7
[51] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 /999, 2020. 13
[52] Yihao Liu, Xiangyu Chen, Xianzheng Ma. Xintao Wang. Jiantao Zhou, Yu Qiao, and Chao Dong. Unifying image processing as visual prompting question answering. arXiv preprint 1.00v:2.300005111 2023. 13, 14
[53] Jiaqi Ma, Tianheng Cheng, Guoli Wang, Qian Zhang, Xinggang Wang, and Lefei Zhang. ProRes: Exploring degradation-awee aware visual prompt for universal image restoration. arXiv preprint arXiv:2306.1300 13653, 2023. 13. 14
[54] Cong Wang, Jinshan Pan, Wei Wang, Jiangxin Dong, Mengzhu Wang, Yakun Ju, and Junyang Chen. Promptrestorer: A prompting image restoration method with degradation pereeption. NeurIPS, 36:8898-
8912, 2023. 13,
[55] Yulun Zhang, Kunpeng Li, Kai Li. Lichen Wang, Bineng Zhong, and Yun Fu. Image super-resolution using very deep residual channel attention networks. In ECCV, pages 286-301, 2018. 15
[56] Jingyun Liang, Jiczhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. SwinIR: Image restoration using swin transformer. ICCV, pages 1833-1844, 2021.
[57] Kai Zhang, Wangmeng Zuo, and Lei Zhang. Fídnet: Toward fast and flexible solution for enn-based image denoising. IEEE TIP, 2709) 2018. 15
[58] Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. Residual dense network for image super-resolution. CVPR, pages 2472 2481, 2018. 15
[59] Zhendong Wang. Xiaodong Cun, Jianmin Bao. Wengang Zhou, Jianzhuang Liu, and Houqiang Li. Ufommer:
A general u-shaped transformer for image restoration. In CVPR, pages 17693, 2022. 16
[60] Wei Chen, Wang Wenjing, Yang Wenhan, and Liu Jiaying. Deep retinex decomposition for low-light enhancement. In BMVC, 2018.
[61] Ke Xu, Xin Yang, Baocai Yin, and Rynson WH Lau. Leamingg to restore low-light images via dccomposition cotomposiiion-an-enntaaee andenmnane,ment In CVPR, pages 2281 12900, 2020. 16
12

---

Appendix
A Basic Background of Parameter Efficient Transfer Learning
Since very little work has been done to study PETL in low-level vision, we re-implement the current state-of-the-ar PETL methods in this work, such VPT [9], Adapter [17], A [21], AdaptFormer [7], SSF [24], and FacT [10]. In this part, we review these methods and provide detailed implementation details for reproduction. Fig. 7 gives illustration of these baseline methods.
Toodhmmr Lager
MLP
Lagadier
dagptrr
Hobics Stat
(c) LoRA
(C) A dapttFomeer
(c) SSP
(0 FacT
Figure 7: Illustrations of exsiting state-of-the-art PETL baselines.
VPT [9], shown in Fig. 7(a), prepends learnable prompt tokens in the input of one trans- former layer [35]. In [9], there are two versions VPT, i.e., VPTshallow and VPT To obtain good performance, use VPT which inserts new prompts at each transformer layer, as our default settings.
Adapter [17], shown in Fig. 7(b), is a bottleneck structure with an intermediate GELU activation function. Following the vanilla Adapter design [17], we insert the Adapter both after the Multi-hean Self- Celf-Attention (MSA) and Multi-Layer Perceptron (MLP).
LoRA [21], shown in Fig. 7(c) use the multiplication of two low-rank to approximate the incremental matrices in projection layers of Query and Value.
AdaptFormer [7], shown Fig. 7(d), similar in model architecture with Adapter, but has different insert position and form. In [7]. the AdaptFormer is placed before the second LayerNorm layer and adopts parallel insertion form.
SSF [24], shown in Fig. 7(e), utilize learnable scale and shift factors modulate frozen features. Based on the settings in [24], we place the SSF layer behind all the attention QKV projection, the LayerNorm, and the MLP layers.
FacT [10], shown in Fig. 7(I), tensorises Vision Transformer [51] and introduces low-rank approximation to the incremental matrix similar to LoRA. Different from LoRA, FacT sets the up- upojrojection and down- projection be shared across layers while setting the projection in the low-rank space to be layer-specific. There are two versions of FacT, namely FacT Tr and FacTTk [10], we FacT TT in this work because of its good performance. Following [10], we introduce the FacT layer in attention QKV projection as well as MLP layers.
B Discussion with Prompt-based Methods
In this section, we briefly discuss the difference between our AdaptIR with other prompt- methods. To the best of our knowledge, only the PromtGIP [52] shows the zero-shot ability when facing unseen degradations. And the other ProRes [53], PromptIR [5], and PromptRestorer [54] can only handle degradations which have been seen during training, which means they still need additional fine-tuning for task generalization. The advantages of our AdaptI compared to these
proset-based based
13

---

Table 8: Comparison with prompt- based restoration methods.
Methods
Type
Fast Adaptation
Adaptation Cost
PSNR on denoising PSNR on deraining
PrompiGiP [52]
Prompt-based
Yes
zero-shot
26.22
25.46
ProRes [53]
Prompt-based
No
8x3090GPUs
Not open-source
Not open-source
PrompilR [5]
Prompt-based
No
7-days 8x3090GPUs
29.39
37.04
PrompiRestorer [54]
Prompt-based
No
8x3090GPUs
Not open-source
Not open-souree
Ours
PETL-bused
Yes
1x3090
29.70
37.81
prompt-based approaches are two-fold. As for efficiency, Promptl, ProRes, and PromptRestorer all need full fine-tuning for adapting to new tasks, e.g., PromptI needs 7-day 8x3090 GPUs for full fine-tuning, while our AdaptIR needs only 8h xx3090 GPU. As for performance, since these methods need to learn multiple degradations within one model, it is inevitable to suffer the problem of negative transfer, which impairs performance. We give a thorough comparison in Tab. 8.
C Discussion on Multi-task Restoration Paradigms.
In this section, we revisit existing paradigms for dealing with multi-task restoration problems, in which multiple degradations need to be handled. Let denote the number of degradation types, i.e. downstream tasks as N, then we can summarize existing paradigm into the following three categories:
1. "N for N training N task-specific models for N downstream tasks, such as Restormer, MPRNet.
2. "1 for N" training 1 all-in-one model for N downstream tasks, such as PromptIR [5]. AirNet [4].
3. "(I+N) for N using task-shared pre-trained weights, and N task-specific lightweight modules.
Early image restoration techniques predominantly employed the first strategy, which trains different models to handle multiple degradations. Although this strategy handle multiple degradations, it usually requires training and storing N copies for each task. The recent all-in-one methods train one model for multiple degradations, although reducing the model copy one to improve the efficiency, this approach usually suffers from performance degradation due to the multi-task learning difficulties and the negative transfer learning problem. In this work, the proposed AdaptIR is the first paradigm categorized in the third category, which trains shared pre-trained backbone as well as N task-specific lightweight modules. This paradigm can be seen as a compromise between effectiveness and efficiency. However, given that the task-specific modules are very lightweight, we believe that the advantages of this paradigm outweigh the disadvantages.
D Further Explanation of the Heterogeneous Representation.
In this work, we pay attention to the learning of the heterogeneous representation. Here, we articulate make it more clear about this term. The heterogeneous representation in this paper represents the learning of discriminative features across different degradation types. The term representation here is instantiated as the Fourier curve in Fig. in the main paper. Previous approaches tend to produce similar representations across various degradations. As common knowledge, restoring different degradations requires different representations, e.g., SR needs high-pass filter network while denoising needs low-pass. As a result, if the representation needed by current degradation matches the specific representation of the existing PETL method, it works. If not, leads to unstable performance. To demonstrate the generality of the problem regarding the unstable performance and the homogeneous representation under different degradations, we provide more evidence in Fig. and Fig. 10.
14

---

Table 9:
Quantitative comparison for x4 image SR on PSNR(dB). We compare the #param when the performance is the same.
Table 10: Quantitative comparison for color image denoising on PSNR(dB). We compare the #param when PSNR is the same.
BSDS
Urban Manga
CBSD68
Urban100
Method
#param
Set5 Set14
Method
Hparam
100
100
109
==3 <=
#=30 a=50
RCAN [55]
15.6M
32.63
28.87
27.77
26.82
31.22
FFDNet [57]
0.8M
30.31
27.96
30.53 28.05
SAN [14]
15.9M
32.64
28.92
27.78
26.79
31.18
RDN [58]
15.6M
28.31 30.67
31.69
29.29
SwintR [56]
11.9M
32.74
29.06
27.89
27.37
31.93
SwinlR [56]
11.9M
28.56
29.82
Full-ft
119M
32,66
29.03
27.82
27.31
31,64
Full-ft
119M
30.75
28.39
32.01
29.72
pretrain
32,58
28.97
27.79
27.18
31.41
Pretrain
30.73 28.36
31.94
29.68
VPT
884K
32.71 29.02
27.82
27.20
31.65
VPT [9]
884K
30.74 28.37
31.97 29.68
Adapter [17]
691K
32.70 29.03
27.82
27.21
31,68
Adapter [17]
691K
30.74 28.37
31,97 29.69
LoRA [21]
995K
32.70 29.03
27.82
27.20
31,68
LoRA [21]
995K
30.75 28.38
31,98 29.70
Adaptfor [7]
677K
32.70 29.03
27.82
27.21
31,68
AdaptFor: [7]
677K
30.73 28.37
31.97 29.68
SSF [24]
373K
29.56 26.84
26.50
23.78
26.02
SSF [24]
373K
30.07 27.64
29.79 27.01
FacT [10)
537K 32.71 29.03
27.82
27.23
31,70
FacT[10)
537K
30.75 28.38
31.98 29.70
Ours
370K
32.71
29.04 27.82
27.22
31.70
Ours
515K
30.74 28.38
31.98 29.69
Table 11:
Quantitative comparison for light
Table 12: Quantitative comparison for heavy rain streak removal on Rain 10H dataset.
rain streak removal on Rain 100L dataset.
Method
#param PSNR SSIM
Method
#param PSNR SSIM
Full-ft
119M 42.14 0.9905
Full-ft
119M
32.23
0.9202
Pretrain
17.30 0,5488
Pretrain
17.30 0.5488
VPT [9]
884K
41.74 0.9896
VPT [9]
884K
30.87 0.8967
Adapter [17]
691K
41.95 0,9900
Adapter [17]
691K
30.99 0.8971
LoRA [21]
995K
41.89 0,9898
LoRA [21]
995K
31.16 0,9002
AdaptFor. [7]
677K
41.90 0.8992
AdaptFor. [7]
677K
31.10
0.8992
FacT [10]
537K
40.61 0.9984
FacT [10]
537K
29.70
0.8824
Ours
697K 42.09 0.9902
Ours
697K
31.23 0.9016
E More Experiment Results
E.1
Comparison on Single-degradation Tasks
Current PETL methods struggle to achieve stable performance due to homogeneous frequency charac- teristics and suffer performance degradation when not well aligned with the frequencies required by a specific degradation. Therefore, needs multiple angle-degradation tasks to obtain convineing evalu- ation results. Here, we give results of single-degradation tasks, including image super-resolution in Tab. 9. color image denoising in Tab. 10, light deraining in Tab. 11, heavy deraining Tab, 12, and low-light enhancement in Tab. 13. It can be seen that our method maintains stable best performance on most single-derrdattin tasks, with the second-best method varying across tasks. For example, the recent state-of-the-an methods FacT [10] obtains comparable performance with our AdapuR, however, it suffers significant performance degradation on the subsequent light and heavy rain streak removal tasks. Another example can also be seen in LoRA [21], which performs the second best in heavy deraining but struggles with low-light image enhancement tasks, In contrast, our method is more stable, achieving consistent best performance across these single-degradation tasks. We also provide quantitative comparisons on the single-degradatton restoration tasks in Fig. 12 and Fig. 13.
E.2 Additional Results on Hybrid Degradation with EDT
In order to demonstrate the generalizability of our AdaptR,, we choose IPT [1]] and EDT [34] as pre-trained base models to evaluate the performance of different PETL methods. Due to the page limit, we mainly present the results of IPT in the main paper. Here, we give more experimental results with EDT. The results on recond-orde degradation LR4&JPEG30 with EDT are shown in Tab. 14. can seen that our method continues to achieve sfate-of-the- -rtt--rrtt performance by a significant margin. For example, our AdaptiR outperforms the second- best method FacT [10] by up to 0. 1.dBB PSNR while using fewer tunable parameters. The results with EDT as the base model demonstrate the robustness of the proposed method.
15

---

Table 13: Quantitative comparison for low-light image enhancement with LOLv1 dataset.
UFormer RetinexNet
FIDE
VPT
Adapter
LoRA
AdaptFor.
FacT
AdaptI
Metric
Pretrain
[59]
[60]
[61]
[9]
[17]
[46]
[7]
[10]
(ours)
#param PSNR
884K
691K
995K
677K
537K
697K
7.64
16.36
16.77
18.27
19.28
19.22
18.94
19.40
19.06
19.46
SSIM
0.2547
0.771
0.560
0.665
0.7198
0.7293
0.7197
0.7352
0.7147
0.7441
Table 14: Quantitative comparison for second-order degradation with LR4&JPEG30 using EDT as pre-trained restoration models. The best results are bolded.
Set5
Set14
Manga109 PSNR/SSIM PSNR/SSIM PSNR/SSIM
BSDS100
Urban100
Method
#param
PSNR/SSIM PSNR/SSIM
Full-ft
11.6M
27.29/0.7800
25.58/0.6598
23.71/0.6768
25. 15.11/0.609 25. 25.69/0.8043
Pretrain
25.08/0.6638 15.9550.5847 21.51/0.5569 24.08/0.5588
22. 22.60/0.6611
VPT [9]
311K
26.39/0.736 24.84/0.6306 22.48/0.6101
24.75/0.5990 23.98/0.7366 7365
Adapter [17]
168K
77.00/0.7698 25.36/0.6518 23.30/0.6566 25.01/0.600 6039 25.13/0.7845
LoRA [21]
155K
27.01/0.7694 25. 65.36/0.6513 23,26/0.6551 25.00/0.600 6.0000.6038 25.09/0.7831
AdaptFor. [7]
162K
27.03/0.7715 25.40/0.6533 23.32/0.6581 25.02/0.60K
15/19907777
SSF [24]
117K
76.91/0.7664 25.33/0.6502 23.21/0.6519 24.98/0.6027 24.98/0.7801
FacT [10]
174K
27.01/0.7703 25.37/0.6521 23.30/0.6569
25.00/0.6041 25.14/0.7855
Ours
170K
27.13/0.7739 25.44/0.6545 23.41/0.6620
25.04/0.6057 25.29/0.7903
E.3 Results on More Degradations.
In this section, we further include another challenging degradation type, namely the real image denoising which is unseen during the pre-training phase and is the real- world degradation type, to further demonstrate the general- ization of the proposed AdaptR. We use the training and testing sets in the SIDD for this experiment. The exper- imental results shown in Tab. 15. It can be seen that our AdaptIR maintains its superiority when transferring to real-world degradation. For instance, method outperforms LoRA by 0.13dB PSNR. The above experimental results demonstrate the robustness of our methods.
Table 15: Results on real-world denois- ing tasks with SIDD datasets.
Methods Aduptioor. LoRA Adapter FacT
MoE Ours
#param PSNR
677K
537K 667K 697K 39.02 39.05 39.10
39.03
38.97
39.00
F Complexity Analysis
In this section, we theoretically analyze the parameter complexity of the proposed method. We omit the bias term the corresponding parameter is small. Assume that the hidden dimension of the pre-trained restoration model [11, 34] is d and the dimension of intrinsic space in AdapuR is d'. For the dimensional up and down operations, the number of parameters is 2dd'. For the Local Interaction Module, assuming the convolution kernel size is K and the pre-defined rank of U, V r. then the number of parameters of LIM with depth- separable design is d'r + rK2 For the Frequency Affine Module, the total parameters of the amplitude and phase projection are 2d'. For the Channel Gating Module, which contains the channel compression as well as the FFN, the number of parameters d' + 2d' _.. As for the Adaptive Feature Ensemble, the compression convolution costs d' number of parameters, and 2d' d' 5 is used in pooling FFN. Summing up the above terms gives 2(a=h) d² + (r + 4 2d)d' + r r In the implementation, we set r = d' /2, = 2,b = 8, K = 3 ab and d' d 7
Therefore, the total parameter complexity of AdaptIR is ( + } )d( + 1/7 0 (
G Feature Response Intensity Analysis.
To make more clear how the proposed multi-branch structure works, we give the distribution of feature response intensity of three branches across various tasks, including SR, heavy deraining,
16

---

brunches of classic stod 5146.030
[4] Representation dificeent
(81 Represeriaatinn diferent arcchhs dessic with 3PA8JE7630
(c) Fourier analysis on LoRA
[d) Fourier analysis on FacT
50
66-
brant_2
Hhowy Kan bleck Ficc RAN
Hinnyy Sun Smaak Semooa
4.5
Woxer 5
10-
-00
+12
3.0-
0.33
3.0
2.0
2.5
165-
4.0-
-35
5.0
-10
3.2
Degguet
Flequricy
Figure 8: The frequency characteristic curves of features from three branches in the classic MoE with hybrid degradation of (a)SR4&DN30 and (b)SR4&JPEG30. (c)&(d) Fourier analysis on more current PETL methods, LoRA [21] and FacT [10], which shows significant representation homogeneity across tasks.
to henout
Lote Adspee NNT OuI
L8A Kapal FadT Can
06A Kape
UdA kaps Fac
Can
Loda Acapse FIcT Oun
Figure 9: More evidence on the unstable performance of previous PETL methods across different single-degradation types, and the unfavorable performance under hybrid degradation.
light deraining, low-light image enhancement, and two hybrid degradations in Fig. 11. These figures indicate that our AdaptR can adjust to different degradation types by enhancing suppressing the outputs from different branches. Specifically, for the heavy&light deraining tasks, AdaptIR adaptively learns to enhance the low-frequency global features, i..., the frequency affine module which is responsible for global spatial modeling has large values. This property ensures the removal of the high-frequency rainstreaks well as the preservation of the global structure of the image. For SR tasks, AdaptIR adaptively enhances the restoration of local texture details by learning large output values from the local spatial modules. For the hybrid degradation task, AdaptIR shows it can distinguish between different hybrid degradations, i.c., three branches exhibit different patterns under two types of hybrid degradations. In short, each branch of AdaptRR can capture discriminative features under different degradations, indicating that our approach is degradation-aware. This ability guarantees robustness on single degradation and superior performance under hybrid degradation.
H Differences from Classic MoE
Although both our AdapIR and the classic MoE employ the multi-branch structure, however, our approach differs from the classic MoE in the following aspects. Firstly, the classic MoE uses the multi-branch structure to enhance the model capabilities, whereas our proposed heterogeneous MoE aims to capture heterogeneous representations across different restoration tasks. Secondly, despite using the multi-branch structure, the classical MoE still tends to capture homogeneous representations since each branch is the same, thus resulting in the sub-optimal results in Tab. 1. In contrast, each branch in our AdaptR is designed orthogonally, thus ensuring the learning of orthogonal representation bases. Thirdly, classical MoE uses simple summation to fuse branches, which is ietraddationagnnstte while our AdapuR uses regradationnsspccine ensemble to learn the combination of orthogonal representation bases, facilitating heterogeneous representation across tasks.
In Fig. 8, we also give the frequency characteristics of the output features from different branches of the well-trained classical MoE. It can be seen that different branches still suffer from homogeneity despite the use of multi-branch structure. In contrast, as shown in Fig. 4 in the main paper, our Ad ensures that different branches capture different representations through the proposed orthogonal design, which promotes heterogencous representations to achieve better performance.
17

---

(a) Fourier analysis Adapter
Fourier analysis LoRA
(c) Fourier analysis Fac
(d) Fourler analysis
Low peancceentd
Looiight rewoomeet haaey mua tiraal femioell spperssolution
Hoaas tain Sirale hermova Sipae ssstlatis
Honry haln Stree Spper vostettee
dinsk kenovl
Alog arpllae
3.00p wopporooooo
0.74
6.6
0.5
0.51
0.20
348
3.08
0.5
1.5
Figure 10: More evidence that shows previous PETL methods struggle to learn distinguishable
features across different degradation types, i.e., homogeneous representation. In contrast, our AdaptIR can learn heterogeneous representations for different degradations.
Modile
Supar noplution
Low Low-lghtt Enhannareent
Hybrdd Digradation (Uldk)ennee)))
Hespy Roin Steek, femoool
Hybrid Degradaton NuN Jicss)
Light Rain Stevkl feemove
Figure 11: The distribution of feature response densities of the three branches across different tasks.
I More Evidence of Homogeneous Representation
In Fig. 1, we give the frequency characteristics of Adapter [8], and find its homogeneous representation when facing different degradations. To demonstrate the prevalence of homogeneous representations in current PETL methods, we provide the frequency characteristics curves of more PETL methods in Fig. 8. It can be seen that the current sfate-of-the-ar PETL methods LoRA [21] and FacT [10] also exhibit homogeneity as Adapter, i.e., the learned feature representations are similar even if they are for different degradations. In contrast, AdapIR utilizes the orthogonal multi-brande design to learn diverse representations, facilitating heterogeneous representations on different restoration tasks.
J Dataset Description
In this work, we evaluate different PETL methods on diverse image restoration tasks, which cover many training and testing datasets. To make the experimental setup more clear, we give detailed description of datasets in Tab. 16.
K Limitations and Future Work
While AdaptIR appears as a competitive PETL alternative across various image restoration bench- marks, it can be further improved with task-specific module designs. For example, in the proposed AdaptIR, different tasks share the same structure, however, different restoration tasks have diverse model preferences. An intuitive solution might be to introduce degradation-aware dynamic networks. Moreover, although this work has covered multiple degradation types, some other degradations can also be explored in the future, e.g. blur and haze, to further demonstrate the generalization ability.
18

---

Table 16: Dataset description for various image restoration tasks.
Tasks
Type
Dataset
Num_sample
train Div2K+Flicke2KK
800+2650
Super-resolution
test
Scd46Sel14BSSDS1000Ubbba 0tte://rraaa0000aaa00000 00+Manga 5+114100+100+100
train
BSD400+WED
400+4744
Denoise
test
BSD68+Urban
68+100
train
Rain I
200
DerainL
test
Rain
100
train
TrinTrainH
1800
DerainHI
test
Rain 100H
100
Second-ords order Restor.
train
Div2K=Flicker2K
800+2650
(SR4& 30)
test
BBSS4Se114BSDDS00000 10b+a (00+Manga109
14+14+00+++000 109
Low- Low-light Enhancement
train
LOLv LOLv--train-splic
485
test
LoL.v -ttestt-spplit
15
OREG
mohan
Rai
noRain
VP7 32 784
Rain100H: lmng 038
dagtee 27 2568
LARA 27,2788
Ours 27,47
Rain img 048
Adigtar 32. BAGB
ToRA 32,9055
Oues 313
Figure 12: Visual comparison of heavy rain streak removal on samples from Rain 100H [15] dataset.
Link
Nomal-light
Low-light
VPT 22,10d0
Nocma-lghh -gtt
Low-ligtt
VPT 17.7388
LOLvi: ing.
Adgeler
LOLvI: img. 547
Aaptter 17.3M6B
LORA 17,861
Oues 17,5hlls
Figure 13: Visual comparison of low-light image enhancement on samples from LOLv1 [45] dataset.
L Broader Impact
Our AdaptIR holds significant promise for improving the quality and generalization of image restoration across various domains, such medical imaging, historical document preservation, and digital media restoration. By enabling more accurate and reliable image restoration with reduced computational resources, AdaptIR
facilitate advancements in these fields, leading better diagnostic tools, preservation of cultural heritage, and enhanced digital media quality. However, the enhanced capabilities of AdaptIR also present potential negative societal impacts, such the risk of misuse in generating realistic fake images or deepfakes, which could be used for disinformation, creating fake profiles, unauthorized surveillance, leading to privacy violations, security concerns, and ethical issues. To mitigate these risks, it is crucial to implement measures like gated releases of models, mechanisms for monitoring misuse and ensuring transparency deployment and training processes, alongside continuous evaluation of the technology's impact.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have clearly outlined the main contributions of this paper in the abstract and introduction sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed detail the limitation of this work in Appendix K.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

All assumptions in the paper are either derived from the conclusions of previous research or validated through extensive experiments.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have provided a detailed description of our algorithm's process and implementation specifics in the paper, and we will release our code after review.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
21

---

reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We will release our code after review, but we have already provided a detailed explanation of how to implement our algorithm and the specific implementation details in the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https: htt///nips..p/ cc/
public/ guides CodeSubmii s siionoooicyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / /nnps cc/ publi icc////// cu des CodeS ubmi ss i onPol icy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have provided detailed experimental details in both the paper and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We have provided the performance fluctuation under different random seeds in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
22

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provided the computational resources we used for experiments in the implementation details section.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /tte:://neurs /nwuriuuuu..////e//ee wttpp//bb.c/ttttccssci...ooooeee
Answer: [Yes]
Justification: We have checked in every respect that this work conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have provided detailed discussion on possible both positive and negative societal impact in ??.
Guidelines:
23

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, the creators and original owners of assets used in this paper are prop- erly credited, and the license and terms of use are explicitly mentioned and respected, as evidenced by thorough citations.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, papers thoode com/ dat a seets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
25

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "CsmbqVwNEW", "reviewer": "Reviewer_Y1LM", "summary": "The paper proposes a Parameter Efficient Transfer Learning (PETL) method for image restoration, which utilizes local, global, and channel-related modules and adaptively combines them to obtain heterogeneous representation for different degradations. Experiments are conducted on multiple degradations and the results demonstrate the effectiveness compared to other PETL methods.", "questions": "Refer to Weaknesses.", "limitations": "None", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Exploring the use of PETL to enhance the image restoration performance is worthwhile. \n2. The experiments in the paper are quite comprehensive.", "weaknesses": "1. A more detailed and precise definition of Heterogeneous Representation is needed. The differences in Figure 2 are results but not the underlying causes of the problem. Moreover, why are the three proposed modules in the paper considered to constitute Heterogeneous Representation?\n2. The local interaction module has the same structure as LoRA, it could say that it is essentially a LoRA. The other two modules are also commonly used in image restoration. Most importantly, the paper does not clearly explain why these three modules are combined.\n3. What is the rank of LoRA in Table 1? More details about the MoE structure should be presented in the paper.\n4. Why is only the single-task setting performance reported in Table 2? How do you determine which type of degradation appears in the image in the all-in-one setting?\n5. There are color marking errors in Table 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### [Q1-1: Precise definition of the Heterogeneous Representation] \n> A more detailed and precise definition of Heterogeneous Representation is needed.  \n\nThe Heterogeneous Representation in this paper represents the learning of discriminative features across different degradation types. The term ‘representation’ here is instantiated as the Fourier curve in Fig.1 in the main paper.\n\nPrevious approaches tend to produce similar representation across various degradations. As common knowledge, restoring different degradations requires different representations, e.g., SR needs high-pass filter network while denoising needs low-pass. As a result, if the representation needed by current degradation **matches** the specific representation of the existing PETL method, it works. If not, it leads to unstable performance. \n\nTo demonstrate the generality of the problem regarding the unstable performance and the homogeneous representation under different degradations, we provide more evidence in Fig.A and Fig.B in the rebuttal PDF.\n\n### [Q1-2: The causal logic of our research line] \n> The differences in Figure 2 are results but not the underlying causes of the problem.\n\n(Fig.2 in the paper is the technical pipeline, so we guess you mean Fig.1 Right part.)\n\nMotivated by the success of PETL, we extensively evaluate their performance on image restoration. However, we surprisingly find these methods perform **not robustly** on **single degradation** tasks, and all suffer **performance drops** on **hybrid degradation**.\n\n\nThis interesting observation motivates us to find the reason. We then find existing methods tend to **learn similar representation across different degradation types**, and thus we further speculate that this may be the **reason** for the above problem. \n\n\nTo **validate the speculation**, we designed a novel MoE to learn Heterogeneous Representation, i.e. learn different features for different degradations. \n\n\nFinally, experiments show that learning **different representation across different degradation types** can indeed obtain robust performance on single degradation and favorable performance on hybrid degradation. This result demonstrates our speculation.\n\n\n\n### [Q1-3: Three modules for Heterogeneous Representation ] \n> Moreover, why are the three proposed modules in the paper considered to constitute Heterogeneous Representation?\n\nThe three modules are specifically designed to force them to learn local-spatial, global-spatial, and channel interactions, respectively. And they indeed work as we expected, which is also verified in Sec.3.2. Here, the features modeled from different perspectives constitute heterogeneous representations, which is also recognized by Reviewer 62kc and EhxW.\n\n### [Q2-1: Differences between local interaction module (LIM) and LoRA] \n> The local interaction module has the same structure as LoRA, it could say that it is essentially a LoRA.\n\nThe technique is different. Notably, the original LoRA can only decompose the **Linear Projection Weight** with shape $W \\in \\mathbb{R}^{C_{out} \\times C_{in}}$, while our proposed LIM focus on the **Convolution Weight** with shape  $W \\in \\mathbb{R}^{C_{out} \\times C_{in} \\times K \\times K}$, where how to handle the additional kernel size $K$ is non-trival.\n\n\nThe placement is different. The original LoRA is inserted in a **serial manner**, i.e., features will go through the frozen linear weights then LoRA. By contrast, our AdaptIR is designed in a **parallel fashion**, which preserves the knowledge of the pre-trained  model and is experimentally verified in Tab.6,\n\nThe goal is different. The LoRA is used for weight transformation, while the low-rank in our LIM is used to learn local spatial for learning heterogeneous representation.\n\n\n\n### [Q2-2: Combination of three modules]\n> The other two modules are also commonly used in image restoration. Most importantly, the paper does not clearly explain why these three modules are combined.\n\nWe would like to clarify that our goal is to **encourage heterogenous representation learning, instead of proposing complex modules**. Therefore, we introduce commonly used module for a simple baseline. \n\nThe output from three modules are complementary to each other, which are further combined to learn **different representation across different degradation types**.\n\n\n### [Q3: Details about LoRA and MoE]\n\n> What is the rank of LoRA in Table 1? More details about the MoE structure should be presented in the paper.\n\n\nThe rank of LoRA is 32. The MoE is a multi-branch bottleneck structure, and we follow [A][B][C] for implementation. We introduce this baseline to show that despite MoE also uses a multi-branch structure, it is still sub-optimal when not learning heterogeneous representations.\n\n\n[A] Zhu et al. Uniperceiver-moe:Learning sparse generalist models with conditional moes. NeurIPS22\n\n[B] Carlos et al. Scaling vision with sparse mixture of experts. NeurIPS21\n\n[C] Sneha et al. Beyond distillation:Task-level mixture-of-experts for efficient inference. arXiv preprint.\n\n\n### [Q4: Comparison with all-in-one methods]\n> Why is only the single-task setting performance reported in Table 2? How do you determine which type of degradation appears in the image in the all-in-one setting?\n\nActually, existing All-in-one methods, e.g. AirNet,PromptIR,IDR,DyNet, **also reported single-task performance in their papers**, in which they use the same model structure and train only using single-degradation data. And the results of this single-task training is **better** than all-in-one training for them.\n\nWe reported the single-task performance of all-in-one methods in Tab.2 for **fair comparison**, since this performance is the **upper-bound** of these methods.  Notably, the performance of all-in-one methods in Tab.2 is the same as their original paper, which uses more training data than ours. We will include all-in-one setting performance in revision.\n\n\n### [Q5: Some typos]\n\nWe will revise them in revision. Thanks!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zla0EvG7Hx", "reviewer": "Reviewer_EhxW", "summary": "This work introduces AdaptIR, a novel heterogeneous Mixture of Experts (MoE) structure, to adapt pre-trained restoration models to various downstream tasks. The proposed method achieves performance comparable to full fine-tuning while only training 0.6% within 8 hours, demonstrating the high efficiency. Extensive experiments, including on hybrid degradation and various single degradation, validate the effectiveness of the proposed method compared to current PETL methods.", "questions": "1. What is computational cost for the proposed method, compared with other parameter-efficiency mehtods?\n2. The reported results are mainly conducted on the synthetic degradation, how does AdaptIR perform on real-world degradation scenarios?", "limitations": "See the weakness above.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. Introducing PETL into image restoration is both interesting and promising, potentially serving as a competitive alternative to existing all-in-one image restoration solutions. Furthermore, Table 2 demonstrates the superiority of the proposed PETL paradigm over the all-in-one approach in terms of performance (PSNR, SSIM) and efficiency (parameters, time, GPU memory).\n2. The motivation and the specific techniques are well communicated. The authors identify that directly applying current PETL methods like LoRA to IR can result in unstable performance, and then further attributed this problem to the homogeneous frequency representations. Based on this, they propose a heterogeneous MoE structure, with each branch capturing orthogonal bases.\n3. The experiments are solid, the authors versify the proposed method across multiple single degradation tasks (image super-resolution, denoising, low-light enhancement, deraining), and demonstrating strong performance on hybrid degradation tasks.\n4. The proposed method achieves state-of-the-art performance, showing significant improvements over previous PETL methods.\n5. The paper is well-written and easy to follow.", "weaknesses": "1. Both the proposed AdaptIR method and LoRA use low-rank matrices for efficiency, clarifying the differences between these two methods would be beneficial.\n2. The proposed method have not been tested on on real-world degradation scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### [Q1: Differences on low-rank strategy]\n\n> Both the proposed AdaptIR method and LoRA use low-rank matrices for efficiency, clarifying the differences between these two methods would be beneficial.\n\nIn fact, not just LoRA, the low-rank strategy is a common practice in existing PETL arts, e.g. Adapter, FacT, AdaptFormer. The main difference lies in **how low-dimensional features are processed**. Our method uses a multi-branch structure to adapt MLP in transformer block, instead of one-branch LoRA placed in self-attention. \n\n\n\n### [Q2: Performance on real-world degradation]\n\n>The reported results are mainly conducted on the synthetic degradation, how does AdaptIR perform on real-world degradation scenarios?\n\nWe conduct experiments on real-world denoising tasks with SIDD datasets. The results are as follows:\n\n\n| Methods    | LORA | Adapter | AdaptFormer | FacT | MoE | Ours |\n|------------|------|---------|-------------|------|-----|------|\n| #param     | 995K | 691K    | 677K        | 537K | 667K| 697K|\n| PSNR       | 38.97| 39.00  | 39.03       | 39.02| 39.05| 39.10|\n\nIt can be seen that our AdaptIR maintains its superiority when transferring to real-world degradation, demonstrating the robustness of our methods.\n\n### [Q3: Computational cost]\n\n> What is computational cost for the proposed method, compared with other parameter-efficiency methods?\n\nSince our AdaptIR only trains about 700K parameters, the GPU memory costs only about 8G with 8h training. e control the parameters of different methods to be roughly similar and compare the performance with other parameter-efficiency methods, the experiments demonstrate the sota performance of our methods.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ByY1aqCJB3", "reviewer": "Reviewer_62kc", "summary": "The paper presents a novel approach to image restoration tasks, which leverages a heterogeneous Mixture-of-Experts (MoE) architecture. The proposed method aims to address the limitations of existing PETL (Patch-Exemplar-based Texture Learning) techniques for image restoration. The key contributions of the paper are:A heterogeneous MoE framework that combines multiple specialized sub-models to enable more robust and effective image representation learning for restoration tasks.A detailed design of the MoE architecture, including the individual module components and their synergistic collaboration to achieve the heterogeneous image modeling.", "questions": "see the weakness.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The key strengths of the paper are: 1) A heterogeneous MoE framework that combines multiple specialized sub-models to enable more robust and effective image representation learning for restoration tasks. 2) A detailed design of the MoE architecture, including the individual module components and their synergistic collaboration to achieve the heterogeneous image modeling.", "weaknesses": "1, lack comprensive comparison with other SOTA all-in-one methods like [1] and [2]\n\n[1] Ingredient-oriented multi-degradation learning for image restoration, cvpr 2023.\n[2] Towards Efficient and Scalable All-in-One Image Restoration, arxiv 2023.\n\n2,  in terms of Fig. 9 and Fig. 10, the visual comparison is not obvious among Adapter, LoRA and Ours.\n\n3, it is expected to present the testing time rather than training time to eveluate its effectiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### [Q1-Comparison with other SOTA all-in-one methods]\n\n>  lack comprehensive comparison with other SOTA all-in-one methods like IDR(cvpr23) and DyNet(arxiv)\n\nThanks for your kind advice, the suggested comparison are are follows.\n\n\n\nTableA Effectiveness and efficiency comparison on Light-deraining.\n\n| Methods   | dataset | #param   | Training time | GPU memory | PSNR  | SSIM  |\n|-----------|---------|----------|---------------|------------|-------|-------|\n| AirNet    | Rain100L| 8.75M    | ~48h          | ~11G       | 34.90 | 0.977 |\n| PromptIR  | Rain100L| 97M      | ~84h          | ~128G      | 37.04 | 0.979 |\n| IDR       | Rain100L| 15M      | ~72h          | ~23G       | 37.64 | 0.979 |\n| Dynet     | Rain100L| 16M      |     ~72h          | ~24G        | 37.80 | 0.981 |\n| Ours      | Rain100L| 697K     | ~8h           | ~8G        | 37.81 | 0.981 |\n\n\nTableB Effectiveness and efficiency comparison on Denoising($\\sigma=50$).\n\n| Methods   | dataset  | #param   | Training time | GPU memory | PSNR  | SSIM  |\n|-----------|----------|----------|---------------|------------|-------|-------|\n| AirNet    | Urban100 | 8.75M    | ~48h          | ~11G       | 28.88 | 0.871|\n| PromptIR  | Urban100 | 97M      | ~84h          | ~128G      | 29.39 | 0.881|\n| IDR       | Urban100 | 15M      | ~72h          | ~23G       | 29.38 | 0.878|\n| Dynet     | Urban100 | 16M      | ~72h          | ~24G       | 29.52 | 0.881|\n| Ours      | Urban100 | 697K     | ~8h           | ~8G        | 29.70 | 0.881|\n\nFrom the comparison results with recent SoTA all-in-one methods, it can be seen that our PETL-based paradigm can achieve better performance gains than exiting all-in-one paradigm, while costs less training time, GPU memory as well as storage room. These experiments demonstrate it is promising to develop the PETL-based framework for image restoration.\n\n\n### [Q2-Visual comparison with other baselines]\n\n> In terms of Fig. 9 and Fig. 10, the visual comparison is not obvious among Adapter, LoRA and Ours.\n\n\nWe are sorry for this confusing presentation. Due to the selected image region is still not small enough, it may be difficult to get a straightforward visual comparison. You may zoom in on the results in Fig.9 and Fig.10. For example,  in the second figure of Fig.10, previous methods produces blur edges of the character 'ink', while ours can obtain sharp character edge with less noise. The quantitative PSNR comparison can also support the observation. We will revise the figure in the revision.\n\n\n\n### [Q3-Testing time for  effectiveness comparison]\n\n\n>  It is expected to present the testing time rather than training time to evaluate its effectiveness\n\n\nAs suggested, we compare the testing time of existing all-in-one methods AirNet, PromptIR, IDR, and DyNet, with the proposed AdaptIR. The experiments are conducted with 3090 GPUs. We give the testing time of different methods on Urban100 datasets.\n\n| Methods   | #param   | PSNR on denoising (dB) | Testing time (s/img) |\n|-----------|----------|------------------------|-----------------------|\n| AirNet    | 8.75M    | 28.88                 | 0.53                 |\n| PromptIR  | 97M      | 29.39                 | 1.33                 |\n| IDR       | 15M      | 29.38                 | 0.86                 |\n| Dynet     | 16M      | 29.52                 | 0.76                 |\n| Ours      | 697K     | 29.70                 | 0.72                 |\n\n\nFrom the above results, it can be seen that our AdaptIR achieves the best performance using moderate testing latency. It should be noted that the latency of the PETL-based paradigm mainly comes from the pre-trained models, e.g., 99.2% of our AdaptIR comes from the pre-trained models. Therefore, the inference speed can be  potentially further improved with future smaller pre-trained models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pdg6S8O8Sm", "reviewer": "Reviewer_jwPV", "summary": "This paper introduces the mixture-of-expert approach to the image restoration community, enabling rapid adaptation of pre-trained models to various image restoration tasks. The proposed AdaptIR framework comprises three parallel branches: local interaction, channel gating, and frequency affine modules, which together extract heterogeneous representations. AdaptIR consistently achieves performance improvements on both single-degradation and hybrid-degradation tasks across two baseline models.", "questions": "Please see Weaknesses.", "limitations": "Authors claimed the limitation and board impact in the Appendix.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper introduces the parameter-efficient transfer learning paradigm to low-level vision tasks, designing the AdaptIR module, which inserts a few trainable parameters into frozen pre-trained restoration backbones.\n2. The proposed AdaptIR adapts the pre-trained model with heterogeneous representations across tasks by applying three parallel branches, excelling in local spatial, global spatial, and channel representations.\n3. Consistent improvements across various image restoration tasks demonstrate the effectiveness and robustness of AdaptIR.", "weaknesses": "1. The proposed AdaptIR combines three experts in local spatial, global spatial, and channel representations and adaptively weights them for adapting to downstream tasks. The authors are suggested to analyze the distribution of feature response intensity of these three branches across various tasks. This analysis will be crucial to evaluate the adaptability and flexibility of the proposed approach in varying tasks.\n2. Experiments exhibit adaptability on image restoration tasks, consistent with the pre-trained stage. The authors are encouraged to conduct generalization experiments on new tasks and new degradation levels within the same task to further validate their approach.\n3. Authors claim that the fine-tuning step on downstream tasks requires 500 epochs of optimization. This is excessive for a fine-tuning strategy. If so, can the proposed method still be considered efficient transfer learning?\n4. AdaptIR requires heavy fine-tuning on the downstream task. However, prompt-based approaches [1,2,3,4] use various task-specific prompts for task generalization without a fine-tuning stage. Please discuss the advantages of AdaptIR compared to these prompt-based approaches.\n\n[1] ProRes: Exploring Degradation-aware Visual Prompt for Universal Image Restoration. Arsiv, 2023. \n\n[2] PromptIR: Prompting for All-in-One Blind Image Restoration. Arxiv, 2023. \n\n[3] PromptRestorer: A Prompting Image Restoration Method with Degradation Perception. NeurIPS, 2023. \n\n[4] Unifying Image Processing as Visual Prompting Question Answering. ICML, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### [Q1: Analysis of feature response intensity]\n\n> The proposed AdaptIR combines three experts in local spatial, global spatial, and channel representations and adaptively weights them for adapting to downstream tasks. The authors are suggested to analyze the distribution of feature response intensity of these three branches across various tasks. This analysis will be crucial to evaluate the adaptability and flexibility of the proposed approach in varying tasks.\n\nAs suggested, we give the distribution of feature response intensity of three branches across various tasks, including SR, heavy deraining, light deraining, low-light image enhancement, and two hybrid degradations. Since OpenReview cannot post figures, these figures are given in Fig.C in the **attached rebuttal PDF**. \n\n\nThese figures indicate that our AdapIR can adjust to different degradation types by enhancing or suppressing the outputs from different branches. Specifically, for the heavy&light deraining tasks, AdaptIR adaptively learns to enhance the low-frequency global features, i.e., the frequency affine module which is responsible for global spatial modeling has large values. This property ensures the removal of the high-frequency rainstreaks as well as the preservation of the global structure of the image. For SR tasks, AdaptIR adaptively enhance the restoration of local texture details by learning large output values from the local spatial modules. For the hybrid degradation task, AdaptIR shows it can distinguish between different hybrid degradations, i.e., three branches exhibit different patterns under two types of hybrid degradations.\n\nIn short, each branch of AdaptIR can capture discriminative features under different degradations, indicating that our approach is degradation-aware. This ability guarantees the robustness on single degradation, and superior performance under hybrid degradation.\n\n\n\n### [Q2: Generalization experiments on new tasks]\n\nExperiments exhibit adaptability on image restoration tasks, consistent with the pre-trained stage. The authors are encouraged to conduct generalization experiments on new tasks and new degradation levels within the same task to further validate their approach.\n\n\nThanks for your advice. In fact, the original pre-trained models are pre-trained only on SR, denosing and light-deraining. Therefore, the other tasks involved in this paper can, to some extent, demonstrate the robustness of our method on unseen degradation levels or types, which includes hybrid degradations, heavy-deraining, low-light enhancement. \n\nMoreover, as suggested, we further give another experiment on unseen Real-world Denosing tasks to further demonstrate the generalization of our AdaptIR.\n\nTableA real-world denosing tasks with SIDD datasets. \n\n| Methods   | #param   | PSNR   |\n|-----------|----------|--------|\n| AdaptFormer | 677K    | 39.03  |\n| LORA     | 995K    | 38.97  |\n| Adapter  | 691K    | 39.00  |\n| FacT     | 537K    | 39.02  |\n| MoE      | 667K    | 39.05  |\n| Ours     | 697K    | 39.10  |\n\nIt can be seen that our AdaptIR maintains its superiority when transferring to real-world degradation, demonstrating the robustness of our methods.\n\n\n\n### [Q3: Reclaim of the Efficiency]\n\n> Authors claim that the fine-tuning step on downstream tasks requires 500 epochs of optimization. This is excessive for a fine-tuning strategy. If so, can the proposed method still be considered efficient transfer learning?\n\n\nWe apologize for the confusion. Actually, existing restoration works usually use the **dataset enlarge strategy** to enlarge the number of samples in one epoch by repeating training data. Therefore, the current training paradigm requires 10 or even 100 times epochs than ours when under a fair setting. This analysis is also supported by the actual training time. For example, the current training paradigm takes a long time to converge, e.g. >2 days+8x3090GPUs for training the SR model, even one week+8x3090GPUs for denoising, as well as the large costs of training all-in-one models given in Table 2. In contrast, our approach requires only <8h+ 1x3080Ti GPU  of training time to adapt the model to unseen degradation levels or even types.\n\n\n### [Q4: Discussion with prompt-based methods]\n\n> AdaptIR requires heavy fine-tuning on the downstream task. However, prompt-based approaches use various task-specific prompts for task generalization without a fine-tuning stage. Please discuss the advantages of AdaptIR compared to these prompt-based approaches.\n\nTo the best of our knowledge, only the PromtGIP shows the zero-shot ability when facing unseen degradations. And the other ProRes, PromptIR, and PromptRestorer can only handle degradations which have seen during training, which means **they still needs additional fine-tuning for task generalization**.\n\n\nThe advantages of AdaptIR compared to these prompt-based approaches are two-folds. As for **efficiency**, PromptIR, ProRes, PromptRestorer all needs full fine-tuning for adapting to new tasks, e.g. PromptIR needs 7-day 8x3090 GPUs for full fine-tuning, while our AdaptIR needs only 8h 1x3090 GPU. As for **performance**, since these methods need to learn multiple degradation within one model, it is inevitable to suffer the problem of negative transfer, which impairs performance. We give a thorough comparison as follows.\n\n\n| Methods| Type| Fast Adaptation To unseen task | Adaptation Cost | PSNR on denoising | PSNR on deraining |\n|----------------|-------------|--------------------------------|-----------------|------------------|-------------------|\n| PromptGIP| Prompt-based| Yes| zero-shot | 26.22| 25.46|\n| ProRes | Prompt-based| No | 8x3090GPUs| Not open-source  | Not open-source   |\n| PromptIR| Prompt-based| No| 7-days 8x3090GPUs| 29.39 | 37.04 |\n| PromptRestorer | Prompt-based| No| 8x3090GPUs | Not open-source  | Not open-source   |\n| Ours | PETL-based  | Yes | 8h 1x3090| 29.70 | 37.81|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## [Global Author Rebuttal]
We would like to express our sincere gratitude to all the reviewers for taking their time reviewing our work and providing fruitful reviews that have definitely improved the paper. And we are encouraged that reviewers find 

- "exploring PETL for image restoration is meaningful" (Reviewer Y1LM, Reviewer EhxW)

- finding a new problem that "directly applying current PETL methods like LoRA to IR can result in unstable performance" due to the homogeneous representation. (Reviewer EhxW)

- introducing "a heterogeneous MoE framework to enable more robust and effective image representation learning for restoration tasks." (Reviewer 62kc)

- "consistent improvements across various image restoration tasks demonstrate the effectiveness and robustness of AdaptIR"(Reviewer jwPV, Reviewer EhxW)

-  and "the experiments are quite comprehensive". (Reviewer Y1LM, Reviewer EhxW)

During the rebuttal period, we have tried our best and made a point-to-point response to address the concerns raised by reviewers. We also use additional figures for the rebuttal, which can be found in the attached PDF. If you have any further questions,  we will actively discuss with you during the author-reviewer discussion period.

Meta-review: This paper received positive scores of (5, 5, 5, 7). Reviewers generally found the exploration of PETL for restoration interesting and acknowledged the effectiveness of the proposed approach. After reviewing the paper, reviews, and rebuttal, the AC finds no reason to overturn the consensus to accept the paper.

However, the AC would like to address the presentation of training time for AdaptIR in Table 2 and the rebuttal. While the model itself may be fine-tuned in eight hours, this does not account for the pre-training time required for IPT and EDT. To avoid misrepresenting the overall time investment, the authors are strongly advised to clearly separate and report pre-training and fine-tuning times. The authors are also advised to incorporate all reviewer feedback into the camera-ready version.
Decision: Accept (poster)