PromptRestorer: A Prompting Image Restoration Method with Degradation Perception
Authors: Cong Wang
Jinshan Pan
Wei Wang
Jiangxin Dong
Mengzhu Wang
Yakun Ju
Junyang Chen
Venue: NeurIPS 2023 poster (2023)
Keywords: Degradation Vanishing
Prompting Learning
Image Restoration

Abstract: We show that raw degradation features can effectively guide deep restoration models, providing accurate degradation priors to facilitate better restoration. While networks that do not consider them for restoration forget gradually degradation during the learning process, model capacity is severely hindered. To address this, we propose a Prompting image Restorer, termed as PromptRestorer. Specifically, PromptRestorer contains two branches: a restoration branch and a prompting branch. The former is used to restore images, while the latter perceives degradation priors to prompt the restoration branch with reliable perceived content to guide the restoration process for better recovery. To better perceive the degradation which is extracted by a pre-trained model from given degradation observations, we propose a prompting degradation perception modulator, which adequately considers the characters of the self-attention mechanism and pixel-wise modulation, to better perceive the degradation priors from global and local perspectives. To control the propagation of the perceived content for the restoration branch, we propose gated degradation perception propagation, enabling the restoration branch to adaptively learn more useful features for better recovery. Extensive experimental results show that our PromptRestorer achieves state-of-the-art results on 4 image restoration tasks, including image deraining, deblurring, dehazing, and desnowing.

PromptRestorer: A Prompting Image Restoration Method with Degradation Perception
Cong Wang'; Jinshan Pan?*, Wei Wang Jiangxin Dong", Mengzhu Wang',
Yakun Ju', Junyang Chen
The Hong Kong Polytechnic University. "Nanjing University of Science and Technology, Dalian University of Technology, 'Hebei University of Technology, "Shenzhen University
Abstract
We show that raw degradation features can effectively guide deep restoration models, providing accurate degradation priors to facilitate better restoration. While networks that do not consider them for restoration forget gradually degradation during the learning process, model capacity severely hindered. To address this, we propose Prompting image Restorer, termed as PromptRestorer. Specifically, PromptRestorer contains two branches: a restoration branch and a prompting branch. The former used to restore images, while the latter perceives degradation priors to prompt the restoration branch with reliable perceived content to guide the restoration process for better recovery. To better perceive the degradation which is extracted by a pre-trained model from given degradation observations, we propose prompting degradation perception modulator, which adequately considers the characters of the self-attention mechanism and pixel-wise modulation, to better perceive the degradation priors from global and local perspectives. To control the propagation of the perceived content for the restoration branch, we propose gated degradation perception propagation, enabling the restoration branch adaptively learn more useful features for better recovery. Extensive experimental results show that our PromptRestorer achieves state-of-the-at results on 4 image restoration tasks, including image deraining, deblurring, dehazing, and desnowing.
Introduction
Image restoration aims to recover clear high-quality images from given degraded ones. is highly ill-posed since only degraded images can be exploited, statistical observations are thus required to well-pose the problems [37, 67, 66, 41, 42]. Although conventional approaches can recover images to some extent, they typically involve solving optimization algorithms that are difficult due to the non-convexity and non-smooth problems. Additionally, the observations may not always hold, which cause algorithms fail.
With the emergence of convolutional neural networks (CNNs) [48] and Transformers [22, 45], which perform well at implicitly learning the priors from large-scale data, learning-based methods have dominated recent image restoration tasks and achieved impressive performance [81, 51, 103, 72, 61, 93, 33, 13]. However, these methods are usually built without explicitly considering the specific degradation information, which accordingly limits model capacity (Case 1 in Fig. 1). An alternative approach is to design conditional branch to learn additional information to provide the restoration network with useful content for modulation [30, 17, 34, 35] (Case 2 in Fig. 1). However, we note that while conditional branches in these models learnable, they may effectively provide degradation information, as the optimizable parameters result gradually clearer features during the learning process, leading the degradation vanishing which accordingly limits model performance.
'These authors equally contribute to this work.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Degraded Image
Input
Etc.
Dec
Output
Case It Single Brunch
Cas
powe (1n)
12
Cau
Input
Input
Iic.
Des
Output
Condition
Case
4823
ic, Kemei Seneatioolapat
Case Leamible
24 F7842214 30 23 20:30
4823
Jpeation (110P)
36.
Input
Des
Output
4823
Case 3
Pre-zaance
Modd
4823
Leamable
GT
Chse 3: Raw Degregstion As Promping
(a) Framerwark Comparisons
(b) Learning Curves
(c) Visual Performance
Figure 1: (a) compares different restoration frameworks. Unlike existing approaches that are built within the architectures such as Cases 1-2, which are unable to memorize the degradation well during the learning process, we propose prompting method (Case 3) that directly exploits raw degradation features extracted by a pre-trained model from the given degradation observations to guide restoration. In (b), observe that both Cases 1-2 outperform our method in early iterations, they effectively memorize degraded information. However, both Cases 1-2 experience degradation vanishing with further iterations (better demonstrated in Sec. 4.3), while our prompting method persists in guiding the restoration network with accurate degradation priors, accordingly producing better restoration quality. In (c), visual performance demonstrates that our prompting method recovers sharper images. Quantitative results are reported in Tab. 5.
Recently, prompt learning has been shown an effective tool to improve model performance by designing various prompts [115, 98, 104, 23, 114, 46, 53, 43]. The prompt usually serves as the guidance tool to correct the networks toward better results [28]. However, prompt learning still keeps a margin for image restoration, and existing prompts may not be suitable for image restoration since they cannot effectively model degradation priors well. Hence, we ask: Is there a reasonable prompting manner to correct degraded image restoration networks to facilitate better recovery?
The answer is in the riddle. This paper proposes the PromptRestorer, Prompting image Restorer, to overcome degradation vanishing in image restoration via promoting by exploring degradation input itself for better restoration (Case 3 in Fig. 1). Our idea is simple: we directly exploit the raw degraded features extracted by pre-trained model from the degraded inputs to generate more reliable prompting content to guide image restoration. Raw degraded features preserve accurately degraded information, which can consistently prompt the restoration network with accurate degraded priors, enabling the restoration network to perceive the degradation for better restoration. Hence, design the PromptRestorer, which consists of two branches: (a) the restoration branch and (b) the prompting branch. The former is used to restore images and the latter is used to generate reliable prompting features to guide the restoration network for better restoration. To better perceive the degradation, we propose Prompting Degradation Perception Modulator (PromptDPM), which consists of Global Prompting Perceptor (G2P) and Local Prompting Perceptor (L.2P). The G2P adequately exploits the self-attention mechanism form global prompting attention, while the I.2P considers the pixel-level perception to build local prompting content. To control the propagation of perceived features in the restoration branch, we propose Gated Degradation Perception Propagation (GDP), enabling the restoration network to adaptively learn more useful features to facilitate better restoration.
The main contributions of this work summarized below:
We propose PromptRestorer, which is the first approach our knowledge that takes advan- tage of the prompting learning for general image restoration by considering raw degradation features restoration, enabling the restoration model to overcome degradation vanishing while consistently retaining the degradation priors to facilitate better restoration.
We propose a prompting degradation perception modulator that is used to perceive degra- dation from global and local perspectives, which is able to provide the restoration network

---

with more reliable perceived content learned from the degradation priors, enabling it to better guide the restoration process.
We propose gated degradation perception propagation that exploits a gating mechanism to control the propagation of the perceived features, enabling the model to adaptively learn more useful features for better image restoration.
Fig. I summarises framework comparisons, and their learning curves and visual performance. Deeper analysis and discussion about them are provided in Sec. 4.2.
2 Related Work
In this section, we review image restoration, conditional modulation, and prompt learning.
Image Restoration. Recently, CNN-based architectures [110, 112, 103, 4, 24, 102, 89, 91, 87, 19, 88, 117] and Transformer- cansffrmmrr-bsedd models [96, 56, 49, 12. 93, 93] have been shown to outperform conventional restoration approaches [37, 83, 64, 47, 7, 79]. These learning-based methods usually adopt U-Net architectures [50, 18, 103, 100, 1, 93, 108, 101], which have been demonstrated the effectiveness because of hierarchical multi-scale representation and effective learning between shallow and deeper layers by skip connection [111, 59. 102, 31]. We refer the readers to recent excellent literature reviews on image restoration [5, 54, 82], which summarise the main designs in deep image restoration models.
Although these models have achieved promising performance, they do not explicitly take degradation into consideration for model design which is vital for restoration, limiting the model capacity.
Conditional Modulation. Conditional modulation usually involves implicitly modulating the ad- ditional content to guide the restoration [30, 10, 16, 17, 36, 35, 34, 60, 57, 92]. These approaches usually contain two branches: basic network and conditional network. The conditional network provides additional information to guide the basic network for restoration via spatial feature transform (SFT) [92]. Among these methods, blur kernel [30], semanties [92], and degraded input [57, 16] which serve as the additional conditions are broadly known.
The learnable nature of the conditional network in these models does not effectively provide degrada- tion information for the basic network. parameters are optimized in the learning process, features become gradually clear.
Prompt Learning. Prompt learning methods have been studied broadly in natural language process- ing (NLP) [75, 78, 8). Due to high effectiveness, prompt learning is recently used in vision-related tasks [115, 98, 104, 23, 114, 46, 53, 43, 71, 29, 40, 86, 28]. In vision prompt learning, many works seek useful prompts to correct task networks toward better performance [28].
Although prompt learning has shown promise in various vision tasks, it still keeps margin in general image restoration. This paper proposes an effective prompting method, enabling the restoration model overcome the degradation vanishing in the learning process for better restoration.
PromptRestorer
Our goal aims overcome degradation vanishing and better perceive degradation in deep restoration models to improve image recovery quality. To achieve this, we introduce prompting strategy that helps the model consistently memorize degradation information, enabling it to prompt restoration with better degradation for better restoration. To better perceive degradation, Degradation Perception Modulator (PromptDPMD, which can provide more reliable perceived content to guide the restoration network. To control the propagation of the perceived content, propose the Gated Degradation Perception Propagation (GDP), enabling the restoration network to adaptively learn more useful features for better restoration.
propose the Prompting
3.1 Overall Pipeline
Fig. 2 shows the framework of our PromptRestorer, which contains two branches: (a) the restoration branch and (b) the prompting branch. The restoration branch is used to restore images, where each block is prompted by the prompting branch. The prompting branch first generates the accurate
3

---

(3)
Restoration Branch
Continuous Galed Transformer
Input (1)
Output (f))
Down X2
Down X2
Up X2
Up X2
GDF
Prompt DPM
Prompi DPM
Prompe
Prompt Y:
Prompt
Frozen Model
X
Addition
Y
Ellement-wise
Multiplication
Degradation Features
Improved ConvbieXi fh
Y
Y
G2P
L2P Seaa
Pre-trained Model
Y
Prompting
Transformer Block (TFB)
Prompting Degradation Perception Modulator (PromptDPM)
Figure 2: Overall pipeline of our PromptRestorer. PromptRestorer contains two branches: (a) the restoration branch and (b) the prompting branch. The restoration branch is used to restore images, where each block (c) in CGT is prompted by the prompting branch. The prompting branch first generates precise degradation features extracted by pre-trained model from degradation observations, then these features prompt the restoration branch to facilitate better restoration via PromptDPM (d).
degradation feature extracted by pre-trained model, and then the feature is to prompt the restoration branch, enabling the restoration branch to better perceive the degradation prior for better recovery.
Restoration Branch. Given degraded input image I Hf Wxx x3 we first applies a 3 3 convolution the feature extraction to obtain low-level embeddings Xg E R "2xx xC; where H x W denotes the spatial dimension and C' is the number of channels. Next, the shallow features X, gradually are hierarchically encoded into deep features X, E R # x 4 xtC After encoding the degraded input into low-resolution latent features X3 E R # # x3C the decoder progressively recovers the high- resolution representations. Finally, a reconstruction layer which contains 4 Transformer blocks as the refinement followed by 3 3 convolution is applied to decoded features to generate residual image S E # W x3 to which degraded image is added to obtain the restored output image: f = I+ S. Both encoder and decoder at level consist of multiple Continuous Gated Transformers (CGT) with expanding channel capacity. To help better recovery, the encoder features concatenated with the decoder features via skip connections [74] by 1 1x1 convolutions.
Prompting Branch. The prompting branch, as shown in Fig. 2(b), aims to generate and perceive degradation features and then provide useful guidance content for the restoration branch. We note VQGAN [25] has been demonstrated that it can generate high-quality images while representing the features of input images. However, tends to damage image structure after vector quantization [116, 11, 32]. To avoid this problem, we only exploit the encoder of pre-trained VQGAN to represent the deep features the degraded inputs. We first use the pretrained encoder to extract degraded features Y W xIC, where 1 denotes the I- level layer in the pre-trained encoder. Then, the degraded features are exploited to generate reliable prompting content to prompt the restoration branch by PromptDPM (see Sec. 3.2). The generated prompting content is transmitted to each CGT to guide the restoration branch.
Continuous Gated Transformers. CGT exploits the perceived features from PromptDPM to provide the Transformer block with more reliable content overcome degradation vanishing to facilitate better restoration. Each CGT consists of three Transformer blocks (Fig. 2(c)) with residual connections [38] and the input in each block is gated by GDP (see Sec. 3.3) to control the propagation of perceived features. Let PP G, and T respectively denote the operations of PromptDPM (expressed in (2)). GDP (expressed in (7)), and Transformer, the features flow in k block in one CGT at l- level encoder/decooder, which can be expressed as:
X = T(G--): Gk-1 ((X+-I.P):PP P(X YI),
(1)

---

Y
(a) G2P
(b) 1.2P
[ Coss
F10mm
Lays Nems
Quindiooi
Porcepiicn Modellsrr
Poreopiart
Degradation Indaced Band
Layer
GELAI Fx Cney
CossNeXi
Splinas
+
KV-Indaod Attetion
3 Matric Multplication
Redlaye
Reskeratirn Trchlood Bund
Figure 3: (a) Global Prompting Perceptor (G2P); (b) Local Prompting Perceptor (L2P).
where X; means the output of 4thh Transformer block in one CGT, especially X, is the downsam- pled/upsampled features at (1 1)- level encoder/decoder; Pz refers to the generated features of PromptDPM at [-level layer; G means the gated features between X and P,, which serves as the input of htth Transformer block. Each Transformer block consists of multi-head attention [101] followed by an improved ConvNeXt [62] as the feed-forward network (see Fig. 2(c)).
3.2 Prompting Degradation Perception Modulator
To better perceive the degradation to prompt the restoration network with more reliable perceived content from the degradation priors, we propose the PromptDPM (see Fig. 2(d)). The PromptDPM consists of Global Prompting Perceptor (G2P, introduced Sec. 3.2.1) and 2) Local Prompting Perceptor (L2P, introduced in Sec. 3.2.2) to respectively perceive the degradation from global and local perspectives, enabling to generate more useful content to guide the restoration branch. From
restoration tensor X E x xc and a degradation tensor Y E IR RR we prompt X with Y:
(C P(X Y) Wp (c glllball (X,Y), Y), v local (X, Y) +X.
(2)
where (ıglohal (..)) and locaal (:.-) respectively denote the operations of G2P and L2P: C(+, means the concatenation at channel dimension; W.(+) refers to the x point-wise convolution.
3.2.1 Global Prompting Perceptor
The G2P, shown in Fig. 3(a), fully exploits the self-attention mechanism to form the global prompting attention induced by the degraded features. The G2P contains the global perception attention followed by an improved ConvNeXt [62]. Our global perception attention consists of 1) Query- Induced Attention (Q-InAtt) and 2) Key- Value-Innucce Attention (KV-InAtt). The QnInAtt considers re-forming the query vector induced by degradation features build representative query to perform attention, while the KV-InAtt re-considers key and value vectors induced by other degradation counterparts search for more similar content with the restoration query. From
layer normalized restoration tensor X E R fxxxx our G2P first generates restoration query (Q), key (K), and value (V) projections from the restoration features. convolutions to aggregate pixel-wise cross-channel context followed by 3x3 depth-wise convolutions Wa(-) encode channel-wis wise spatial context, yielding Q=WaW, p.X. K-W,W,X, and V=WaW,..
achieved by applying 1x1
Meanwhile, we similarly convert the degradation tensor Y E R x ò into degradation query (Q), key (K), and value (V) projections: Q=W,W,Y, K KWW,W, Y, and V=WaW,Y. Then, we respectively conduct Q-InAtt and KV-InAtt:
Aq.InAn AQ. W, (CiQ. QI) K. v): AKVAInAn :
AKV.nAu (Q. W, (C (CK, W, VI))
where Ae) (Q:K, v) = - Softmax K Q/a): Here, a learnable scaling parameter control
5

---

the magnitude of the dot product of K and Q before applying the softmax function. Similar to the conventional multi head SA [22], we divide the number of channels into 'heads' and lear separate attention maps. Then two induced attentions are fused and followed by an improved ConvNeXt:
A = (C[AAA AKVIAA)) + X; A = W,Waow, W. a (LN(AA) )) + +A', where the W W
(4)
means the improved ConvNeXt shown in the latter Fig. 2(c); LN(-) means the operation of layer normalization [6].
3.2.2 Local Prompting Perceptor
The L2P, as shown in Fig. 3(b), adequately considers the pixel-Ievel degradation perception, enabling to better perceive degradation from spatially neighboring pixel positions. The L2P consists of a local perception modulator followed by a separable depth-level convolution. The local perception modulator contains two core components: 1) Degradation-Indee
Band (Deg-InBan) and 2) IintorationIndducer Band (Res-InBan). The former is achieved by exploiting the degradation features to induce spatially useful content from restoration content to guide restoration gating fusion, while the latter utilizes the deep restoration features to induce more useful features from another degradation counterpart to form the degradation gating. Given the degradation tensor Y TR x xc we first exploit the point-wise convolution and 3 x 3 depth-wise convolution to encode two degradation projections, yielding @-W-wey p and K-W Meanwhile, the restoration tensor X E BR xC are also encoded into two restoration projections: @= and K- w.W.WWKK X. Then, we respectively conduct Deg-InBan and Res-InBan:
Dog-lIBan = a WW,oWa(C(@
Q1) @ K; ZRes-InBan = Q o (Wa WWoCK(
(5)
where (.) denotes the sigmoid function that controls the gating level. Then, the perceived features in the two bands are fused via concatenation and x convolution and followed by a depth-level separable convolution W WppWa())
z' Wp (C[Z InBan, ZRss InBaan) + +: W WpoWa(Z) +Z
3.3 Gated Degradation Perception Propagation
The GDP aims to control the propagation of the perceived degradation, enabling to adaptively learn more useful features in Transformer blocks to facilitate better restoration. Given the output restoration tensor X E IR. xxxx of (A: - 1) 1)th Transformer block one CGT and the perceived tensor P. E R #xxWxc xc which is the output feature of one PromptDPM at (- level, the input of t Transformer block can be obtained by gating the X with P. by convolution and gated control function sigmoid (-) with residual learning [39]:
G ((Xx-1,PP) Pt) = (W Pi)OX + +X-11-
3.4 Learning Strategy
To train the network, two objective loss functions adopted, including image reconstruction loss (L.) for pixel recovery and frequency loss (L.)) for detail enhancement [18]:
L L, + >Cy; where L, = | H| C,- = |[F( - F(H)),,
(8)
where H denotes the ground truth image; F denotes the Fast Fourier transform; À is weight that is empirically set to be 0.1.
Experiment
We evaluate PromptRestorer on benchmarks for 4 image restoration tasks: (a) deraining, (b) de- blurring, (c) desnowing, and (d) dehazing. We train separate models for different image restoration tasks. Our PromptRestorer employs a 3-level encoder-decoded From level-1 to level-3, the number of CGT [2, 6]. attention heads are [2, 4, 8], and number of channels is [48, 96, 192]. The expanding channel capacity factor 3 is 4. For downsampling and upsampling, we adopt pixel-unshuffle and pixel-shuffle [77], respectively. We train models with AdamW optimizer with the initial learning rate
3e gradually reduced to le with the cosine annealing [63]. The patch size is set 256

---

Table 1: Image deraining results. Our PromptRestorer advances 14 state-of-the-arts on average.
Method
PSNR
Test100 [1U7]
SSIM
PSNR
Rain10011 (97)
SSIM 1
PSNR.
Raint00L (97)
SSDM
PSNK 1
Test2800
|27] SSIM
PSNK
Test1200
106) SSIM
PSNR
Averuge
SSIM
DerzinNet (26)
22.77
0,810
14.92
01,592
27.03
0.864
24.31
0.361
23.38
0,835
22.45
0.796
SUMI (95)
22.35
0.768
16.55
0.485
25.00
0.842
24.43
0.782
26.05
0.822
22.85
0.744
DIDMON [106)
22.56
0.818
17.35
0.522
25.23
0.741
28.13
0.367
29.65
0.901
24.55
0.770
UMRL (19)
34.41
0.829
26,11
11.532
29.18
0.923
20.97
0.911
30.55
0.910
28,012
0.880
RESCAN [55]
25,00
0.833
26.36
01.786
29.0 50
0.851
31.29
0.904
30.51
0,882
28.59
0.857
PreNet (72)
24.81
0.51
26.77
01555
12.44
0,950
31.75
0.916
31.36
0.911
25.42
0.897
MSPEN [6s]
27.90
0.876
25.05
0.860
32.40
0.933
32.82
0.930
32.39
0.910
30:75
0.902
DCSEN (90)
27.46
0.887
25.95
0.887
34,70
0.961
30.96
0.903
32.92
0.937
31,00
0.915
MPRN21 [1(E]]
30.27
0.897
30.41
01550
36.40
0,965
33.64
01538
32.91
0.910
32.73
0.923
SPAIR [69]
30.35
0.509
30.95
018922
36.93
0,969
33.34
0.936
33.04
0.922
32.91
0.926
Utormer (93)
29.17
0.880
30.05
01588
36.34
0.956
33.35
0.935
31.98
0.909
32.18
0.915
MAXIM 25 [8-]
31,17
0.922
30.81
0.903
38.06
0,977
33,80
0.943
32.37
0.922
33.24
0.933
Restammer [01]
32,00
0.923
31.46
019844
38,9
0,978
34.15
0.944
35.19
0.936
33.96
0.335
SENe: 120]
31.47
0.919
31,90
0.908
38.21
0.974
33.69
0.937
32.55
0.911
33.56
0.929
PromptRestoer
0.120
34,05
0.936
PSNR (a) Input
20.8 20.83 (b) RESCAN
21.89 dB (c) DCSFN
21.51 dB (d) MPRNet
22.41 dB (e) Restormer (f) PromptRestorer
22.46 dB
DC
(g) GT
Figure 4: Image deraining example Rain [97].
Table 2: Image deblurring results. Our PromptRestorer is trained only on the GoPro dataset [65] and directly applied to the HIDE [76] and RealBlur [73] benchmark datasets.
Benchmurk
Metries
[65]
al.
[81]
DEGAN MEENNN
[109]
[68]
DMPHN
[105]
[80]
al. SPAIR MIMO-UNet+
[69]
[18]
MPRNet Restormer
[103]
[101]
PrompdRestorer
GoPro [65]
PSNR SSIM
21.00
30.26
31.10
31.15
31,20
31.85
32.06
32.45
32.66
32.92
33,06
0.914
0.934
0.942
0.945
0.940
0.948
0.953
0.957
0.959
0.961
0.962
HIDE [76]
PSNR
25.73
28.36
29.15
29.09
29.98
30.29
29.99
30.96
31,22
31,36
SSIM
0.874
0.915
0.915
0.918
0.924
0.930
0.931
0.930
0.939
0.942
0.944
RealBlur-R (73]
32.51
35.79
35.54
SSIM
0.841
0.947
0.909
0.951
0.948
0.947
0.952
0.957
0.954
RealBlur-J
PSNR
27.87
28.56
24.93
28 .4
28.42
28.81
27.63
28.70
28.96
28.82
SSIM
0.827
0.867
0.745
0.862
0,860
0.875
0.837
0.873
0.879
0.873
4853
632 4823
63x 4823
634 4823
6 63X 4823
63X 4823
24.83 dR
31 (e) MPRNet
(a) Degraded Image (b) Degraded Patch
(c) Nah et al.
(d) SRN
(i) PromplRestorer
(g) GTT
Figure 5: Image deblurring example on GoPro [65].
4.1
Main Results
Image Deraining Results. Similar to existing methods [44, 103, 69], we report PSNR/SSIM scores using Y channel in YCbCr color. Tab. shows that our PromptRestorer outperforms current state- of-the-art approaches when averaged across all five datasets. Compared to the recent best method Restormer [101], PromptRestorer achieves 0.09 dB improvement average. On individual datasets, the gain can be as large as 0.22 dB, e.g., Test2800 [27]. Fig. 4, we present a challenging visual deraining example, where our PromptRestorer is able to generate clearer result with finer details.
Image Deblurring Results. We evaluate deblurring results on both synthetic datasets (GoPro [65], HIDE [76]) and real-world datasets (RealBlur-R R [73]. RealBlur-J [73]). Tab. 2 summarises the results, where our PromptRestorer advances current sfate-of-the-art approaches on GoPro [65] and HIDE [76]. Compared with MPRNet [103], our PromptRestorer obtains a performance 0.12 dB gains. Fig. 5 provides visual deblurring example. Our PromptRestorer produces sharper result with fewer artifacts.
7

---

Table Image dehazing results on SOTS-Indoor [52] real-world benchmarks Dense-Haze [2] and NHI-
Haze [3]. Our PromptRestorer significantly advances state-of-the-arts on SOTS-Indoor [52].
Beuchmarc
[Metrics| [37] DelazsNei
[53] GialNet
UHD
PromstRRester
S0TS-Indior (52)
PSNR
SSIM
7
0.8546
0.8209
0.8622
0.9836
32.16
0.9836
0.9812
0.8736
21.75
0.9910
0.9881
36.63
0.9945
Demse-Have
SSIM
7
0.4665
0.326
0.4400
0.551
0.4994
12.16
0.5602
16,62
12:72
20,66
SSIM
0.4419
0.691
0J4612
0.6844
PSNR Iaput
26. 18 dB GridNet
29.14 dB (c) MSBDN
15.32 dB (d) UHD
32.77 dB
36.86 dB
DeHamer
PromptRestorer
(g)
Figure 6: Image dehazing example on SOTS-Indoor [52].
Table Image desmowing results on CSD (2000) [65]. SRRS (2000) [76], and Snow 100K (2000) [73]. Our
PromptRestorer achieves the best metrics on all datasets on the image desnowing problem.
Benchmark
Metrics (esnowNerr [61) JSTASR [14] HDCW Net [15]
TransWeather [85] MSP Former [13] Utcemer [94] Resiemer [101] Promptitstorer
CSD (2000) [15]
SSIM
PSNR
20.13
0.81
27.96
0.88
29.06
0.91
31.76
0.93
$3.75
0.56
33.50
0.96
35.43
0.97
37.48
0.99
SRRS (2000)
PSNR
SSIM
20.38
25.82
0.89
27.78
0.92
38.29
0.92
30.76
0.95
0.96
30.12
32.34
0.96
33.99
0,99
1008. (2000) [61]
SSIM
PSNR
30.50
23.12
31.54
31,82
0.5
33.45
33.81
34,67
36,02
PSNR Input
24. 18 dR JSTASR
26.81 dB HDCWWet
29.07 dB Uformer
29.72 dR Restormer
34.70 dR
PromptRestorer
(g) GT
Figure 7: Image desnowing example on CSD (2000) [15].
Image Dehazing Results. We perform the image dehazing experiments on both synthetic benehmark RESIDE SOTS-Indoor [52], and real-world hazy benchmarks Dense-Haze [2] and NH-Haze [3]. Tab. 3 summarise the quantitative results. Compared to the recent works DeHamer [33] and MAXIM [84], our method receives 4.33 dB and 5.91 dB PSNR gains the SOTS-Indoor, re- spectively. the real-world benchmark NH-Haze [3], our PromptRestorer can achieve 0. 0.7203 SSIM result, which is new record and significantly outperforms current state-of-
approaches DeHamer [33]. The results on both synthetic and real-world benchmarks have demonstrated the effectiveness of our PromptRestorer
the image dehazing task. Fig. 6 shows the visual results, where our PromptRestorer is more effective in removing haze than other methods.
Image Desnowing Results. For the image desnowing task, we compare our PromptRestorer on the CSD [15]. SRRS [14], and Snow 100k [61] datasets with existing state-of-the-r methods [61, 14, 15, 13, 85]. We also compare recent Transformer based general image restoration approaches Restormer [101] and Uformer [93]. As shown in Tab. 4, our PromptRestorer yields a 2.05
PSNR improvement over the state-of-the-art approach [101] on the CSD benchmark [15]. The visual results in Fig. 7 show that our PromptRestorer is able to remove spatially varying snowflakes. than competitors.

---

Table 6: Ablation experiments PromptDPM. Each component in L.2P and G2P is effective.
(a) Effect on L.P. Both Res-InBan and Deg-InBan
(b) Effect G2P. Both Q-InAtt and VV-InAtt play
play positive roles for image restoration.
positive effect on high-quality image restoration.
Experiment
PSNR
FLOPs (G)
Params (M)
Experiment
PSNR
FLOPs (G) Params (M)
w/o L2P
30.819
148.34
12.60
w/o G2P
30.697
123,44
10.69
w/o Res-InBan
30.952
153.39
12.97
w/o Q-InAtt
30.914
148.93
12.65
w/o Deg-InBan
30.964
153.39
12.97
w/o KV-InAut
30.906
147.26
12.52
Full (Ours)
31.015
157.04
13.24
Full (Ours)
31.015
157.04
13.24
4.2 Analysis and Discussion
For ablation experiments, following [84, 20], train the image deblurring model on GoPro dataset [65] for 1000 epochs only and set the number of Transformer in each CGT is 1. Params mean the number of learnable parameters. Testing is performed the GoPro testing dataset [65]. FLOPs are computed on image size 256x256. Next, we describe the influence of each component individually.
Effect on Prompting. The core design of our PromptRestorer the 'prompting', which ex- ploits a pre-trained model to extract raw degradation features from the degraded observations and then generate perceived content to guide the restoration branch (i.c., Case in Fig. 1). Compared to existing frameworks such Cases 1-2 in Fig. 1, our proposed prompting strategy shows superior performance, as demon- strated in Tab. 5. Our method achieved 0.877 dB gains compared Case 1, and 0.369 dB higher than Case 2. Interestingly, the learnable con- dition branch in Case 2 despite consuming more FLOPs and Params, results in worse perfor- mance than ours. Our approach directly exploits raw degradation features to prompt restoration
Table 5: Effeet on prompting, Our method that directly exploits the raw degradation to prompt restoration performs better.
Case Fig.
PSNR
FLOPss (G)
Params (M)
1
30.138
105.58
10.16
2
30.646
157.04
16.77
3 (Ours)
31.015
157.04
13.24
with persistent degradation priors facilitate better recovery. Fig. I(c) shows two examples, where our model that exploits raw degradation features as prompting generates sharper and clearer images.
Effect on PromptDPM. We analyze the impact of PromptDPM on restoration quality in Tab. by disabling one component time. Each model in L2P and G2P consumes similar Params and FLOPs, while our full model achieves the best performance. Disabling L.P or G2P results in a decrease in performance by 0.196 dB and 0.318 dB, respectively. These experiments conclusively demonstrate the effectiveness each component in I.2P and G2P for restoration.
Effect on GDP. To understand the impact of GDP, Note that the computational cost of the GDP is negligible compared to disabling it as it only involves a convolution and sigmoid func- tion for the gating mechanism, while it leads to a gain of 0.091 dB. This finding highlights the significance of controlling the propagation of the perceived degradation features.
we disable it to compare with full model in Tab. 7. Table 7: Effect on GDP. Our GDP which controls the degradation propagation is effective.
Experiment
PSNR
FLOPS (G) Params (M)
w/o GDP
30.924
154.60
12.95
w/ GDP (Ours)
31.015
157.04
13.24
4.3 Visualization Understanding for Degradation Vanishing
To emphasize the understanding of degradation vanishing, we visualize the features learned in the condition/prompptnng branches to better comprehend the learned status of these branches in Fig. 8. Notably, both Cases 1-2 exhibit sharper results in later iterations compared to carlier ones, which fail to provide the restoration branch with sufficient degraded information and cause the restoration
We ensure fairness in the comparison by using the same network architecture in the condition branch of Case 2 and the encoder of the pre- are-trained model in network.

---

models to not perceive the degradation well, thereby hindering the model capacity. In contrast, as the restoration branch needs to adapt perceived features from the PrompiDPM which is perceive the raw degradation features from inputs, our model (Case 3) initially exhibits inferior performance (around 20K iterations) as shown in Fig. 1(b). However, with better adaptation to the degradation information after more iterations, the prompting branch can better prompt the restoration branch consistently with more reliable perceived content learned from the raw degradation, enabling our restoration branch to overcome degradation vanishing and improve restoration quality, shown in Fig. I(c).
1OX
S00K Derntione
IOK Humation
500K Dastioes
Candorree Degrodal Fores
Figure 8: Visualization. We show the average features the channel dimension in the condi- tion/prompting branches for the second example in Fig. I(c). We obtain GT/degraded features by inputting GT/degraded images into the pre-trained VQGAN. As single-branch models (Case I in Fig. 1) do not have condition branches, we visualize the last layer in the 1-level encoder for reference.
5 Concluding Remarks
In this paper, we investigate the degradation vanishing in the learning process for image restoration. To solve this problem, we have proposed the PromptRestorer which explores the raw degradation features extracted by pre-trained model from the given degraded observations to guide the restoration process facilitate better recovery. Extensive experiments have demonstrated that our PromptRestorer favors against state-of-ther the-ttr-rr approaches on 4 restoration tasks, including image deraining, deblurring, dehazing, and desnowing.
References
[1] A. Abuolaim and M. S. Brown. Defocus deblurring using dual-pixel data. In ECCV, 2020.
[2] C. O. Ancuti, C. Ancuti, M. Shert, and R. Timofte. Dense-haze: A benchmark for image dehazing with dense-haze and haze-free images. In ICIP, pages 1014-1018, 2019.
[3] C. O. Ancuti, C. Ancuti, and R. Timofte. Nh-haze: An image dehazing benchmark with non non-hon homogeneous hazy and haze-free images. In CVPR workshops, pages 444 445, 2020.
[4] Anwar and N. Barnes. Densely residual laplacian super-resolution. TPAMI, 2020.
[5] Anwar, S. Khan, and N. Barnes, A deep jourcy into super-resolutioo:: A survey. ACM Computing Surveys, 2019.
[6] L. Ba, R. Kiros, and G. E. Hinton. Layer normalization. arXtv:1607.06450 2016.
[7] D. Berman, T. Treibitz, and S. Avidan. Non-local image dchazing. In CVPR, pages 1674-1682, 2016.
[8] Brown. Mann, N. Ryder, Subbiah, Kaplan, Dhariwal, Neelakantan, P. Shyam, G. Sastry,
A. Askell, et al. Language models are few-shot learners. M.Xiv:2005.14100 2020.
[9] Cai, X. K. Jia, Qing, and Tao. Dehazenet: end-to-end system single image haze removal. IEEE TIP, 25(11):518-5119 2016.
[10] H. Cai, J. He, Y. Qiao, and C. Dong. Toward interactive modulation for photo-realistic image restoration. CVPR Workshops, pages
2021.
[11] C. Chen, X. Shi, Y. Qin, X. Li, X. Han, T. Yang, and S. Guo. Real-world blind super-resolution via feature matching with implicit high-resolution priors. In ACM MM, pages 1329-1338, 2022.
[12] H. Chen, Y. Wang. T. Guo, C. Xu, Y. Deng. Z. Liu, S. Ma, C. Xu, C. Xu, and W. Gao. Pre-traine trainadddd image processing transformer. In CVPR, 2021.
10

---

[13] S. Chen, T. Ye. Y. Liu, T. Liao, Y. Ye, and E. Chen. Msp-former: Multi-scale projection transformer for single image desnowing. arXiv preprint arXiv:2207.05621, 2022.
[14] W.-.. Chen, H.-Y. Fang, J.-J. Ding, C.-C. Tsai, and S.-Y. Kuo. Jstasr: Joint size and transparency- aware snow removal algorithm based on modified partial convolution and veiling effect removal. In ECCV, pages 754-770, 2020.
[15] W. Chen, H.-Y. Fang, C.-L. Hsieh, C.-C. Tsai, I. Chen, J.JJ Ding, S.- Kuo, et al. All snow removed: Single image desnowing algorithm using hierarchical dual-tree complex wavelet representation and contradict channel loss. In ICCV, pages 4196-4205, 2021.
[16] X. Chen, Liu, Z. Zhang, Y. Qiao, and C. Dong. Hdrunet: Single image hdr reconstruction with denoising and dequantization. In CVPR Workshops, pages 354-363, 2021.
[17] X. Chen, 2. 7hang, J. S. Ren, L. Tian, Y. Qiao, and C. Dong. A new journey from sdrtv to hdrtv. In ICCV, pages 4500-4509, 2021.
[18] S.JJ. Cho, S.-W. Ji, J.-P. Hong, S.-W. Jung, and S.-J. Ko. Rethinking coarse-to-ine approach in single image deblurring. In ICCV, 2021.
[19] X. Cui, C. Wang, D. Ren, Y. Chen, and P. Zhu. Semi-supervised image deraining using knowledge distillation, IERE TCSVT, 3212883377-8,,, 2022.
[20] Y. Cui, Y. Tao, Z. Bing, W. Ren, X. Gao, X. Cao, K. Huang, and A. Knoll, Selective frequency network for image restoration. In ICLR, 2023.
[21] H. Dong, J. Pan, L. Xiang, Z. Hu, X. Zhang, F. Wang, and M. Yang. Multi-scale boosted dehazing network with dense feature fusion. CVPR, pages 2154-2164, 2020.
[22] A. Dosovitskiy, L. Beyer. A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani. M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[23] Y. Du, F. Wei, Z. Zhang, M. Shi, Y. Gao, and G. Li. Learning to prompt for open-vocabulary object detection with lision-languag model. In CVPR, pages 14074,,4143,.--00 2022.
[24] A. Dudhane, S. W. 7amir, S. Khan, Khan, and M.-H. Yang. Burst image restoration and enhancement. In CVPR, 2022.
[25] P. Esser, R. Rombach, and B. Ommer. Taming transformers for high-resolution image synthesis. In CVPR, pages 12873- 12883, 2021.
[26] X. Fu, Huang, X. Ding, Y. Liao, and J. Paisley. Clearing the skies: A deep network architecture for single-image rain removal. TIP, 2017.
[27] X. Fu, J. Huang, D. Zeng, Y. Huang, X. Ding, and J. Paisley, Removing rain from single images via a deep detail network. In CVPR, 2017.
[28] Y. Gan, X. Ma. Y. Lou, Y. Bai, R. Zhang, N. Shi, and L. Luo. Decorate the newcomers: Visual domain prompt for continual test time adaptation. In AAAI, 2023.
[29] P. Gao, Geng, Zhang, T. Ma, R. Fang, Y. Zhang, H. Li, and Y. Qiao. Clip-adapter: Better vision-language models with feature adapters. arXiv preprint arXiv:21 10.04544, 2021.
[30] J. Gu, H. Lu, W. Zuo, and C. Dong. Blind super-resolution with iterative kernel correction. In CVPR, 2019.
[31] S. Gu, L.,, V. Gool, and R. Timofte. Self-guided network for fast image denoising. In ICCV, 2019.
[32] Y. Gu, X. Wang, L. Xie, C. Dong, G. Li, Y. Shan, and M. Cheng. VQFR: blind face restoration with vector-quantized dictionary and parallel decoder. In ECCV, pages 126-143, 2022.
[33] C.-L. Guo, Q. Yan, Anwar, R. Cong, W. Ren, and C. Image dehazing transformer with transmission- aware 3d position embedding. In CVPR, pages 58 12-5820. 2022.
[34] J. He. C. Dong, Y. Liu, and Y. Qiao. Interactive multi-dimension modulation for image restoration. TPAMI, 44(12):3363 9379, 2022.

---

[35] J. He. C. Dong, and Y. Qiao. Interactive multi-dimension modulation with dynamic controllable residual learning for image restoration. In A. Vedaldi, H. Bischof, T. Brox, and J. Frahm, editors, ECCV, volume 12365, pages 53-68, 2020.
[36] J. He, Y. Liu, Y. Qiao, and C. Dong. Conditional sequential modulation for efficient global image retouching. ECCV, volume 12358, pages 679-695. Springer, 2020.
[37] K. He, Sun, and X. Tang. Single image haze removal using dark channel prior. TPAMI, 2010.
[38] K. Hc, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[39] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016.
[40] R. Herzig, O. Abramovich, E. Nen-Avraham, A. Arbelle, Karlinsky, A. Shamir, Darrell, and
A. Globerson. Promptonomyvit: Multi-task prompt learning improves video transformers using synthetic scene data. CoRR, abs/2212.04821, 2022.
[41] 7. Hu, S. Cho, J. Wang, and M.-H. Yang. Deblurring low-light images with light streaks. In CVPR, 2014.
[42] J.-B. Huang, A. Singh, and N. Ahuja. Single image super-resolution from transformed self-exemplars, In CVPR, 2015.
[43] M. Jia. L. Tang, B.-C. Chen, C. Cardie, Belongie, Hariharan, and S.-N. Lim. Visual prompt tuning. In ECCV, pages 709-727, 2022.
[44] Jiang, Wang. P. Yi, B. Huang, Luo, Ma, and Jiang. Multi-scale progressive fusion network for single image deraining. In CVPR, 2020.
[45] Khan, M. Naseer, M. Hayat, S. Zamir, F. S. Khan, and M. Shah. Transformers vision: A survey, arXiv:2101.01 2021.
[46] M. U. Khattak, H. A. Rasheed, M. Maaz, Khan, and F. S. Khan. Maplc: Multi-modal prompt learning. CoRR, abs/2210.001111 2022.
[47] J. Kopf, B. Neuhert, B. Chen, M. Cohen, D. Cohen-Or, Deussen, M. Uyttendacle, and D. Lischinski. Deep photo: Model-hased photograph enhancement and viewing. ACM TOG, 2008.
[48] A. Krizhevsky, Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.
[49] M. Kumar, D. Weissenbom, and N. Kalchbrenner. Colorization transformer. In ICLR, 2021.
2021.
[50] O. Kupyn, T. Martyniuk. J. Wu, and Z. Wang. DeblurGAN-v2. Deblurring (orders-of-magniuude) faster and better. In ICCV, 2019.
[51] B. Li, X. Peng, Z. Wang, J. Xu, and D. Feng. Aod-net: All-in-one dehazing network. In ICCV, pages 4780 4788, 2017.
[52] B. Li. W. Ren, D. Fu, D. Tao, D. Feng. W. Zeng, and Z. Wang. Benchmarking single-image dehazing and beyond. 77P, 28(1 (8(1):492550 2019.
[53] M. Li, L. Chen, Y. Duan, Z. Hu, J. Feng. J. Zhou, and J. Lu. Bridge-prooe prompt: Towards ordinal action understanding in instructional vidcos. In CVPR, pages 19880- 19889, 2022.
[54] S. Li,, I. B. Araujo, W. Ren, Z. Wang, E. K. Tokuda, R. H. Junior, R. Cesar-Junior, J. Zhang, X. Guo, and X. Cao. Single image deraining: A comprehensive benchmark analysis. In CVPR, 2019.
[55] X. J. J. Wu, 7. Lin, H. Liu, and H. Zha. Recurrent squeere-and-exciiation context aggregation net for single image deraining, In BCCV, 2018.
[56] J. Liang, J. Cao, G. Sun, K. Zhang, L. Van Gool, and R. Timofte. SwinIR: Image restoration using swin transformer. In ICCV Workshops, 2021.
[57] X. Liu, J. Hu, X. Chen, and C. Dong. Udc-unet: Under-display camera image restoration via u-shape dynamic network. In L. Karlinsky, T. Michaeli, and K. Nishino, editors, ECCV Workshops, volume 13805, pages 113 129, 2022.
[58] X. Liu, Y. Ma. Z. Shi, and J. Chen. Griddehazenet: Attention- based multi-scale network for image dehazing, In ICCV, pages 2312, 2019.
12

---

[59] X. Liu, M. Suganuma, Z. Sun, and T. Okatani. Dual residual networks leveraging the potential of paired operations for image restoration. In CVPR, 2019.
[60] Y. Liu, J. He, X. Chen, Z. Zhang. H. Zhao, C. Dong, and Y. Qiao. Very lightweight photo retouching network with conditional sequential modulation. TMM, 2022.
[61] Y.FF Liu, D.-W. Jaw. S.-C. Huang. and J.-N. Hwang. Desnownet: Context-aware deep network for snow removal. TIP, 27(6))3064 2018.
[62] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie. A convnet for the 2020s. In CVPR, pages 11976- 1 11986, 2022.
[63] I. Loshchilov and F. Hutter. SGDR: Stochastic gradient descent with warm restarts. In ICLR, 2017.
[64] T. Michaeli and M. Irani. Nonparametric blind super-resolutio. In ICCV, 2013,
[65] S. Nah. T. Hyun Kim, and K. Mu Lee. Deep multi-scale convolutional neural network for dynamic scene deblurring. In CVPR, 2017.
[66] J. Pan, Z. Hu, Z. Su, and M.-H. Yang. to rregularized intensity and gradient prior for deblurring text images and beyond. TPAMI, 39(2):342-355 355, 2017.
[67] J. Pan, D. Sun, H. Pfister, and M.-H. Yang. Blind image dehlurring using dark channel prior, In CVPR, 2016.
[68] D. Park, D. U. Kang, J. Kim, and S. Y. Chun. temporal recurrent neural networks for progressive non-uniform single image deblurring with incremental temporal training. In ECCV, 2020.
[69] K. Purohit, M. Suin, A. Rajagopalan, and V. N. Boddeti. Spatially-adaptive image restoration using distortion-guided networks. In ICCV, 2021.
[70] X. Qin, Z. Wang, Y. Bai, X. Xie, and H. Jia. Ffa-net: Feature fusion attention network for single image dehazing. In AAAL, volume 34, pages 11988-1911,, 2020.
[71] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry. A. Askell, P. Mishkin.
J. Clark, et al. Learning transferable visual models from natural language supervision. ICML, pages 8748-8763. 2021.
[72] D. Ren, W. Zuo, Q. Hu, P. Zhu, and D. Meng. Progressive image deraining networks: better and simpler baseline. In CVPR, 2019.
[73] J. Rim, H. Lee, J. Won, and S. Cho. Real-world blur dataset for learning and benchmarking debluring algorithms. In ECCV, 2020.
[74] O. Ronneberger, P. Fischer, and T. Brox. U-Net: convolutional networks for biomedical image segmenta- tion. In MICCAI, 2015.
[75] T. Schick and H. Schütee. Exploiting cloze-questions for few-shot classification and natural language inference. In EACI, pages 255-269, 2021.
[76] Shen, W. Wang, X. Lu, J. Shen, H. Ling, T. Xu, and L. Shao. Human- motion deblurring. In ICCV, 2019.
[77] W. Shi, J. Caballero, F. Huszár, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert, and Z. Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In CVPR, 2016.
[78] T. Shin, Y. Razeghi, R. L. IV, E. Wallace, and Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. EMNLP, pages 4222-4235, 2020.
[79] H. Singh, Kumar, L. Balyan, and K. Singh. A novel optimally gamma corrected intensity span maximization approach for dark image enhancement. In DSP, pages 1-5, 2017.
[80] M. Suin, K. Purohit, and A. N. Rajagopalan. Spatially-attentive patch-hierarchiiall network for adaptive motion deblurring. In CVPR, 2020.
[81] X. Tao, H. Gao. X. Shen, J. Wang, and J. Jia. Scale-recurrett network for deep image deblurring. In CVPR, 2018.
13

---

[82] Tian, L. Fei, W. Zheng, Y. Xu, W. Zuo, and C.-W. Deep learning image denoising: An overview. Neural Networks, 2020.
[83] R. Timofte, V. De Smet, and L. Van Gool. Anchored neighborhood regression for fast example-based In MCCV/CCV220 2013.
[84] Tu, Talebi, Zhang, Yang. P. Milanfar, A. Bovik, and Y. Li. Maxim: Multi-axis mlp for image processing. In CVPR, pages 5769-5780, 2022.
[85] J. M. J. Valanarasu, R. Yasarla, and V. M. Patel. Transweather: Transformer based restoration of images degraded by adverse weather conditions, In CVPR, pages 2533-2363,22 2022.
[86] C. Wang, J. Pan, W. Lin, J. Dong, and X.-M. Wu. Selfpromer: Self-prompt dehazing transformers with depth-consisteney. arXiv preprint 2rX31v3303.0703, 2023.
[87] C. Wang. J. Pan, and X. Wu. Online-updated high-order collaborative networks for single image deraining. In AAAI, pages 2406-2413, 2022.
[88] C. Wang, Y. Wu, Z. Su, and J. Chen. Joint self-attention and aggreegggnnt for self-calibrated deraining network. In ACM MM, pages 2517-2525, 2020.
[89] C. Wang. X. Xing. Wu, Z. Su, and J. Chen. DCSEN: deep cross-scale fusion network for single image rain removal. In ACM MM, pages 1643-1651, ACM, 2020.
[90] C. Wang, X. Xing. Y. Wu, 7. Su, and J. Chen. DCSFN: deep cross-scale fusion network for single image rain removal. In ACM MM, pages 1643-1651, 2020.
[91] C. Wang, H. Zhu, W. Fan, X. Wu, and J. Chen. Single image rain removal using recurrent scale-guide networks. Neurocomputing, 467:242-255, 2022.
[92] X. Wang, K. Yu, C. Dong, and C. C. Loy. Recovering realistic texture in image super-resolution by deep spatial feature transform. CVPR, 2018.
[93] Wang. X. Cun, J. Bao, and J. Liu. Uformer: A general u-shaped transformer for image restoration. MXiv:2006.03166 2021.
[94] Z. Wang. X. Cun, J. Bao, W. Zhou, J. Liu, and H. Li. Uformer: A general u-shaped transformer for image restoration, In CVPR, pages 17683-17693, 2022.
[95] W. Wei, D. Meng, Q Zhao, Z. Xu, and Y. Wu. Semi-supervised transfer lcarning for image rain removal. In CVPR, 2019.
[96] F. Yang, H. Yang, J. Fu, H. Lu, and B. Guo. Learning texture transformer network for image super- resolution. In CVPR, 2020.
[97] W. Yang, R. Tan, Feng, Liu, Z. Guo, and S. Yan. Deep joint rain detection and removal from single image. In CVPR, 2017.
[98] Yao, A. Zhang, 7. Zhang, Liu, T. Chua, and M. Sun. CPT: colorful prompt tuning for pre-trained vision-language models. CoRR, ahs/2109.1 11797, 2021.
[99] R. Yasarla and V. M. Patel. Uncertainty guided multi-scale residual learning-using a cycle spinning cnn for single image de-raining. In CVPR, 2019.
[100] 7. Yue, Q. 7hao, I.. Zhang, and D. Meng. Dual adversarial network: Toward real-world noise removal and noise generation. In ECCV, 2020.
[101] S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, and M.-H. Yang. Restormer: Efficient transformer for high-resolution image restoration. In CVPR, pages 5718-5729, 2022.
[102] W. Zamir, A. Arora, Khan, M. Hayat,
Khan, M.-H. Yang, and L. Shao. Learning enriched features for real image restoration and enhancement. In ECCV, 2020.
[103] S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, M.-H. Yang, and L. Shao. Multi-stage progressive image restoration. In CVPR, 2021.
[104] Y. Zang, L. K. Zhou, C. Huang, and C. C, Loy. Unified vision and language prompt learning. CoRR, abs/2210.07225, 2022.
14

---

[105] H. Zhang, Y. Dai, H. Li, and P. Koniusz. Deep stacked hierarchical multi-patch network for image deblurring. In CVPR, 2019.
[106] H. Zhang and V. M. Patel, Density-aware single image de-raining using multi-stream dense network. In CVPR, 2018.
[107] H. Zhang, V. Sindagi, and V. M. Patel. Image de-raining using conditional generative adversarial network. TCSVT, 2019.
[108] K. Zhang. Li, W. Zuo, I. Zhang, L. Van Gool, and R. Timofte. Plug-and-play image restoration with deep denoiser prior. TPAMI. 2021.
[109] K. Zhang. W. Luo, Y. Zhong, L. Ma, B. Stenger, W. Liu, and H. Li. Deblurring by realistic blurring. In CVPR, 2020.
[110] Y. Zhang, Li, K. Li, L. Wang, B. Zhong, and Y. Fu. Image super-resolution using very deep residual channel attention networks. In ECCV, 2018.
[111] Y. Zhang, K. Li, K. Li, B. Zhong, and Y. Fu. Residual non-local attention networks for image restoration. In ICLR, 2019.
[112] Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu. Residual dense network for image restoration. TPAMI, 2020.
[113] Z. Zheng, W. Ren, X. Cao, X. Hu, T. Wang, F. Song, and X. Jia. Utra-high-defnitioon image dehazing via multi-guided bilateral learning. CVPR, pages 16185-16194, 2021.
[114] Z. Zheng, X. Yue, K. Wang, and Y. You. Prompt vision transformer for domain generalization. CoRR, abs/2208.065 08914, 2022.
[115] K. Zhou, J. Yang, C. C. Loy, and Z. Liu. Conditional prompt learning for vision-language models. In CVPR, pages 16795-16804 16804, 2022.
[116] S. Zhou, K. C. K. Chan, C. Li, and C. C. Loy. Towards robust blind face restoration with codebook lookup transformer. NeurlPS, 2022.
[117] H. Zhu, C. Wang, Y. Zhang, Z. Su, and G, Zhao. Physical model guided deep image deraining. In ICME, 2020.
15


Reviews:
[{"review_id": "msS87uybAS", "reviewer": "Reviewer_TXGv", "summary": "This paper proposes a Prompt image Restorer, which contains a restoration branch and a prompting branch. A pre-trained model is utilized to extract features in the prompting branch and a prompting modulator is proposed to better perceive features from global and local perspectives in the restoration branch. Extensive experimental results of multiple restoration tasks show the effectiveness of their method.", "questions": "Please refer to Weaknesses.", "limitations": "None", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The results in some experiments have shown improvement compared to previous methods.\n2. The paper is well-written and easy to understand.", "weaknesses": "1. The prompting way proposed in this paper is similar to the approach taken in some other papers, such as \"Take a Prior from Other Tasks for Severe Blur Removal\", where features are added as conditions. PromptRestorer doesn't have a significant difference, which makes me feel like it is more of a repackaging of prompting learning.\n2. As a model for image generation, why does the pre-trained model provide raw degradation information instead of image content information? What advantages does this approach have compared to some other methods of degradation representation? For example, \"Learning Degradation Representations for Image Deblurring\", and \"Learning Disentangled Feature Representation for Hybrid-distorted Image Restoration\".\n3. The comparisons with the latest methods in Table 2, such as Restormer and NAFNet, are missing.\n4. The article lacks comparisons with previous methods in terms of computational complexity and the number of parameters.\n5. The method in this paper employs a pre-trained  VQGAN, which introduces additional data. Is this experimental comparison fair? Did the authors account for the impact of this aspect?\n6. From Figure 8, it can be observed that your method visually resembles the features of the ground truth (GT). However, GT does not contain any degradation information, which contradicts the claims of raw degradation information made in your paper. This makes the motivation and claims of this paper unconvincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q: The prompting way proposed in this paper is similar to the approach taken in some other papers, such as \"Take a Prior from Other Tasks for Severe Blur Removal\", where features are added as conditions. PromptRestorer doesn't have a significant difference.\n\nA: The abovementioned method uses a semantic/classification prior to guide the restoration branch. We note that the condition networks in this method are learnable and optimized by the features of GT. This manner is similar to the model of Case 2 in our paper. However, our method is different from the abovementioned method, where we use a pre-trained model to extract the raw degraded features and the pre-trained model keeps freezing when optimizing the restoration branch. Such a design provides the restoration branch with original degradation features instead of optimized features. Our method is more direct, simpler, and more effective than the ones that usually use a learnable branch to guide the restoration network. \n\nMoreover, different from existing methods, we also show that using the pre-trained model to extract raw degraded features to guide the restoration network overcomes the problem of ''degradation vanishing'' (as shown in Fig. 8), which provides the restoration networks with precise degraded information, enabling the restoration network to perceive the degradation priors for better restoration.\n\nTo the best of our knowledge, we are the first to use the raw degraded features to guide image restoration and show that raw degraded features are more effective than learnable conditions. \n\nQ: As a model for image generation, why does the pre-trained model provide raw degradation information instead of image content information?\n\nA: Since the pre-trained VQGAN can reconstruct the input images as illustrated in VQFR [1], the pre-trained VQGAN model represents the features of the input images in feature space. While the input is the degraded image, the pre-trained VQGAN model generates degradation features. Hence, naming the raw degradation information is more precise than image content information.\n\n[1] Gu et al. VQFR: Blind Face Restoration with Vector-Quantized Dictionary and Parallel Decoder. ECCV2022.\n\nQ: What advantages does this approach have compared to some other methods of degradation representation? e.g., \"Learning Degradation Representations for Image Deblurring\" (Ref-A), and \"Learning Disentangled Feature Representation for Hybrid-distorted Image Restoration\" (Ref-B).\n\nA: We note that the method in Ref-A first jointly trains the conditional encoder that extracts the degraded information with the deblurring networks within 200K iterations in the first training phase, then freezes the conditional encoder and trains the deblurring networks in the second training phase.\n\nThe first training phase will make the encoder cannot effectively preserve the degraded information as the encoder is optimized toward sharper features. In contrast, our pre-trained model which extracts the degraded information keeps freezing when training the restoration networks and thus can more precisely generate the degraded information. Hence, our method is a pure method that provides degraded information for restoration networks.\n\nThe method in Ref-B uses the learnable disentangled features in the restoration networks instead of using degraded information to guide the deep networks. \n\nHence, our method is different from the above methods.\n\nQ: Comparisons with Restormer and NAF.\n\nA: We re-implement our method and compare it with the Restormer and NAF and the results are summarised in the following Table. Compared with Restormer, our method consumes fewer parameters while improving the PSNR by 0.14dB and producing better generalization on the HIDE dataset. On RealBlur benchmarks, our PromptRestorer is competitive. Compared with the NAF, our method only consumes about 1/3 parameters of NAF but achieves consistent improvement on HIDE, RealBlur-R, and RealBlur-J. These results demonstrate our PromptRestorer is competitive.\n\n|  Datasets  |  Restormer  |     NAF     |     Ours    |\n|:----------:|:-----------:|:-----------:|:-----------:|\n|    GoPro   | 32.92/0.961 | 33.71/0.967 | 33.06/0.962 |\n|    HIDE    | 31.22/0.942 | 31.31/0.943 | 31.36/0.944 |\n| RealBlur-R | 36.19/0.957 | 35.97/0.951 | 36.06/0.954 |\n| RealBlur-J | 28.96/0.879 | 28.31/0.856 | 28.82/0.873 |\n|  Para (M)  |     26.1    |     67.9    |     24.4    |\n|  FLOPs (G) |    140.99   |    63.33    |    186.3    |\n\nWe set the number of channels as 32, 64, and 128, the number of CGT is [2, 3, 5] from level-1 to level-3, and the expanding channel capacity factor is 3. The iterations are 600K and the batch size is 15.\n\nQ: The method in this paper employs a pre-trained VQGAN, which introduces additional data. Is this comparison fair? Did the authors account for the impact?\n\nA: The comparison is fair. We compare our method with the model with learnable conditional modulation, where the learnable conditional modulation has the same network architecture as the pre-trained VQGAN encoder used in our method for fair comparisons (footnote on Page 8). Tab. 5 shows that our method performs better.\n\nQ: From Fig. 8, it can be observed that your method visually resembles the features of the GT. However, GT does not contain any degradation information, which contradicts the claims of raw degradation information made in your paper.\n\nA: As GT features and the raw degraded features are extracted from the same pre-trained VQGAN, their colors are similar but the GT features are sharper and the raw degraded features are blurry. As other features are shown from their individual learning networks, both of them have different patterns (colors). By viewing on a high-resolution display, the degraded features are more blurry while other features become sharper. The GT feature put in this paper only plays a reference. Hence, our conclusion is consistent with our motivation."}, {"review_id": "JT3D04HCNV", "reviewer": "Reviewer_NMEt", "summary": "The paper proposes PromptRestorer for image restoration, addressing the issue of degradation vanishing in existing methods. Unlike previous approaches that do not explicitly consider degradation information or fail to effectively model degradation priors, PromptRestorer leverages the raw degraded features extracted from pre-trained models to generate reliable prompting content. The contributions of the work include PromptRestorer, the first approach to leverage prompting learning for general image restoration by considering raw degradation features, overcoming degradation vanishing while consistently retaining degradation priors. Additionally, the proposed prompting degradation perception modulator and gated degradation perception propagation enhance the restoration process by providing more reliable perceived content and controlling feature propagation. Extensive experiments show the superiority of the proposed method.", "questions": "See weaknesses section.", "limitations": "Limitations and broader societal impacts are not discussed in this paper.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper introduces a new approach called PromptRestorer for image restoration, which addresses the issue of degradation vanishing. The integral designs take advantage of prompting learning and explicitly consider degradation information, making it a unique and innovative contribution to the field.\n\n- The proposed method effectively perceives and retains degradation priors by leveraging raw degraded features and designing a prompting degradation perception modulator, resulting in improved performance compared to existing methods.\n\n- This comprehensive framework ensures that all aspects of restoration, including generating reliable prompting content and controlling feature propagation.\n\n- The paper provides a detailed analysis of the proposed method and the experimental results demonstrate the effectiveness of the PromptRestorer and achieve better restoration outcomes.\n", "weaknesses": "- The main problem **degradation vanishing** is not well defined and discussed in this paper, leading to the confusing and distracting presentation that what problem the authors are trying to solve. In Section 4.3 the authors claim sharper details in features fail to provide sufficient degraded information. Why this is the case? Is there any evidence that can support sharper features of the conditional branch would deteriorate the restoration performance? Overall the analysis of degradation vanishing is insufficient.\n\n- The core idea of this paper seems like another attempt to utilize prior information from pre-trained large models (*e.g.*, StyleGAN, DDPM, CLIP) to facilitate restoration, which has long been explored [1,2,3,4] in the image restoration community, though the designs and technical details are rather different. Could the authors further elaborate on the main differences between previous works?\n\n- As claimed that PromptRestorer can consistently retain the degradation priors (L62), the reviewer would expect a better adaptation of this model to various degradations. Hence, it would be better if the authors can show how well this algorithm can perform on different degradations (*e.g.*, various sizes of blur kernels, various levels of noise).\n\n[1] Chan, Kelvin CK, et al. \"Glean: Generative latent bank for large-factor image super-resolution.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.\n\n[2] Lugmayr, Andreas, et al. \"Repaint: Inpainting using denoising diffusion probabilistic models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[3] Kim, Geonung, et al. \"Bigcolor: colorization using a generative color prior for natural images.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.\n\n[4] Chen, Chaofeng, et al. \"Real-world blind super-resolution via feature matching with implicit high-resolution priors.\" Proceedings of the 30th ACM International Conference on Multimedia. 2022.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q: The main problem ''degradation vanishing'' is not well defined and discussed in this paper, leading to a confusing and distracting presentation.\n\nA: ''Degradation vanishing'' means the degradation features gradually vanish to become sharper features in the learning process of restoration networks. It is a straightforward phenomenon in most restoration networks as the model parameters are optimized toward producing clearer content, as shown in Case 1-2 of Fig. 8.\n\nWe discuss that using previously widely-used learnable conditional modulation to guide restoration models is worse than our solution which directly provides raw degradation information into deep restoration models (see Fig. 1, Tab. 5, and Case 2-3 in Fig. 8).\n\nHence, we introduce and show the degradation vanishing by visualization in Fig. 8, solve it by providing raw degradation features to restoration networks, discuss and compare it with previous related works in Tab. 5, and show that the proposed solution is simpler and more effective.\n\nQ: In Sec. 4.3 the authors claim sharper details in features fail to provide sufficient degraded information. Why this is the case? Is there any evidence that can support sharper features of the conditional branch would deteriorate the restoration performance?\n\nA: Sharper details contain less degraded content as the model parameters are optimized in the learning process of restoration networks, while degraded features can provide the restoration networks with accurate degraded priors so that the restoration networks can perceive degraded content to guide the restoration process for better recovery.\n\nThe visualization of Case 2 in Fig. 8 reveals that the learnable conditional branch learns sharper features, which thus cannot effectively provide restoration networks with effective degradation priors. Tab. 5 shows that the learnable conditional manner deteriorates the restoration performance, where our method produces 0.369dB gains.\n\nMoreover, Fig. 1(b) shows that the Case 2 has better performance in early iterations (about 25K iterations) but deteriorates the restoration performance after latter iterations compared with our method since Case 2 exhibits sharper results in later iterations while our method can constantly provide the restoration branch with raw degradation information, which would help better recovery. \n\nOur visualization and quantitative results support that using a learnable conditional branch to produce sharper features is not more effective than our method that using raw degradation to guide restoration networks. \n\nQ: The core idea seems like another attempt to utilize prior information from pre-trained large models to facilitate restoration, which has long been explored [1,2,3,4]. Could the authors further elaborate on the main differences between previous works?\n\nA: GLEAN utilizes pre-trained GANs to generate a latent bank, which produces the ''generative priors'' to embed into the encoder-decoder framework for image SR. Repaint employs a pre-trained unconditional DDPM as the ''generative prior'' to condition the generation process by sampling from the given pixels during the reverse diffusion iterations. BigColor also adopts the ''generative color prior'' to guide the image colorization. FeMaSR utilizes the VQGAN to generate the codebook of clean images to supervise the networks.\n\nDifferent from all previous methods that utilize the generative priors or codebook of clean images, our method directly utilizes the ''degradation information'' of the degraded image itself to guide restoration.\n\nWe note that there is NOT any literature to explore the original degraded features to guide image restoration up to now. We give the validation that raw degradation content extracted from the degraded images itself is more effective than previous commonly-used methods that usually use a learnable conditional branch to guide the image restoration. We make it possible that NOT learnable condition is better than learnable condition methods. \n\nQ: As claimed that PromptRestorer can consistently retain the degradation priors (L62), the reviewer would expect a better adaptation of this model to various degradations. Hence, it would be better if the authors can show how well this algorithm can perform on different degradations (e.g., various sizes of blur kernels, various levels of noise).\n\nA: First, blur kernels or various levels of noise are not available in practical images. Second, estimating blur kernels or various levels of noise may be not an easy task, which will introduce additional estimating errors that may subsequently affect the restoration quality.\n\nOur PromptRestorer does not require any additional estimation modules to estimate the gap between the degraded images and clean ones. In contrast, we directly utilize the degradation features extracted from the degraded image itself by a pre-trained model to provide the restoration networks with more precise degraded information. Hence, our PromptRestorer is simpler, more general, and more suitable for general image restoration tasks."}, {"review_id": "vd4Z5ib7Zk", "reviewer": "Reviewer_24Ac", "summary": "This paper proposes a prompting image restoration method with degradation perception. The authors show that raw degradation features can effectively guide deep restoration models, providing accurate degradation priors to facilitate better restoration. To perceive the degradation, the authors propose prompting degradation perception modulator. To control the propagation of the perceived content for restoration branch, the authors propose gated degradation perception propagation. \n\n", "questions": "see weakness", "limitations": "see weakness", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "+ This paper is professionally wrote and presents a better organization.\n+ The proposed prompting image restoration method is interesting, which clearly demonstrate that the raw degradation features are better guidance than the learnable parameters used in previous methods.\n+ The proposed prompting degradation perception modulator is novel. It has potential applications to latter features fusion manner.\n+ The gated degradation perception propagation seems to be useful.\n", "weaknesses": "1, The author claim that the pre-trained model is the same with the learnable conditional network. Hence, the FLOPs in Tab. 5 should be same.\n\n2, The authors would better to explain why the raw degradation features are better than features from learnable parameters network since more learnable parameters should have better results within the similar network framework.\n3, SFT is a commonly used feature fusion module, is the proposed PromptDPM than that? What is the result if replacing the PromptDPM with SFT?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q: The author claims that the pre-trained model is the same as the learnable conditional network. Hence, the FLOPs in Tab. 5 should be the same.\n\nA: Thanks for your reminder. The reported FLOPs of Case 3 in Tab. 5 are the restoration network. We will correct it in the revised paper.\n\nQ: The authors would better explain why the raw degradation features are better than features from the learnable parameters network since more learnable parameters should have better results within a similar network framework.\n\nA: The learnable nature of the conditional network in these models does not effectively provide degradation information for the restoration network. As parameters are optimized in the learning process, features become gradually clear, which cannot effectively guide the restoration networks.\n\nIn contrast, we directly exploit the raw degraded features extracted by a pre-trained model from the degraded inputs to generate more reliable prompting content to guide image restoration. Raw degraded features preserve accurately degraded information, which can consistently prompt the restoration network with accurate degraded priors, enabling the restoration network to perceive the degradation for better restoration. Tab. 5 clearly shows using the raw degraded features to guide restoration produces 0.369dB PSNR gains than the model with a learnable conditional branch.\n\nQ: SFT is a commonly used feature fusion module, is the proposed PromptDPM better than that? What is the result if we replace the PromptDPM with SFT?\n\nA: We compare the SFT with our PromptDPM by replacing the PromptDPM with SFT in our model. The PSNR results are summarised as follows:\n\nSFT: 30.699; Ours: 31.015.\n\nOur PromptDPM improves by 0.316dB PSNR compared with the commonly used SFT module, which further demonstrates the effectiveness of the proposed PromptDPM."}, {"review_id": "aqKjBMxUSX", "reviewer": "Reviewer_T7gZ", "summary": "This paper proposes the PromptRestorer by taking advantage of the prompting learning for image restoration. The author considers raw degradation features into restoration which consistently retains the degradation priors to facilitate better restoration. To better perceive degradation, the authors propose Global Prompting Perceptor and Local Prompting Perceptor. \nTo control the propagation of the perceived features, the authors propose gated degradation perception propagation, enabling the model to adaptively learn more useful features for better image restoration.\n", "questions": "Please refer to the questions in Weaknesses", "limitations": "The authors do not present the limitations in the paper, which are suggested to add them.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1, Good writing skills. The paper shows excellent presentation and clearly presents former disadvantages and better solves them by introducing the prompting strategy.\n2, Clear motivation. To solve the problem of degradation vanishing in image restoration (Fig. 1 and 8), the authors introduce a Prompting Degradation Perception by extracting the raw degradation features to directly guide the image restoration process. The experiments (Tab. 5) show that the proposed learning strategy is better than the former methods.\n3, Interesting module. The proposed Prompting Degradation Perception Modulator is interesting and Gated Degradation Perception Propagation is simple.\n", "weaknesses": "1, The authors can give a clearer understanding of the degradation priors guidance. Why does it work better than the model with a learnable conditional branch?\n2, Although the Prompting Degradation Perception Modulator including the Global Prompting Perceptor and Local Prompting Perceptor is interesting, it would be better if the authors can give a more direct explanation of them as the two modules seem to look the key contributions of this work.\n3, The paper exists some typos. For example, in line 89, knew->known.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q: The authors can give a clearer understanding of the degradation priors guidance. Why does it work better than the model with a learnable conditional branch?\n\nA: The learnable nature of the conditional network in these models does not effectively provide degradation information for the restoration network since the parameters in the learnable conditional branch are optimized in the learning process so that the features become gradually clear, which cannot effectively provide the restoration networks with degraded priors well.\n\nIn contrast, we directly exploit the raw degraded features extracted by a pre-trained model from the degraded inputs to generate more reliable prompting content to guide image restoration. Raw degraded features preserve accurately degraded information, which can consistently prompt the restoration network with accurate degraded priors, enabling the restoration network to perceive the degradation for better restoration.\n\nQ: Although the Prompting Degradation Perception Modulator including the Global Prompting Perceptor (G2P) and Local Prompting Perceptor (L2P) is interesting, it would be better if the authors can give a more direct explanation of them as the two modules seem to look the key contributions of this work.\n\nA: The G2P fully exploits the self-attention mechanism to form the global prompting attention induced by the degraded features. The G2P contains the global perception attention followed by an improved ConvNeXt. Our global perception attention consists of 1) Query-Induced Attention (Q-InAtt) and 2) Key-Value-Induced Attention (KV-InAtt). The Q-InAtt considers re-forming the query vector induced by degradation features to build a representative query to perform attention, while the KV-InAtt re-considers key and value vectors induced by other degradation counterparts to search for more similar content with the restoration query.\n\n\nThe L2P adequately considers the pixel-level degradation perception, enabling it to better perceive degradation from spatially neighboring pixel positions. The L2P consists of a local perception modulator followed by a separable depth-level convolution. The local perception modulator contains two core components: 1) Degradation-Induced Band (Deg-InBan) and 2) Restoration-Induced Band (Res-InBan). The former is achieved by exploiting the degradation features to induce spatially useful content from restoration content to guide restoration gating fusion, while the latter utilizes the deep restoration features to induce more useful features from another degradation counterpart to form the degradation gating.\n\nQ: The paper exists some typos. For example, in line 89.\n\nA: Thanks for your careful reading. We will fix them in the revised paper."}, {"review_id": "PSDnCTXf1g", "reviewer": "Reviewer_Hjzg", "summary": "In this paper, the authors find that the degradation vanishes in the learning process of the image restoration problem. Which further hinders the model's capacity. To solve this problem, this paper proposes the PromptRestorer, a prompting image restoration method by exploring the raw degradation features extracted by a pre-trained model from the given degraded observations to guide the restoration process. The authors demonstrate that the proposed guidance manner is better than both the encoder-decoder model without any guidance and the model with learnable guidance. The experiments on image deraining, deblurring, dehazing, and desnowing demonstrate the superiority of the proposed algorithm.", "questions": "See Weaknesses", "limitations": "The authors are suggested to discuss the limitations of this paper.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1, The method is simple and effective and is easy to follow. The writing is professional and clear to understand.\n2, The proposed Prompting Degradation Perception Modulator including global prompting perceptor and local prompting perceptor is novel and effective.\n3, The proposed Gated Degradation Perception Propagation is simple and meaningful.\n4, This paper clarifies a new perspective that the raw degradation features are better than the learnable features which are usually utilized in former methods.\n5, The extensive experiments on various image restoration tasks demonstrate the generalization of the proposed algorithm.", "weaknesses": "1, Although this paper clarifies many strengths of the method, what’s the limitation? The author should present it in this paper.\n2, The authors should give more detailed explanation of the proposed global prompting perceptor and local prompting perceptor to facilitate better understanding.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q: Although this paper clarifies many strengths of the method, what’s the limitation? The author should present it in this paper. \n\nA: Since our method needs to utilize a pre-trained model to extract degraded priors, it consumes more parameters and FLOPs compared with the model without any conditional modulation (i.e., Case 1), as illustrated in Tab. 5. However, we demonstrate that using raw features to guide restoration produces better performance than previous widely-used learnable conditional modulation (Case 2) while consuming fewer learnable parameters.\n\n\nQ: The authors should give a more detailed explanation of the proposed global prompting perceptor and local prompting perceptor to facilitate better understanding.\n\nA: The G2P fully exploits the self-attention mechanism to form the global prompting attention induced by the degraded features. The G2P contains the global perception attention followed by an improved ConvNeXt. Our global perception attention consists of 1) Query-Induced Attention (Q-InAtt) and 2) Key-Value-Induced Attention (KV-InAtt). The Q-InAtt considers re-forming the query vector induced by degradation features to build a representative query to perform attention, while the KV-InAtt re-considers key and value vectors induced by other degradation counterparts to search for more similar content with the restoration query.\n\nThe L2P adequately considers the pixel-level degradation perception, enabling it to better perceive degradation from spatially neighboring pixel positions. The L2P consists of a local perception modulator followed by a separable depth-level convolution. The local perception modulator contains two core components: 1) Degradation-Induced Band (Deg-InBan) and 2) Restoration-Induced Band (Res-InBan). The former is achieved by exploiting the degradation features to induce spatially useful content from restoration content to guide restoration gating fusion, while the latter utilizes the deep restoration features to induce more useful features from another degradation counterpart to form the degradation gating."}, {"review_id": "NyW0nY0n7E", "reviewer": "Reviewer_zfG4", "summary": "This paper introduces the PromptRestorer, a method for image restoration that effectively utilizes raw degradation features to guide the restoration process. PromptRestorer consists of two branches: a restoration branch and a prompting branch. The restoration branch restores the images, while the prompting branch perceives degradation priors and provides reliable content to guide the restoration process. Experimental results demonstrate that PromptRestorer achieves state-of-the-art performance on four image restoration tasks: deraining, deblurring, dehazing, and desnowing.\n", "questions": "- What is a more accurate degradation priors? Can we measure it? \n- What are the motivations behind the introduction of key modules and the issues they address in previous methods? \n- It appears that the network parameter count and FLOPS of this method may increase significantly compared to those of Case 1 and Case 2 methods. Is there more detailed comparative data available in this regard?\n", "limitations": "- The performance improvement of this method appears to be relatively modest compared to some previous approaches, and it remains unclear what the magnitude of the associated numerical variances is.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The idea of utilizing intermediate layer features of pre-trained models to enhance the effectiveness of image restoration is innovative.\n- The experimental validation tasks and comparison methods employed in this study are comprehensive and diverse.\n- The comparative figure (Figure 1) between the proposed method and previous approaches are presented in a clear manner, highlighting the distinctive aspects of the framework.\n", "weaknesses": "- The motivation behind the introduction of the prompting degradation perception modulator is not clearly articulated in the paper. The authors fail to thoroughly analyze the existing deficiencies of previous methods and instead claim that the incorporation of prompting can lead to more accurate degradation priors or more reliable perceived content learned from the degradation priors. However, these descriptions lack quantifiable metrics and robust support from relevant literature. Despite the inclusion of visualizations depicting feature maps in the experimental section, there remains an ambiguity regarding the definition of better degradation priors.\n- Similarly, I am confused regarding the motivation behind the introduction of local prompting attention and global prompting attention. It remains unclear to what extent these components address existing shortcomings in the framework. Furthermore, the results of the ablation experiments indicate relatively minor improvements resulting from the incorporation of these components.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q: What is a more accurate degradation prior? Can we measure it? \n\nA: The most accurate degradation priors should be the degraded image itself. In feature space, better degradation should have the property that can better represent the features of the degraded image. We note that VQGAN can reconstruct the input images and thus can accurately represent the features of the input images [1]. Hence, we use the pre-trained VQGAN to accurately extract degradation features as the degradation priors. Fig. 8 (GT features and Case 3) clearly shows that pre-trained VQGAN can better preserve the features of the input images.\n\n[1] Gu et al. VQFR: Blind Face Restoration with Vector-Quantized Dictionary and Parallel Decoder. ECCV2022.\n\nQ: What are the motivations behind the introduction of key modules and the issues they address in previous methods? I am confused regarding the motivation of local prompting attention and global prompting attention.\n\nA: Our method is motivated by that existing image restoration models usually appear to degradation vanishing, as shown in Case 1-2 of Fig. 8, as the model parameters are optimized toward producing clean content. However, the degradation content is critical for image restoration, which implies important cues for better image restoration. To solve this problem, we propose to use raw degradation features to overcome degradation vanishing so that helps the networks constantly perceive the degradation priors to facilitate better recovery. To better perceive the degradation, we propose G2P and L2P to respectively perceive the degradation from global and local perspectives, enabling to generate more useful content to guide the restoration branch. To control the propagation of the perceived content, we propose the GDP, enabling the restoration network to adaptively learn more useful features for better restoration.\n\nOur global perception attention consists of Q-InAtt and KV-InAtt. The Q-InAtt considers re-forming the query vector induced by degradation features to build a representative query to perform attention, while the KV-InAtt re-considers key and value vectors induced by other degradation counterparts to search for more similar content with the restoration query. The local perception modulator contains Deg-InBan and Res-InBan. The former is achieved by exploiting the degradation features to induce spatially useful content from restoration content to guide restoration gating fusion, while the latter utilizes the deep restoration features to induce more useful features from another degradation counterpart to form the degradation gating.\n\nPrevious commonly-used fusion methods SFT usually only consider spatial feature transformation with a single direction transformation, i.e., from conditions to restoration fusion at pixel-level, while neglecting the global perspectives. We consider both local and global transformation by re-forming attention and local modulation with bidirectional interaction, i.e., from restoration to degradation, and from degradation to restoration, to guide restoration.\n\nQ: It appears that the network parameters count and FLOPs of this method may increase significantly compared to those of Case 1 and Case 2 methods. Is there more detailed comparative data available in this regard?\n\nA: The learnable parameters of our method are fewer than those of the model with learnable conditional modulation (Case 2), as shown in Tab. 5. This is because we use a pre-trained model to extract raw degradation information which keeps frozen when training the restoration models. However, our method still performs better than the model with learnable conditional modulation.\n\nFor Case 1, since it does not have the conditional branch, our model (Case 3) and model with learnable conditional modulation (Case 2) naturally contain more parameters and FLOPs.\n\nQ: The improvement of this method appears to be relatively modest compared to some previous approaches.\n\nA: Our method consistently achieves better performance on $4$ image restoration tasks on various benchmarks, which demonstrates the effectiveness of our method.\n\nQ: The ablation indicates relatively minor improvements resulting from the incorporation of these components.\n\nA: Our experiments demonstrate that each component is effective for restoration. Note that our method with fewer parameters performs better than the model with learnable conditional modulation, which validates our key contribution that using raw degradation content is more effective than learnable conditions."}]

Author rebuttal:

Meta-review: This paper points out the problem of degradation vanishing in image recovery and puts forward the idea that accurate original degradation features can guide the deep recovery model. Based on this, this paper designs a PromptRestorer network containing recovery branches and prompting branches, which cues the recovery branches for better-guiding image restoration through reliable perceptual content prompting with the designed PromptDMP. Almost all reviewers recognized the technical novelty as well as the contribution of the method. The authors' response successfully addressed most of the reviewers' concerns and provided additional (supportive) experimental results. Thus, the AC recommends accepting this paper. However, some reviewers still expressed partial concerns about the experiments in this paper. I suggest that the authors should further add to the experiments and organize the open source in the final version.
Decision: Accept (poster)