Title: PromptRestorer: A Prompting Image Restoration Method with Degradation Perception
Venue: NeurIPS 2023
Accepted: yes
Keywords: Degradation Vanishing

Prompting Learning

Image Restoration
Abstract: We show that raw degradation features can effectively guide deep restoration models, providing accurate degradation priors to facilitate better restoration. While networks that do not consider them for restoration forget gradually degradation during the learning process, model capacity is severely hindered. To address this, we propose a Prompting image Restorer, termed as PromptRestorer. Specifically, PromptRestorer contains two branches: a restoration branch and a prompting branch. The former is used to restore images, while the latter perceives degradation priors to prompt the restoration branch with reliable perceived content to guide the restoration process for better recovery. To better perceive the degradation which is extracted by a pre-trained model from given degradation observations, we propose a prompting degradation perception modulator, which adequately considers the characters of the self-attention mechanism and pixel-wise modulation, to better perceive the degradation priors from global and local perspectives. To control the propagation of the perceived content for the restoration branch, we propose gated degradation perception propagation, enabling the restoration branch to adaptively learn more useful features for better recovery. Extensive experimental results show that our PromptRestorer achieves state-of-the-art results on 4 image restoration tasks, including image deraining, deblurring, dehazing, and desnowing.

PromptRestorer: A Prompting Image Restoration Method with Degradation Perception

Abstract

We show that raw degradation features can effectively guide deep restoration models, providing accurate degradation priors to facilitate better restoration. While networks that do not consider them for restoration forget gradually degradation during the learning process, model capacity is severely hindered. To address this, we propose a Prompt ing image Restorer , termed as PromptRestorer . Specifically, PromptRestorer contains two branches: a restoration branch and a prompting branch. The former is used to restore images, while the latter perceives degradation priors to prompt the restoration branch with reliable perceived content to guide the restoration process for better recovery. To better perceive the degradation which is extracted by a pre-trained model from given degradation observations, we propose a prompting degradation perception modulator, which adequately considers the characters of the self-attention mechanism and pixel-wise modulation, to better perceive the degradation priors from global and local perspectives. To control the propagation of the perceived content for the restoration branch, we propose gated degradation perception propagation, enabling the restoration branch to adaptively learn more useful features for better recovery. Extensive experimental results show that our PromptRestorer achieves state-of-the-art results on 4 image restoration tasks, including image deraining, deblurring, dehazing, and desnowing.

1 Introduction

Image restoration aims to recover clear high-quality images from given degraded ones. It is highly ill-posed since only degraded images can be exploited, statistical observations are thus required to well-pose the problems [37, 67, 66, 41, 42]. Although conventional approaches can recover images to some extent, they typically involve solving optimization algorithms that are difficult due to the non-convexity and non-smooth problems. Additionally, the observations may not always hold, which can cause algorithms to fail.

With the emergence of convolutional neural networks (CNNs) [48] and Transformers [22, 45], which perform well at implicitly learning the priors from large-scale data, learning-based methods have dominated recent image restoration tasks and achieved impressive performance [81, 51, 103, 72, 61, 93, 33, 13]. However, these methods are usually built without explicitly considering the specific degradation information, which accordingly limits model capacity ( Case 1 in Fig. 1). An alternative approach is to design a conditional branch to learn additional information to provide the restoration network with useful content for modulation [30, 17, 34, 35] ( Case 2 in Fig. 1). However, we note that while conditional branches in these models are learnable, they may not effectively provide degradation information, as the optimizable parameters result in gradually clearer features during the learning process, leading to the degradation vanishing which accordingly limits model performance.

CRndiWiRn

i.e., KeUnel/SemenWicV/InSXW

(D)

(b) LeaUning CXUYeV

(c) ViVXal PeUfRUmance

LeaUnable CRndiWiRnal

GXidance

GLYPH<c=29,font=/MLQVRC+TimesNewRomanPS-BoldMT>

RaZ DegUegaWiRn AV PURmSWing

Learning curves of different frameworks

Degraded

Input

Enc.

Dec.

31

Case1

Image

30

Case2

PSNR (dB)

29

Case3

Input

Input

Enc.

Dec.

PSNR

PSNR

28

Learmable

1

1

27

Case1

Net

1

1

1

26

Iterations(×10)

Case2

25.8

4823

29.

26.1

29.

Input

Enc.

Dec.

27.0

4823

26.8

Case3

(BP) aNSd

30.0

Pre-trained

Model

26.4

4823

NOTLearnable

26.2

GT

8

Recently, prompt learning has been shown an effective tool to improve model performance by designing various prompts [115, 98, 104, 23, 114, 46, 53, 43]. The prompt usually serves as the guidance tool to correct the networks toward better results [28]. However, prompt learning still keeps a margin for image restoration, and existing prompts may not be suitable for image restoration since they cannot effectively model degradation priors well. Hence, we ask: Is there a reasonable prompting manner to correct degraded image restoration networks to facilitate better recovery?

The answer is in the riddle . This paper proposes the PromptRestorer , a Prompt ing image Restorer , to overcome degradation vanishing in image restoration via promoting by exploring degradation input itself for better restoration ( Case 3 in Fig. 1). Our idea is simple: we directly exploit the raw degraded features extracted by a pre-trained model from the degraded inputs to generate more reliable prompting content to guide image restoration. Raw degraded features preserve accurately degraded information, which can consistently prompt the restoration network with accurate degraded priors, enabling the restoration network to perceive the degradation for better restoration. Hence, we design the PromptRestorer, which consists of two branches: (a) the restoration branch and (b) the prompting branch. The former is used to restore images and the latter is used to generate reliable prompting features to guide the restoration network for better restoration. To better perceive the degradation, we propose a Prompt ing D egradation P erception M odulator ( PromptDPM ), which consists of G lobal P rompting P erceptor ( G2P ) and L ocal P rompting P erceptor ( L2P ). The G2P adequately exploits the self-attention mechanism to form global prompting attention, while the L2P considers the pixel-level perception to build local prompting content. To control the propagation of perceived features in the restoration branch, we propose G ated D egradation Perception P ropagation ( GDP ), enabling the restoration network to adaptively learn more useful features to facilitate better restoration.

The main contributions of this work are summarized below:

We propose PromptRestorer, which is the first approach to our knowledge that takes advantage of the prompting learning for general image restoration by considering raw degradation features in restoration, enabling the restoration model to overcome degradation vanishing while consistently retaining the degradation priors to facilitate better restoration.

We propose a prompting degradation perception modulator that is used to perceive degradation from global and local perspectives, which is able to provide the restoration network

with more reliable perceived content learned from the degradation priors, enabling it to better guide the restoration process.

We propose gated degradation perception propagation that exploits a gating mechanism to control the propagation of the perceived features, enabling the model to adaptively learn more useful features for better image restoration.

Fig. 1 summarises framework comparisons, and their learning curves and visual performance. Deeper analysis and discussion about them are provided in Sec. 4.2.

2 Related Work

In this section, we review image restoration, conditional modulation, and prompt learning.

Image Restoration. Recently, CNN-based architectures [110, 112, 103, 4, 24, 102, 89, 91, 87, 19, 88, 117] and Transformer-based models [96, 56, 49, 12, 93, 93] have been shown to outperform conventional restoration approaches [37, 83, 64, 47, 7, 79]. These learning-based methods usually adopt U-Net architectures [50, 18, 103, 100, 1, 93, 108, 101], which have been demonstrated the effectiveness because of hierarchical multi-scale representation and effective learning between shallow and deeper layers by skip connection [111, 59, 102, 31]. We refer the readers to recent excellent literature reviews on image restoration [5, 54, 82], which summarise the main designs in deep image restoration models.

Although these models have achieved promising performance, they do not explicitly take degradation into consideration for model design which is vital for restoration, limiting the model capacity.

Conditional Modulation. Conditional modulation usually involves implicitly modulating the additional content to guide the restoration [30, 10, 16, 17, 36, 35, 34, 60, 57, 92]. These approaches usually contain two branches: a basic network and a conditional network. The conditional network provides additional information to guide the basic network for restoration via spatial feature transform (SFT) [92]. Among these methods, blur kernel [30], semantics [92], and degraded input [57, 16] which serve as the additional conditions are broadly known.

The learnable nature of the conditional network in these models does not effectively provide degradation information for the basic network. As parameters are optimized in the learning process, features become gradually clear.

Prompt Learning. Prompt learning methods have been studied broadly in natural language processing (NLP) [75, 78, 8]. Due to high effectiveness, prompt learning is recently used in vision-related tasks [115, 98, 104, 23, 114, 46, 53, 43, 71, 29, 40, 86, 28]. In vision prompt learning, many works seek useful prompts to correct task networks toward better performance [28].

Although prompt learning has shown promise in various vision tasks, it still keeps a margin in general image restoration. This paper proposes an effective prompting method, enabling the restoration model to overcome the degradation vanishing in the learning process for better restoration.

3 PromptRestorer

Our goal aims to overcome degradation vanishing and better perceive degradation in deep restoration models to improve image recovery quality. To achieve this, we introduce a prompting strategy that helps the model consistently memorize degradation information, enabling it to prompt restoration with better degradation for better restoration. To better perceive degradation, we propose the Prompt ing D egradation P erception M odulator ( PromptDPM ), which can provide more reliable perceived content to guide the restoration network. To control the propagation of the perceived content, we propose the G ated D egradation Perception P ropagation ( GDP ), enabling the restoration network to adaptively learn more useful features for better restoration.

3.1 Overall Pipeline

Fig. 2 shows the framework of our PromptRestorer, which contains two branches: (a) the restoration branch and (b) the prompting branch. The restoration branch is used to restore images, where each block is prompted by the prompting branch. The prompting branch first generates the accurate

SLgPRLd

1

h

1

ʦ

h

ˠ

h

Ĉ

ʦ

h

ˠ

h

ȕ

Ĉ

ʦ

h

ˠ

h

Ĉ

ʦ

h

ˠ

h

Ĉ

TFB

TFB

TFB

(a)Restoration

ContinuousGatedTransformer

Branch

Input (I)

Output(H)

Extraction

xL1

XL2

XL3

xL2

Reconstruction

自

自

Down

Down

CGT

×2↑

Up

CGT

×2↑

Up

CGT

GDP

Prompt

Prompt

Prompt

Prompt

Prompt

X

DPM

DPM

DPM

DPM

DPM

Addition

Y1

Element-wise

Multiplication

RawDegradation

Improved ConvNeXt

Features

LayerNorm

Attention

Multi-Head

Φ

LayerNorm

DConv

1x1Conv

GELU

DConv

1x1Conv

X

Concat

1x1Conv

L2P

Pre-trained

Y

Model

(b)PromptingBranch

(c)TransformerBlock(TFB)

(d)PromptingDegradationPerception

Modulator (PromptDPM)

X

degradation feature extracted by a pre-trained model, and then the feature is to prompt the restoration branch, enabling the restoration branch to better perceive the degradation prior for better recovery.

Restoration Branch. Given a degraded input image I ∈ R H × W × 3 , we first applies a 3 × 3 convolution as the feature extraction to obtain low-level embeddings X 0 ∈ R H × W × C ; where H × W denotes the spatial dimension and C is the number of channels. Next, the shallow features X 0 gradually are hierarchically encoded into deep features X l ∈ R H l × W l × lC . After encoding the degraded input into low-resolution latent features X 3 ∈ R H 3 × W 3 × 3 C , the decoder progressively recovers the highresolution representations. Finally, a reconstruction layer which contains 4 Transformer blocks as the refinement followed by a 3 × 3 convolution is applied to decoded features to generate residual image S ∈ R H × W × 3 to which degraded image is added to obtain the restored output image: ˆ H = I + S . Both encoder and decoder at l - level consist of multiple C ontinuous G ated T ransformers ( CGT ) with expanding channel capacity. To help better recovery, the encoder features are concatenated with the decoder features via skip connections [74] by 1 × 1 convolutions.

Prompting Branch. The prompting branch, as shown in Fig. 2(b), aims to generate and perceive degradation features and then provide useful guidance content for the restoration branch. We note VQGAN [25] has been demonstrated that it can generate high-quality images while representing the features of input images. However, it tends to damage image structure after vector quantization [116, 11, 32]. To avoid this problem, we only exploit the encoder of pre-trained VQGAN to represent the deep features of the degraded inputs. We first use the pretrained encoder to extract degraded features Y l ∈ R H l × W l × lC ; where l denotes the l - level layer in the pre-trained encoder. Then, the degraded features are exploited to generate reliable prompting content to prompt the restoration branch by PromptDPM (see Sec. 3.2). The generated prompting content is transmitted to each CGT to guide the restoration branch.

Continuous Gated Transformers. CGT exploits the perceived features from PromptDPM to provide the Transformer block with more reliable content to overcome degradation vanishing to facilitate better restoration. Each CGT consists of three Transformer blocks (Fig. 2(c)) with residual connections [38] and the input in each block is gated by GDP (see Sec. 3.3) to control the propagation of perceived features. Let P , G , and T respectively denote the operations of PromptDPM (expressed in (2)), GDP (expressed in (7)), and Transformer, the features flow in k th block in one CGT at l - level encoder/decoder, which can be expressed as:

GlRbal PeUceSWiRQ

AWWeQWiRQ

ImSURYed

CRQYNeXW

DCRQY

1

h

1

CRQcaW







1

h

1

CRQcaW

1

h

1









1

h

1

CRQcaW

(D) G2P

LRcal PeUceSWiRQ

MRdXlaWRU

DCRQY

1

h

1 CRQY

GELU

1

h

1

CRQcaW

SRfWma[

DCRQY

1

h

1 CRQY

DCRQY

1

h

1 CRQY

CRQcaW

1

h

1 CRQY

DCRQY

1

h

1 CRQY

ReVWRUaWiRQ-IQdXced BaQd

DegUadaWiRQ-IQdXced BaQd

Q-IQdXced AWWeQWiRQ

KV-IQdXced AWWeQWiRQ

(E) L2P

1

h

1

CRQcaW

DCRQc

GELU

SigmRid

SRfWma[



MaWUi[ MXlWiSlicaWiRQ



ReVhaSe

1

h

1

CRQcaW

DCRQc

GELU

SigmRid

where X k means the output of k th Transformer block in one CGT, especially X 0 is the downsampled/upsampled features at ( l -1) - level encoder/decoder; P l refers to the generated features of PromptDPM at l -level layer; G k -1 means the gated features between X k -1 and P l , which serves as the input of k th Transformer block. Each Transformer block consists of multi-head attention [101] followed by an improved ConvNeXt [62] as the feed-forward network (see Fig. 2(c)).

3.2 Prompting Degradation Perception Modulator

To better perceive the degradation to prompt the restoration network with more reliable perceived content from the degradation priors, we propose the PromptDPM (see Fig. 2(d)). The PromptDPM consists of 1) G lobal P rompting P erceptor ( G2P , introduced in Sec. 3.2.1) and 2) L ocal P rompting P erceptor ( L2P , introduced in Sec. 3.2.2) to respectively perceive the degradation from global and local perspectives, enabling to generate more useful content to guide the restoration branch. From a restoration tensor X ∈ R ˆ H × ˆ W × ˆ C and a degradation tensor Y ∈ R ˆ H × ˆ W × ˆ C , we prompt X with Y :

where Ψ global ( · , · ) and Ψ local ( · , · ) respectively denote the operations of G2P and L2P; C [ · , · ] means the concatenation at channel dimension; W p ( · ) refers to the 1 × 1 point-wise convolution.

3.2.1 Global Prompting Perceptor

The G2P, shown in Fig. 3(a), fully exploits the self-attention mechanism to form the global prompting attention induced by the degraded features. The G2P contains the global perception attention followed by an improved ConvNeXt [62]. Our global perception attention consists of 1) Q ueryIn duced Att ention ( Q-InAtt ) and 2) K eyV alueIn duced Att ention ( KV-InAtt ). The Q-InAtt considers re-forming the query vector induced by degradation features to build a representative query to perform attention, while the KV-InAtt re-considers key and value vectors induced by other degradation counterparts to search for more similar content with the restoration query. From a layer normalized restoration tensor X ∈ R ˆ H × ˆ W × ˆ C , our G2P first generates restoration query ( Q ), key ( K ), and value ( V ) projections from the restoration features. It is achieved by applying 1 × 1 convolutions to aggregate pixel-wise cross-channel context followed by 3 × 3 depth-wise convolutions W d ( · ) to encode channel-wise spatial context, yielding Q = W d W p X , K = W d W p X , and V = W d W p X . Meanwhile, we similarly convert the degradation tensor Y ∈ R ˆ H × ˆ W × ˆ C into degradation query ( ˜ Q ), key ( ˜ K ), and value ( ˜ V ) projections: ˜ Q = W d W p Y , ˜ K = W d W p Y , and ˜ V = W d W p Y . Then, we respectively conduct Q-InAtt and KV-InAtt:

where A ( · ) ( ˆ Q , ˆ K , ˆ V ) = ˆ V · Softmax ( ˆ K · ˆ Q /α ) ; Here, α is a learnable scaling parameter to control

the magnitude of the dot product of ˆ K and ˆ Q before applying the softmax function. Similar to the conventional multi-head SA [22], we divide the number of channels into 'heads' and learn separate attention maps. Then two induced attentions are fused and followed by an improved ConvNeXt:

where the W p W d ϕW p W d ( · ) means the improved ConvNeXt shown in the latter of Fig. 2(c); LN ( · ) means the operation of layer normalization [6].

3.2.2 Local Prompting Perceptor

The L2P, as shown in Fig. 3(b), adequately considers the pixel-level degradation perception, enabling to better perceive degradation from spatially neighboring pixel positions. The L2P consists of a local perception modulator followed by a separable depth-level convolution. The local perception modulator contains two core components: 1) Deg radationIn duced Ban d ( Deg-InBan ) and 2) Res torationIn duced Ban d ( Res-InBan ). The former is achieved by exploiting the degradation features to induce spatially useful content from restoration content to guide restoration gating fusion, while the latter utilizes the deep restoration features to induce more useful features from another degradation counterpart to form the degradation gating. Given the degradation tensor Y ∈ R ˆ H × ˆ W × ˆ C , we first exploit the point-wise convolution and 3 × 3 depth-wise convolution to encode two degradation projections, yielding ˜ Q = W Q d W Q p Y and ˜ K = W K d W p K Y . Meanwhile, the restoration tensor X ∈ R ˆ H × ˆ W × ˆ C are also encoded into two restoration projections: Q = W Q d W Q p X and K = W K d W p K X . Then, we respectively conduct Deg-InBan and Res-InBan:

where σ ( · ) denotes the sigmoid function that controls the gating level. Then, the perceived features in the two bands are fused via concatenation and 1 × 1 convolution and followed by a depth-level separable convolution W p ϕW d ( · ) :

3.3 Gated Degradation Perception Propagation

The GDP aims to control the propagation of the perceived degradation, enabling to adaptively learn more useful features in Transformer blocks to facilitate better restoration. Given the output restoration tensor X k -1 ∈ R ˆ H × ˆ W × ˆ C of ( k -1) th Transformer block in one CGT and the perceived tensor P l ∈ R ˆ H × ˆ W × ˆ C which is the output feature of one PromptDPM at l - level, the input of k th Transformer block can be obtained by gating the X k -1 with P l by 1 × 1 convolution and gated control function sigmoid σ ( · ) with residual learning [39]:

3.4 Learning Strategy

To train the network, two objective loss functions are adopted, including image reconstruction loss ( L i ) for pixel recovery and frequency loss ( L f ) for detail enhancement [18]:

where H denotes the ground truth image; F denotes the Fast Fourier transform; λ is a weight that is empirically set to be 0.1.

4 Experiment

We evaluate PromptRestorer on benchmarks for 4 image restoration tasks: (a) deraining, (b) deblurring, (c) desnowing, and (d) dehazing. We train separate models for different image restoration tasks. Our PromptRestorer employs a 3-level encoder-decoder. From level-1 to level-3, the number of CGT is [ 2 , 3 , 6 ], attention heads are [ 2 , 4 , 8 ], and number of channels is [48 , 96 , 192] . The expanding channel capacity factor β is 4. For downsampling and upsampling, we adopt pixel-unshuffle and pixel-shuffle [77], respectively. We train models with AdamW optimizer with the initial learning rate 3 e -4 gradually reduced to 1 e -6 with the cosine annealing [63]. The patch size is set as 256 × 256 .

Method

Test100

[107]

Rain100H

[97]

Rain100L

[97]

Test2800

[27]

Test1200

[106]

Average

PSNR

↑

SSIM

↑

PSNR

↑

SSIM

↑

PSNR

↑

SSIM

↑

PSNR

↑

SSIM

↑

PSNR

↑

SSIM

↑

PSNR

↑

SSIM

↑

DerainNet [26]

22.77

0.810

14.92

0.592

27.03

0.884

24.31

0.861

23.38

0.835

22.48

0.796

SEMI [95]

22.35

0.788

16.56

0.486

25.03

0.842

24.43

0.782

26.05

0.822

22.88

0.744

DIDMDN [106]

22.56

0.818

17.35

0.524

25.23

0.741

28.13

0.867

29.65

0.901

24.58

0.770

UMRL [99]

24.41

0.829

26.01

0.832

29.18

0.923

29.97

0.905

30.55

0.910

28.02

0.880

RESCAN [55]

25.00

0.835

26.36

0.786

29.80

0.881

31.29

0.904

30.51

0.882

28.59

0.857

PreNet [72]

24.81

0.851

26.77

0.858

32.44

0.950

31.75

0.916

31.36

0.911

29.42

0.897

MSPFN [44]

27.50

0.876

28.66

0.860

32.40

0.933

32.82

0.930

32.39

0.916

30.75

0.903

DCSFN [90]

27.46

0.887

28.98

0.887

34.70

0.961

30.96

0.903

32.92

0.937

31.00

0.915

MPRNet [103]

30.27

0.897

30.41

0.890

36.40

0.965

33.64

0.938

32.91

0.916

32.73

0.921

SPAIR [69]

30.35

0.909

30.95

0.892

36.93

0.969

33.34

0.936

33.04

0.922

32.91

0.926

Uformer [93]

29.17

0.880

30.06

0.884

36.34

0.966

33.36

0.935

31.98

0.909

32.18

0.915

MAXIM-2S [84]

31.17

0.922

30.81

0.903

38.06

0.977

33.80

0.943

32.37

0.922

33.24

0.933

Restormer [101]

32.00

0.923

31.46

0.904

38.99

0.978

34.18

0.944

33.19

0.926

33.96

0.935

SFNet [20]

31.47

0.919

31.90

0.908

38.21

0.974

33.69

0.937

32.55

0.911

33.56

0.929

PromptRestorer

31.84

0.920

31.72

0.908

39.04

0.977

34.40

0.947

33.27

0.928

34.05

0.936

PSNR

20

.

83

dB

21

.

89

dB

21

.

51

dB

22

.

41

dB

22.46

dB

∞

(a) Input

(b) RESCAN

(c) DCSFN

(d) MPRNet

(e) Restormer

(f)

PromptRestorer

(g) GT

Benchmark

Metrics

Nah et al. SRN DBGAN MT-RNN DMPHN Suin et al. SPAIR MIMO-UNet+ MPRNet Restormer

PromptRestorer

[65]

[81]

[109]

[68]

[105]

[80]

[69]

[18]

[103]

[101]

GoPro

[65]

PSNR

↑

21.00

30.26

31.10

31.15

31.20

31.85

32.06

32.45

32.66

32.92

33.06

SSIM

↑

0.914

0.934

0.942

0.945

0.940

0.948

0.953

0.957

0.959

0.961

0.962

HIDE

[76]

PSNR

↑

25.73

28.36

28.94

29.15

29.09

29.98

30.29

29.99

30.96

31.22

31.36

SSIM

↑

0.874

0.915

0.915

0.918

0.924

0.930

0.931

0.930

0.939

0.942

0.944

RealBlur-R

[73]

PSNR

↑

32.51

35.66

33.78

35.79

35.70

-

-

35.54

35.99

36.19

36.06

SSIM

↑

0.841

0.947

0.909

0.951

0.948

-

-

0.947

0.952

0.957

0.954

RealBlur-J

[73]

PSNR

↑

27.87

28.56

24.93

28.44

28.42

-

28.81

27.63

28.70

28.96

28.82

SSIM

↑

0.827

0.867

0.745

0.862

0.860

-

0.875

0.837

0.873

0.879

0.873

PSNR

24

.

83

dB

30

.

40

dB

31

.

04

dB

31

.

07

dB

31.68

dB

∞

(a) Degraded Image (b) Degraded Patch

(d) SRN

(e) MPRNet

(f)

PromptRestorer

(g) GT

6314823

6314823

6374823

6374823

63双4823

4.1 Main Results

Image Deraining Results. Similar to existing methods [44, 103, 69], we report PSNR/SSIM scores using Y channel in YCbCr color. Tab. 1 shows that our PromptRestorer outperforms current stateof-the-art approaches when averaged across all five datasets. Compared to the recent best method Restormer [101], PromptRestorer achieves 0 . 09 dB improvement on average. On individual datasets, the gain can be as large as 0 . 22 dB, e.g., Test2800 [27]. In Fig. 4, we present a challenging visual deraining example, where our PromptRestorer is able to generate a clearer result with finer details.

Image Deblurring Results. We evaluate deblurring results on both synthetic datasets (GoPro [65], HIDE [76]) and real-world datasets (RealBlur-R [73], RealBlur-J [73]). Tab. 2 summarises the results, where our PromptRestorer advances current state-of-the-art approaches on GoPro [65] and HIDE [76]. Compared with MPRNet [103], our PromptRestorer obtains a performance 0 . 12 dB gains. Fig. 5 provides a visual deblurring example. Our PromptRestorer produces a sharper result with fewer artifacts.

PSNR

26

.

18

dB

29

.

14

dB

15

.

52

dB

32

.

77

dB

36.86

dB

∞

(a) Input

(b) GridNet

(c) MSBDN

(d) UHD

(f)

PromptRestorer

(g) GT

PSNR

24

.

18

dB

26

.

81

dB

29

.

07

dB

29

.

72

dB

34.70

dB

∞

(a) Input

(b) JSTASR

(c) HDCWNet

(e) Restormer

(f)

PromptRestorer

(g) GT

12

12

1

12

Image Dehazing Results. We perform the image dehazing experiments on both synthetic benchmark RESIDE SOTS-Indoor [52], and real-world hazy benchmarks Dense-Haze [2] and NH-Haze [3]. Tab. 3 summarise the quantitative results. Compared to the recent works DeHamer [33] and MAXIM [84], our method receives 4 . 33 dB and 5 . 91 dB PSNR gains on the SOTS-Indoor, respectively. On the real-world benchmark NH-Haze [3], our PromptRestorer can achieve 0 . 7203 of the SSIM result, which is a new record and significantly outperforms current state-of-the-art approaches DeHamer [33]. The results on both synthetic and real-world benchmarks have demonstrated the effectiveness of our PromptRestorer on the image dehazing task. Fig. 6 shows the visual results, where our PromptRestorer is more effective in removing haze than other methods.

Image Desnowing Results. For the image desnowing task, we compare our PromptRestorer on the CSD [15], SRRS [14], and Snow100K [61] datasets with existing state-of-the-art methods [61, 14, 15, 13, 85]. We also compare recent Transformer-based general image restoration approaches Restormer [101] and Uformer [93]. As shown in Tab. 4, our PromptRestorer yields a 2 . 05 dB PSNR improvement over the state-of-the-art approach [101] on the CSD benchmark [15]. The visual results in Fig. 7 show that our PromptRestorer is able to remove spatially varying snowflakes than competitors.

. Each component in L2P and G2P is effective.

4.2 Analysis and Discussion

For ablation experiments, following [84, 20], we train the image deblurring model on GoPro dataset [65] for 1000 epochs only and set the number of Transformer in each CGT is 1. Params mean the number of learnable parameters. Testing is performed on the GoPro testing dataset [65]. FLOPs are computed on image size 256 × 256 . Next, we describe the influence of each component individually.

Effect on Prompting. The core design of our PromptRestorer is the 'prompting', which exploits a pre-trained model to extract raw degradation features from the degraded observations and then generate perceived content to guide the restoration branch (i.e., Case 3 in Fig. 1).

Compared to existing frameworks such as Cases 1-2 in Fig. 1, our proposed prompting strategy shows superior performance, as demonstrated in Tab. 5. Our method achieved 0 . 877 dB gains compared to Case 1 , and 0 . 369 dB higher than Case 2 . Interestingly, the learnable condition branch in Case 2 2 , despite consuming more FLOPs and Params, results in worse performance than ours. Our approach directly exploits raw degradation features to prompt restoration with persistent degradation priors to facilitate

better recovery. Fig. 1(c) shows two examples, where our model that exploits raw degradation features as prompting generates sharper and clearer images.

Effect on PromptDPM. We analyze the impact of PromptDPM on restoration quality in Tab. 6 by disabling one component at a time. Each model in L2P and G2P consumes similar Params and FLOPs, while our full model achieves the best performance. Disabling L2P or G2P results in a decrease in performance by 0 . 196 dB and 0 . 318 dB, respectively. These experiments conclusively demonstrate the effectiveness of each component in L2P and G2P for restoration.

Effect on GDP. To understand the impact of GDP, we disable it to compare with full model in Tab. 7. Table 7: Effect on GDP . Our GDP which controls

Note that the computational cost of the GDP is negligible compared to disabling it as it only involves a 1 × 1 convolution and sigmoid function for the gating mechanism, while it leads to a gain of 0 . 091 dB. This finding highlights the significance of controlling the propagation of the perceived degradation features.

4.3 Visualization Understanding for Degradation Vanishing

To emphasize the understanding of degradation vanishing, we visualize the features learned in the condition/prompting branches to better comprehend the learned status of these branches in Fig. 8. Notably, both Cases 1-2 exhibit sharper results in later iterations compared to earlier ones, which fail to provide the restoration branch with sufficient degraded information and cause the restoration

models to not perceive the degradation well, thereby hindering the model capacity. In contrast, as the restoration branch needs to adapt perceived features from the PromptDPM which is to perceive the raw degradation features from inputs, our model ( Case 3 ) initially exhibits inferior performance (around 20K iterations) as shown in Fig. 1(b). However, with better adaptation to the degradation information after more iterations, the prompting branch can better prompt the restoration branch consistently with more reliable perceived content learned from the raw degradation, enabling our restoration branch to overcome degradation vanishing and improve restoration quality, as shown in Fig. 1(c).

10K IWeUaWiRQV

500K IWeUaWiRQV

10K IWeUaWiRQV

500K IWeUaWiRQV

CRQViVWeQW DegUaded FeaWXUeV

GT FeaWXUeV

Case1

Case2

Case3

5 Concluding Remarks

In this paper, we investigate the degradation vanishing in the learning process for image restoration. To solve this problem, we have proposed the PromptRestorer which explores the raw degradation features extracted by a pre-trained model from the given degraded observations to guide the restoration process to facilitate better recovery. Extensive experiments have demonstrated that our PromptRestorer favors against state-of-the-art approaches on 4 restoration tasks, including image deraining, deblurring, dehazing, and desnowing.

References

A. Abuolaim and M. S. Brown. Defocus deblurring using dual-pixel data. In ECCV , 2020.

C. O. Ancuti, C. Ancuti, M. Sbert, and R. Timofte. Dense-haze: A benchmark for image dehazing with dense-haze and haze-free images. In ICIP , pages 1014-1018, 2019.

C. O. Ancuti, C. Ancuti, and R. Timofte. Nh-haze: An image dehazing benchmark with non-homogeneous hazy and haze-free images. In CVPR workshops , pages 444-445, 2020.

S. Anwar and N. Barnes. Densely residual laplacian super-resolution. TPAMI , 2020.

S. Anwar, S. Khan, and N. Barnes. A deep journey into super-resolution: A survey. ACM Computing Surveys , 2019.

J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv:1607.06450 , 2016.

D. Berman, T. Treibitz, and S. Avidan. Non-local image dehazing. In CVPR , pages 1674-1682, 2016.

T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. arXiv:2005.14165 , 2020.

B. Cai, X. Xu, K. Jia, C. Qing, and D. Tao. Dehazenet: An end-to-end system for single image haze removal. IEEE TIP , 25(11):5187-5198, 2016.

H. Cai, J. He, Y. Qiao, and C. Dong. Toward interactive modulation for photo-realistic image restoration. In CVPR Workshops , pages 294-303, 2021.

C. Chen, X. Shi, Y. Qin, X. Li, X. Han, T. Yang, and S. Guo. Real-world blind super-resolution via feature matching with implicit high-resolution priors. In ACM MM , pages 1329-1338, 2022.

H. Chen, Y. Wang, T. Guo, C. Xu, Y. Deng, Z. Liu, S. Ma, C. Xu, C. Xu, and W. Gao. Pre-trained image processing transformer. In CVPR , 2021.

S. Chen, T. Ye, Y. Liu, T. Liao, Y. Ye, and E. Chen. Msp-former: Multi-scale projection transformer for single image desnowing. arXiv preprint arXiv:2207.05621 , 2022.

W.-T. Chen, H.-Y. Fang, J.-J. Ding, C.-C. Tsai, and S.-Y. Kuo. Jstasr: Joint size and transparency-aware snow removal algorithm based on modified partial convolution and veiling effect removal. In ECCV , pages 754-770, 2020.

W.-T. Chen, H.-Y. Fang, C.-L. Hsieh, C.-C. Tsai, I. Chen, J.-J. Ding, S.-Y. Kuo, et al. All snow removed: Single image desnowing algorithm using hierarchical dual-tree complex wavelet representation and contradict channel loss. In ICCV , pages 4196-4205, 2021.

X. Chen, Y. Liu, Z. Zhang, Y. Qiao, and C. Dong. Hdrunet: Single image hdr reconstruction with denoising and dequantization. In CVPR Workshops , pages 354-363, 2021.

X. Chen, Z. Zhang, J. S. Ren, L. Tian, Y. Qiao, and C. Dong. A new journey from sdrtv to hdrtv. In ICCV , pages 4500-4509, 2021.

S.-J. Cho, S.-W. Ji, J.-P. Hong, S.-W. Jung, and S.-J. Ko. Rethinking coarse-to-fine approach in single image deblurring. In ICCV , 2021.

X. Cui, C. Wang, D. Ren, Y. Chen, and P. Zhu. Semi-supervised image deraining using knowledge distillation. IEEE TCSVT , 32(12):8327-8341, 2022.

Y. Cui, Y. Tao, Z. Bing, W. Ren, X. Gao, X. Cao, K. Huang, and A. Knoll. Selective frequency network for image restoration. In ICLR , 2023.

H. Dong, J. Pan, L. Xiang, Z. Hu, X. Zhang, F. Wang, and M. Yang. Multi-scale boosted dehazing network with dense feature fusion. In CVPR , pages 2154-2164, 2020.

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR , 2021.

Y. Du, F. Wei, Z. Zhang, M. Shi, Y. Gao, and G. Li. Learning to prompt for open-vocabulary object detection with vision-language model. In CVPR , pages 14064-14073, 2022.

A. Dudhane, S. W. Zamir, S. Khan, F. Khan, and M.-H. Yang. Burst image restoration and enhancement. In CVPR , 2022.

P. Esser, R. Rombach, and B. Ommer. Taming transformers for high-resolution image synthesis. In CVPR , pages 12873-12883, 2021.

X. Fu, J. Huang, X. Ding, Y. Liao, and J. Paisley. Clearing the skies: A deep network architecture for single-image rain removal. TIP , 2017.

X. Fu, J. Huang, D. Zeng, Y. Huang, X. Ding, and J. Paisley. Removing rain from single images via a deep detail network. In CVPR , 2017.

Y. Gan, X. Ma, Y. Lou, Y. Bai, R. Zhang, N. Shi, and L. Luo. Decorate the newcomers: Visual domain prompt for continual test time adaptation. In AAAI , 2023.

P. Gao, S. Geng, R. Zhang, T. Ma, R. Fang, Y. Zhang, H. Li, and Y. Qiao. Clip-adapter: Better vision-language models with feature adapters. arXiv preprint arXiv:2110.04544 , 2021.

J. Gu, H. Lu, W. Zuo, and C. Dong. Blind super-resolution with iterative kernel correction. In CVPR , 2019.

S. Gu, Y. Li, L. V. Gool, and R. Timofte. Self-guided network for fast image denoising. In ICCV , 2019.

Y. Gu, X. Wang, L. Xie, C. Dong, G. Li, Y. Shan, and M. Cheng. VQFR: blind face restoration with vector-quantized dictionary and parallel decoder. In ECCV , pages 126-143, 2022.

C.-L. Guo, Q. Yan, S. Anwar, R. Cong, W. Ren, and C. Li. Image dehazing transformer with transmissionaware 3d position embedding. In CVPR , pages 5812-5820, 2022.

J. He, C. Dong, Y. Liu, and Y. Qiao. Interactive multi-dimension modulation for image restoration. TPAMI , 44(12):9363-9379, 2022.

J. He, C. Dong, and Y. Qiao. Interactive multi-dimension modulation with dynamic controllable residual learning for image restoration. In A. Vedaldi, H. Bischof, T. Brox, and J. Frahm, editors, ECCV , volume 12365, pages 53-68, 2020.

J. He, Y. Liu, Y. Qiao, and C. Dong. Conditional sequential modulation for efficient global image retouching. In ECCV , volume 12358, pages 679-695. Springer, 2020.

K. He, J. Sun, and X. Tang. Single image haze removal using dark channel prior. TPAMI , 2010.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR , pages 770-778, 2016.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR , 2016.

R. Herzig, O. Abramovich, E. Ben-Avraham, A. Arbelle, L. Karlinsky, A. Shamir, T. Darrell, and A. Globerson. Promptonomyvit: Multi-task prompt learning improves video transformers using synthetic scene data. CoRR , abs/2212.04821, 2022.

Z. Hu, S. Cho, J. Wang, and M.-H. Yang. Deblurring low-light images with light streaks. In CVPR , 2014.

J.-B. Huang, A. Singh, and N. Ahuja. Single image super-resolution from transformed self-exemplars. In CVPR , 2015.

M. Jia, L. Tang, B.-C. Chen, C. Cardie, S. Belongie, B. Hariharan, and S.-N. Lim. Visual prompt tuning. In ECCV , pages 709-727, 2022.

K. Jiang, Z. Wang, P. Yi, B. Huang, Y. Luo, J. Ma, and J. Jiang. Multi-scale progressive fusion network for single image deraining. In CVPR , 2020.

S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah. Transformers in vision: A survey. arXiv:2101.01169 , 2021.

M. U. Khattak, H. A. Rasheed, M. Maaz, S. Khan, and F. S. Khan. Maple: Multi-modal prompt learning. CoRR , abs/2210.03117, 2022.

J. Kopf, B. Neubert, B. Chen, M. Cohen, D. Cohen-Or, O. Deussen, M. Uyttendaele, and D. Lischinski. Deep photo: Model-based photograph enhancement and viewing. ACM TOG , 2008.

A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS , 2012.

M. Kumar, D. Weissenborn, and N. Kalchbrenner. Colorization transformer. In ICLR , 2021.

O. Kupyn, T. Martyniuk, J. Wu, and Z. Wang. DeblurGAN-v2: Deblurring (orders-of-magnitude) faster and better. In ICCV , 2019.

B. Li, X. Peng, Z. Wang, J. Xu, and D. Feng. Aod-net: All-in-one dehazing network. In ICCV , pages 4780-4788, 2017.

B. Li, W. Ren, D. Fu, D. Tao, D. Feng, W. Zeng, and Z. Wang. Benchmarking single-image dehazing and beyond. TIP , 28(1):492-505, 2019.

M. Li, L. Chen, Y. Duan, Z. Hu, J. Feng, J. Zhou, and J. Lu. Bridge-prompt: Towards ordinal action understanding in instructional videos. In CVPR , pages 19880-19889, 2022.

S. Li, I. B. Araujo, W. Ren, Z. Wang, E. K. Tokuda, R. H. Junior, R. Cesar-Junior, J. Zhang, X. Guo, and X. Cao. Single image deraining: A comprehensive benchmark analysis. In CVPR , 2019.

X. Li, J. Wu, Z. Lin, H. Liu, and H. Zha. Recurrent squeeze-and-excitation context aggregation net for single image deraining. In ECCV , 2018.

J. Liang, J. Cao, G. Sun, K. Zhang, L. Van Gool, and R. Timofte. SwinIR: Image restoration using swin transformer. In ICCV Workshops , 2021.

X. Liu, J. Hu, X. Chen, and C. Dong. Udc-unet: Under-display camera image restoration via u-shape dynamic network. In L. Karlinsky, T. Michaeli, and K. Nishino, editors, ECCV Workshops , volume 13805, pages 113-129, 2022.

X. Liu, Y. Ma, Z. Shi, and J. Chen. Griddehazenet: Attention-based multi-scale network for image dehazing. In ICCV , pages 7313-7322, 2019.

X. Liu, M. Suganuma, Z. Sun, and T. Okatani. Dual residual networks leveraging the potential of paired operations for image restoration. In CVPR , 2019.

Y. Liu, J. He, X. Chen, Z. Zhang, H. Zhao, C. Dong, and Y. Qiao. Very lightweight photo retouching network with conditional sequential modulation. TMM , 2022.

Y.-F. Liu, D.-W. Jaw, S.-C. Huang, and J.-N. Hwang. Desnownet: Context-aware deep network for snow removal. TIP , 27(6):3064-3073, 2018.

Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie. A convnet for the 2020s. In CVPR , pages 11976-11986, 2022.

I. Loshchilov and F. Hutter. SGDR: Stochastic gradient descent with warm restarts. In ICLR , 2017.

T. Michaeli and M. Irani. Nonparametric blind super-resolution. In ICCV , 2013.

S. Nah, T. Hyun Kim, and K. Mu Lee. Deep multi-scale convolutional neural network for dynamic scene deblurring. In CVPR , 2017.

J. Pan, Z. Hu, Z. Su, and M.-H. Yang. l 0 -regularized intensity and gradient prior for deblurring text images and beyond. TPAMI , 39(2):342-355, 2017.

J. Pan, D. Sun, H. Pfister, and M.-H. Yang. Blind image deblurring using dark channel prior. In CVPR , 2016.

D. Park, D. U. Kang, J. Kim, and S. Y. Chun. Multi-temporal recurrent neural networks for progressive non-uniform single image deblurring with incremental temporal training. In ECCV , 2020.

K. Purohit, M. Suin, A. Rajagopalan, and V. N. Boddeti. Spatially-adaptive image restoration using distortion-guided networks. In ICCV , 2021.

X. Qin, Z. Wang, Y. Bai, X. Xie, and H. Jia. Ffa-net: Feature fusion attention network for single image dehazing. In AAAI , volume 34, pages 11908-11915, 2020.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In ICML , pages 8748-8763, 2021.

D. Ren, W. Zuo, Q. Hu, P. Zhu, and D. Meng. Progressive image deraining networks: A better and simpler baseline. In CVPR , 2019.

J. Rim, H. Lee, J. Won, and S. Cho. Real-world blur dataset for learning and benchmarking deblurring algorithms. In ECCV , 2020.

O. Ronneberger, P. Fischer, and T. Brox. U-Net: convolutional networks for biomedical image segmentation. In MICCAI , 2015.

T. Schick and H. Schütze. Exploiting cloze-questions for few-shot text classification and natural language inference. In EACL , pages 255-269, 2021.

Z. Shen, W. Wang, X. Lu, J. Shen, H. Ling, T. Xu, and L. Shao. Human-aware motion deblurring. In ICCV , 2019.

W. Shi, J. Caballero, F. Huszár, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert, and Z. Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In CVPR , 2016.

T. Shin, Y. Razeghi, R. L. L. IV, E. Wallace, and S. Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. In EMNLP , pages 4222-4235, 2020.

H. Singh, A. Kumar, L. K. Balyan, and G. K. Singh. A novel optimally gamma corrected intensity span maximization approach for dark image enhancement. In DSP , pages 1-5, 2017.

M. Suin, K. Purohit, and A. N. Rajagopalan. Spatially-attentive patch-hierarchical network for adaptive motion deblurring. In CVPR , 2020.

X. Tao, H. Gao, X. Shen, J. Wang, and J. Jia. Scale-recurrent network for deep image deblurring. In CVPR , 2018.

C. Tian, L. Fei, W. Zheng, Y. Xu, W. Zuo, and C.-W. Lin. Deep learning on image denoising: An overview. Neural Networks , 2020.

R. Timofte, V. De Smet, and L. Van Gool. Anchored neighborhood regression for fast example-based super-resolution. In ICCV , 2013.

Z. Tu, H. Talebi, H. Zhang, F. Yang, P. Milanfar, A. Bovik, and Y. Li. Maxim: Multi-axis mlp for image processing. In CVPR , pages 5769-5780, 2022.

J. M. J. Valanarasu, R. Yasarla, and V. M. Patel. Transweather: Transformer-based restoration of images degraded by adverse weather conditions. In CVPR , pages 2353-2363, 2022.

C. Wang, J. Pan, W. Lin, J. Dong, and X.-M. Wu. Selfpromer: Self-prompt dehazing transformers with depth-consistency. arXiv preprint arXiv:2303.07033 , 2023.

C. Wang, J. Pan, and X. Wu. Online-updated high-order collaborative networks for single image deraining. In AAAI , pages 2406-2413, 2022.

C. Wang, Y. Wu, Z. Su, and J. Chen. Joint self-attention and scale-aggregation for self-calibrated deraining network. In ACM MM , pages 2517-2525, 2020.

C. Wang, X. Xing, Y. Wu, Z. Su, and J. Chen. DCSFN: deep cross-scale fusion network for single image rain removal. In ACM MM , pages 1643-1651. ACM, 2020.

C. Wang, X. Xing, Y. Wu, Z. Su, and J. Chen. DCSFN: deep cross-scale fusion network for single image rain removal. In ACM MM , pages 1643-1651, 2020.

C. Wang, H. Zhu, W. Fan, X. Wu, and J. Chen. Single image rain removal using recurrent scale-guide networks. Neurocomputing , 467:242-255, 2022.

X. Wang, K. Yu, C. Dong, and C. C. Loy. Recovering realistic texture in image super-resolution by deep spatial feature transform. In CVPR , 2018.

Z. Wang, X. Cun, J. Bao, and J. Liu. Uformer: A general u-shaped transformer for image restoration. arXiv:2106.03106 , 2021.

Z. Wang, X. Cun, J. Bao, W. Zhou, J. Liu, and H. Li. Uformer: A general u-shaped transformer for image restoration. In CVPR , pages 17683-17693, 2022.

W. Wei, D. Meng, Q. Zhao, Z. Xu, and Y. Wu. Semi-supervised transfer learning for image rain removal. In CVPR , 2019.

F. Yang, H. Yang, J. Fu, H. Lu, and B. Guo. Learning texture transformer network for image superresolution. In CVPR , 2020.

W. Yang, R. T. Tan, J. Feng, J. Liu, Z. Guo, and S. Yan. Deep joint rain detection and removal from a single image. In CVPR , 2017.

Y. Yao, A. Zhang, Z. Zhang, Z. Liu, T. Chua, and M. Sun. CPT: colorful prompt tuning for pre-trained vision-language models. CoRR , abs/2109.11797, 2021.

R. Yasarla and V. M. Patel. Uncertainty guided multi-scale residual learning-using a cycle spinning cnn for single image de-raining. In CVPR , 2019.

Z. Yue, Q. Zhao, L. Zhang, and D. Meng. Dual adversarial network: Toward real-world noise removal and noise generation. In ECCV , 2020.

S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, and M.-H. Yang. Restormer: Efficient transformer for high-resolution image restoration. In CVPR , pages 5718-5729, 2022.

S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, M.-H. Yang, and L. Shao. Learning enriched features for real image restoration and enhancement. In ECCV , 2020.

S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, M.-H. Yang, and L. Shao. Multi-stage progressive image restoration. In CVPR , 2021.

Y. Zang, W. Li, K. Zhou, C. Huang, and C. C. Loy. Unified vision and language prompt learning. CoRR , abs/2210.07225, 2022.

H. Zhang, Y. Dai, H. Li, and P. Koniusz. Deep stacked hierarchical multi-patch network for image deblurring. In CVPR , 2019.

H. Zhang and V. M. Patel. Density-aware single image de-raining using a multi-stream dense network. In CVPR , 2018.

H. Zhang, V. Sindagi, and V. M. Patel. Image de-raining using a conditional generative adversarial network. TCSVT , 2019.

K. Zhang, Y. Li, W. Zuo, L. Zhang, L. Van Gool, and R. Timofte. Plug-and-play image restoration with deep denoiser prior. TPAMI , 2021.

K. Zhang, W. Luo, Y. Zhong, L. Ma, B. Stenger, W. Liu, and H. Li. Deblurring by realistic blurring. In CVPR , 2020.

Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong, and Y. Fu. Image super-resolution using very deep residual channel attention networks. In ECCV , 2018.

Y. Zhang, K. Li, K. Li, B. Zhong, and Y. Fu. Residual non-local attention networks for image restoration. In ICLR , 2019.

Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu. Residual dense network for image restoration. TPAMI , 2020.

Z. Zheng, W. Ren, X. Cao, X. Hu, T. Wang, F. Song, and X. Jia. Ultra-high-definition image dehazing via multi-guided bilateral learning. In CVPR , pages 16185-16194, 2021.

Z. Zheng, X. Yue, K. Wang, and Y. You. Prompt vision transformer for domain generalization. CoRR , abs/2208.08914, 2022.

K. Zhou, J. Yang, C. C. Loy, and Z. Liu. Conditional prompt learning for vision-language models. In CVPR , pages 16795-16804, 2022.

S. Zhou, K. C. K. Chan, C. Li, and C. C. Loy. Towards robust blind face restoration with codebook lookup transformer. In NeurIPS , 2022.

H. Zhu, C. Wang, Y. Zhang, Z. Su, and G. Zhao. Physical model guided deep image deraining. In ICME , 2020.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the PromptRestorer, a method for image restoration that effectively utilizes raw degradation features to guide the restoration process. PromptRestorer consists of two branches: a restoration branch and a prompting branch. The restoration branch restores the images, while the prompting branch perceives degradation priors and provides reliable content to guide the restoration process. Experimental results demonstrate that PromptRestorer achieves state-of-the-art performance on four image restoration tasks: deraining, deblurring, dehazing, and desnowing.\n\nSTRENGTHS:\n- The idea of utilizing intermediate layer features of pre-trained models to enhance the effectiveness of image restoration is innovative.\n- The experimental validation tasks and comparison methods employed in this study are comprehensive and diverse.\n- The comparative figure (Figure 1) between the proposed method and previous approaches are presented in a clear manner, highlighting the distinctive aspects of the framework.\n\nWEAKNESSES:\n- The motivation behind the introduction of the prompting degradation perception modulator is not clearly articulated in the paper. The authors fail to thoroughly analyze the existing deficiencies of previous methods and instead claim that the incorporation of prompting can lead to more accurate degradation priors or more reliable perceived content learned from the degradation priors. However, these descriptions lack quantifiable metrics and robust support from relevant literature. Despite the inclusion of visualizations depicting feature maps in the experimental section, there remains an ambiguity regarding the definition of better degradation priors.\n- Similarly, I am confused regarding the motivation behind the introduction of local prompting attention and global prompting attention. It remains unclear to what extent these components address existing shortcomings in the framework. Furthermore, the results of the ablation experiments indicate relatively minor improvements resulting from the incorporation of these components.\n\nQUESTIONS:\n- What is a more accurate degradation priors? Can we measure it? \n- What are the motivations behind the introduction of key modules and the issues they address in previous methods? \n- It appears that the network parameter count and FLOPS of this method may increase significantly compared to those of Case 1 and Case 2 methods. Is there more detailed comparative data available in this regard?\n\nLIMITATIONS:\n- The performance improvement of this method appears to be relatively modest compared to some previous approaches, and it remains unclear what the magnitude of the associated numerical variances is.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

comments: SUMMARY:
In this paper, the authors find that the degradation vanishes in the learning process of the image restoration problem. Which further hinders the model's capacity. To solve this problem, this paper proposes the PromptRestorer, a prompting image restoration method by exploring the raw degradation features extracted by a pre-trained model from the given degraded observations to guide the restoration process. The authors demonstrate that the proposed guidance manner is better than both the encoder-decoder model without any guidance and the model with learnable guidance. The experiments on image deraining, deblurring, dehazing, and desnowing demonstrate the superiority of the proposed algorithm.

STRENGTHS:
1, The method is simple and effective and is easy to follow. The writing is professional and clear to understand.
2, The proposed Prompting Degradation Perception Modulator including global prompting perceptor and local prompting perceptor is novel and effective.
3, The proposed Gated Degradation Perception Propagation is simple and meaningful.
4, This paper clarifies a new perspective that the raw degradation features are better than the learnable features which are usually utilized in former methods.
5, The extensive experiments on various image restoration tasks demonstrate the generalization of the proposed algorithm.

WEAKNESSES:
1, Although this paper clarifies many strengths of the method, what’s the limitation? The author should present it in this paper.
2, The authors should give more detailed explanation of the proposed global prompting perceptor and local prompting perceptor to facilitate better understanding.

QUESTIONS:
See Weaknesses

LIMITATIONS:
The authors are suggested to discuss the limitations of this paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the PromptRestorer by taking advantage of the prompting learning for image restoration. The author considers raw degradation features into restoration which consistently retains the degradation priors to facilitate better restoration. To better perceive degradation, the authors propose Global Prompting Perceptor and Local Prompting Perceptor. \nTo control the propagation of the perceived features, the authors propose gated degradation perception propagation, enabling the model to adaptively learn more useful features for better image restoration.\n\nSTRENGTHS:\n1, Good writing skills. The paper shows excellent presentation and clearly presents former disadvantages and better solves them by introducing the prompting strategy.\n2, Clear motivation. To solve the problem of degradation vanishing in image restoration (Fig. 1 and 8), the authors introduce a Prompting Degradation Perception by extracting the raw degradation features to directly guide the image restoration process. The experiments (Tab. 5) show that the proposed learning strategy is better than the former methods.\n3, Interesting module. The proposed Prompting Degradation Perception Modulator is interesting and Gated Degradation Perception Propagation is simple.\n\nWEAKNESSES:\n1, The authors can give a clearer understanding of the degradation priors guidance. Why does it work better than the model with a learnable conditional branch?\n2, Although the Prompting Degradation Perception Modulator including the Global Prompting Perceptor and Local Prompting Perceptor is interesting, it would be better if the authors can give a more direct explanation of them as the two modules seem to look the key contributions of this work.\n3, The paper exists some typos. For example, in line 89, knew->known.\n\nQUESTIONS:\nPlease refer to the questions in Weaknesses\n\nLIMITATIONS:\nThe authors do not present the limitations in the paper, which are suggested to add them.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

comments: SUMMARY:
This paper proposes a prompting image restoration method with degradation perception. The authors show that raw degradation features can effectively guide deep restoration models, providing accurate degradation priors to facilitate better restoration. To perceive the degradation, the authors propose prompting degradation perception modulator. To control the propagation of the perceived content for restoration branch, the authors propose gated degradation perception propagation.

STRENGTHS:
+ This paper is professionally wrote and presents a better organization.
+ The proposed prompting image restoration method is interesting, which clearly demonstrate that the raw degradation features are better guidance than the learnable parameters used in previous methods.
+ The proposed prompting degradation perception modulator is novel. It has potential applications to latter features fusion manner.
+ The gated degradation perception propagation seems to be useful.

WEAKNESSES:
1, The author claim that the pre-trained model is the same with the learnable conditional network. Hence, the FLOPs in Tab. 5 should be same.

2, The authors would better to explain why the raw degradation features are better than features from learnable parameters network since more learnable parameters should have better results within the similar network framework.
3, SFT is a commonly used feature fusion module, is the proposed PromptDPM than that? What is the result if replacing the PromptDPM with SFT?

QUESTIONS:
see weakness

LIMITATIONS:
see weakness
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes PromptRestorer for image restoration, addressing the issue of degradation vanishing in existing methods. Unlike previous approaches that do not explicitly consider degradation information or fail to effectively model degradation priors, PromptRestorer leverages the raw degraded features extracted from pre-trained models to generate reliable prompting content. The contributions of the work include PromptRestorer, the first approach to leverage prompting learning for general image restoration by considering raw degradation features, overcoming degradation vanishing while consistently retaining degradation priors. Additionally, the proposed prompting degradation perception modulator and gated degradation perception propagation enhance the restoration process by providing more reliable perceived content and controlling feature propagation. Extensive experiments show the superiority of the proposed method.\n\nSTRENGTHS:\n- The paper introduces a new approach called PromptRestorer for image restoration, which addresses the issue of degradation vanishing. The integral designs take advantage of prompting learning and explicitly consider degradation information, making it a unique and innovative contribution to the field.\n\n- The proposed method effectively perceives and retains degradation priors by leveraging raw degraded features and designing a prompting degradation perception modulator, resulting in improved performance compared to existing methods.\n\n- This comprehensive framework ensures that all aspects of restoration, including generating reliable prompting content and controlling feature propagation.\n\n- The paper provides a detailed analysis of the proposed method and the experimental results demonstrate the effectiveness of the PromptRestorer and achieve better restoration outcomes.\n\nWEAKNESSES:\n- The main problem **degradation vanishing** is not well defined and discussed in this paper, leading to the confusing and distracting presentation that what problem the authors are trying to solve. In Section 4.3 the authors claim sharper details in features fail to provide sufficient degraded information. Why this is the case? Is there any evidence that can support sharper features of the conditional branch would deteriorate the restoration performance? Overall the analysis of degradation vanishing is insufficient.\n\n- The core idea of this paper seems like another attempt to utilize prior information from pre-trained large models (*e.g.*, StyleGAN, DDPM, CLIP) to facilitate restoration, which has long been explored [1,2,3,4] in the image restoration community, though the designs and technical details are rather different. Could the authors further elaborate on the main differences between previous works?\n\n- As claimed that PromptRestorer can consistently retain the degradation priors (L62), the reviewer would expect a better adaptation of this model to various degradations. Hence, it would be better if the authors can show how well this algorithm can perform on different degradations (*e.g.*, various sizes of blur kernels, various levels of noise).\n\n[1] Chan, Kelvin CK, et al. \"Glean: Generative latent bank for large-factor image super-resolution.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.\n\n[2] Lugmayr, Andreas, et al. \"Repaint: Inpainting using denoising diffusion probabilistic models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[3] Kim, Geonung, et al. \"Bigcolor: colorization using a generative color prior for natural images.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.\n\n[4] Chen, Chaofeng, et al. \"Real-world blind super-resolution via feature matching with implicit high-resolution priors.\" Proceedings of the 30th ACM International Conference on Multimedia. 2022.\n\nQUESTIONS:\nSee weaknesses section.\n\nLIMITATIONS:\nLimitations and broader societal impacts are not discussed in this paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a Prompt image Restorer, which contains a restoration branch and a prompting branch. A pre-trained model is utilized to extract features in the prompting branch and a prompting modulator is proposed to better perceive features from global and local perspectives in the restoration branch. Extensive experimental results of multiple restoration tasks show the effectiveness of their method.\n\nSTRENGTHS:\n1. The results in some experiments have shown improvement compared to previous methods.\n2. The paper is well-written and easy to understand.\n\nWEAKNESSES:\n1. The prompting way proposed in this paper is similar to the approach taken in some other papers, such as \"Take a Prior from Other Tasks for Severe Blur Removal\", where features are added as conditions. PromptRestorer doesn't have a significant difference, which makes me feel like it is more of a repackaging of prompting learning.\n2. As a model for image generation, why does the pre-trained model provide raw degradation information instead of image content information? What advantages does this approach have compared to some other methods of degradation representation? For example, \"Learning Degradation Representations for Image Deblurring\", and \"Learning Disentangled Feature Representation for Hybrid-distorted Image Restoration\".\n3. The comparisons with the latest methods in Table 2, such as Restormer and NAFNet, are missing.\n4. The article lacks comparisons with previous methods in terms of computational complexity and the number of parameters.\n5. The method in this paper employs a pre-trained  VQGAN, which introduces additional data. Is this experimental comparison fair? Did the authors account for the impact of this aspect?\n6. From Figure 8, it can be observed that your method visually resembles the features of the ground truth (GT). However, GT does not contain any degradation information, which contradicts the claims of raw degradation information made in your paper. This makes the motivation and claims of this paper unconvincing.\n\nQUESTIONS:\nPlease refer to Weaknesses.\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-16"}