PromptIR: Prompting for All-in-One Image Restoration
Authors: Vaishnav Potlapalli
Syed Waqas Zamir
Salman Khan
Fahad Khan
Venue: NeurIPS 2023 poster (2023)
Keywords: Image Restoration

Abstract: Image restoration involves recovering a high-quality clean image from its degraded version. Deep learning-based methods have significantly improved image restoration performance, however, they have limited generalization ability to different degradation types and levels. This restricts their real-world application since it requires training individual models for each specific degradation and knowing the input degradation type to apply the relevant model. We present a prompt-based learning approach, PromptIR, for All-In-One image restoration that can effectively restore images from various types and levels of degradation. In particular, our method uses prompts to encode degradation-specific information, which is then used to dynamically guide the restoration network. This allows our method to generalize to different degradation types and levels, while still achieving state-of-the-art results on image denoising, deraining, and dehazing.  Overall, PromptIR offers a generic and efficient plugin module with few lightweight prompts that can be used to restore images of various types and levels of degradation with no prior information on the corruptions present in the image. Our code and pre-trained models are available here: https://github.com/va1shn9v/PromptIR

PromptIR: Prompting for All-in-One Blind Image Restoration
Vaishnav Potlapalli", Syed Waqas Zamiri, Salman Khan', Fahad Shahbaz Khan" "Mohamed bin Zayed University of AI, Core42, 'Linköping University f firstname. 1 las testtammmmbzaii... .c
Abstract
Image restoration involves recovering a high-quality clean image from its degraded version. Deep learning-based methods have significantly improved image restora- tion performance, however, they have limited generalization ability to different degradation types and levels. This restricts their real-world application since it requires training individual models for each specific degradation and knowing the input degradation type to apply the relevant model. We present a prompt-based learning approach, PromptIR, for All-In-One image restoration that can effectively restore images from various types and levels of degradation. In particular, our method uses prompts to encode degradation-speciff information, which is then used to dynamically guide the restoration network. This allows our method to generalize to different degradation types and levels, while still achieving of the-art results on image denoising, deraining, and dehazing. Overall, PromptIR offers generic and efficient plugin module with few lightweight prompts that be used to restore images of various types and levels of degradation with no prior information on the corruptions present in the image. Our code and pre-trained models are available here: https://gi com / http/vaisshhnnn ww1shh9//Prottt..
Introduction
During image acquisition, degradations (such noise, blur, haze, rain, etc.) often introduced either due to the physical limitations of cameras or unsuitable ambient conditions. Image restoration refers the process of recovering a high-quality clean image from its degraded version. It is a highly challenging problem due to its ill-posed nature as there exists many feasible solutions, both natural and
unnatural. Recently, deep learning based restoration approaches (47 12
69
45
76
55
43 74 have
emerged as more effective choice in comparison to conventional methods 19
36
Deep neural based methods broadly differ in their approach addressing the image restora- tion problem. Some works incorporate explicit task-specific knowledge in the network to deal with the corresponding restoration task, such as denoising 45 76), deblurring 43 74), and dehaz- ing [4 12 36). However, these methods lack generalization beyond the specific degradation type and level. On the other hand, some works (56 68 59
g focus developing robust architecture design and learn image priors from data implicitly. These methods train separate copies of the same network for different degradation types, degradation levels, and in more extreme cases on different datasets. However, replicating the same restoration model for different degradation types, levels, and data distributions is compute-intensive and tedious process, and oftentimes impractical for essource-constrainee platforms like mobile and edge devices. Furthermore, to select an appropriate restoration model during testing, these approaches require prior knowledge regarding the degradation present in the input image.
71
70
37th Conference on Neural Information Processing Systems (NeurIPS 2023).

---

Degradation Specific Learned Context
Prompts
Plug-in Prompt Blocks
All-in-ane Restoration Network
(single stage training, apriari degracation opecification)
Unknown Degradation
Recovered Images
Figure I: This figure illustrates our PromptIR approach. We propose a plug-and-pla prompt module that implicitly predicts degradation-cont condiiaioon--oniiiiooneedd prompts to guide the restoration process of an input image with unknown degradation. The guidance from prompts is injected into the network at multiple decoding stages with few-learnable parameters. This allows learning an all-in-one unified model that can perform well across several image restoration tasks (e.g., draining, dehazing, and denoising).
Therefore, there a pressing need develop an all-in-one method that can effectively restore images from various types and levels of degradation.
One recent method, AirNet [29] addresses the all-in-one restoration task by employing the contrastive learning paradigm. This involves training an extra encoder to differentiate various types of image degradations. Although AirNet [29] yields state-of-the of-the-rrtttttt results, struggles to model fully disen- langled representations of different corruption types. Furthermore, the usage of an additional encoder for contrastive learning leads to higher training burden due the two-stage training approach.
To overcome these challenges, in this paper, we present a leatt--eernnnnbeeeeee approach to perform all-in-one image restoration (see Fig. [T]. Our method utilizes prompts, which parameters that encode crucial discriminative information about various types image degradation (as shown in Fig. 2). By interacting prompts with the feature representations of the main restoration network, we dynamically enhance the representations with degradation-specins knowledge, This adaptation enables the network to effectively restore images by dynamically adjusting its behavior. The main highlights of our work include,
of tunable
We present a prompting-basee based all-in-one blind restoration framework PromptIR that relies solely the input image recover clean image, without requiring any prior knowledge of the degradation present in the image.
Our prompt block is a plug-in module that be easily integrated into any existing restoration network. It consists of prompt generation module (PGM) and a prompt interaction module (PIM). The goal of the prompt block to generate input-conditioned prompts (via PGM) that are equipped with useful contextual information to guide the restoration network (with PIM) effectively remove the corruption from the input image.
Figure 2: The figure shows tSNE plots of the degradation embeddings used in PrompuR (ours) and the state-of- state-of-the-ar AirNet 22 Distinct colors denote dif- ferent degradation types. In our case, the embeddings for each task better clus- tered, showing the effectiveness of prompt tokens to learn discriminative degradation context that helps in restoration.
Hazy Rainy Noisy
AirNet
PromptI

---

Our comprehensive experiments demonstrate the dynamic adaptation behavior of PromptIR by achieving state-of-the-ar performance on various image restoration tasks, including image denoising, deraining, and dehazing using only unified PromptIR model.
2 Related Works
Multi-degradation Image Restoration: While single degradation image restoration methods 668 47
12
51] have received significant interest, multi-degradation image restoration relatively under-explored in the literature. A body of work focuses on images corrupted due to multiple weather conditions e.g., snow, fog, and rain [37 57 (3]. However, they train specific encoder or decoder parallel pathways for each weather degradation which requires knowing specific degradation type and is less scalable. Chen et al. [6] build unified model for multiple restoration tasks, like super-resolution denoising, and deraining, however, the model needs prior information about the corruption present in the input image it uses a multi- calli-heea-tail -till architecture. In blind image restoration, we have no prior information on the degradation present in the image. This kind of problem setting has been tackled in the context of image super-resolus resouutoso [3 39 TO Li er al. [32] introduce a unified model for denoising, draining, and dehazing, which uses an image encoder trained through contrastive learning to model good representations of the degradation, which are later used to predict the deformable convolution offsets in another network to perform the restoration. This method requires two-stage training and the effectiveness of contrastive learning hinges on accurately choosing the positive-negative pairs and the amount of data available. In comparison, our work is focused on developing a single-stage training pipeline for unified all-in-one image restoration that is conceptually simpler and works as drop-in module for multiple degradations.
69
45
76
55
43
74
Transformer- based restoration: Transformer [58] architectures have found great success across various computer vision tasks [25] such as image recognition
[4,66], object detection [5] 80 38) and semantic segmentation 664 59 78). Owing to their strong feature representation capability,
they are extended to image restoration tasks 7
59 56 D However, naive self-attention has
quadratic complexity w.r.t. the image size and this poses challenge for image restoration tasks where inputs are typically high-resolution. To address this, some works have proposed efficient transformer architectures [33 68 35) to reduce the computational costs. Specifically, SwinlR (35) uses windowed self-attention blocks along with convolutional layers to improve the efficiency of the model. Restormer (68) uses multi-depth convolution head attention
reduce the number of operations. In this work, we apply our PromptR to Restormer owing to its efficient design and high performance, however, our prompt block is generic and can work with other architectures.
Prompt learning: In natural language processing, prompting- based methods are means to provide in- context information models finetune them target task 3 However, instead of using specific manual instruction sets as prompts, learnable prompts enable better parameter-efficient adaptation of models 77). Prompt learning techniques can effectively model task-specific context hence they have
been used for finetuning to vision tasks 22 34 26 and incremental learning [62] 49 6 I. Prompt
learning-based based techniques have also been applied in the case of multitask learning 20 60 where
choosing the right prompt for each task remains critical. All these approaches target high-level
vision problems, however, our goal here is develop a generic model for low-level vision that can dynamically restore inputs based on their interaction with the prompts. The prompts act as an adaptive lightweight module to encode degradation context across multiple scales in the restoration network.
3 Method
In "All-in-one" image restoration, we aim to learn a single model M to restore an image I from a degraded image I, that has been degraded using a degradation D, while having no prior information about D. While the model initially "blind" to the nature of degradation, its performance in recovering a clean image can be enhanced by providing implicit contextual information about the type of degradation. In this paper, we present prompt learning- based image restoration framework PromptIR, shown in Fig. 3 Prompting is an efficient [33 and suitable (0) method for supplementing the model with relevant knowledge of the degradation type while recovering the clean image. The key element of PrompuR is the prompt blocks that first generate learnable prompt parameters, and then use these prompts to guide the model during the restoration process. Next, we describe the overall pipeline of our PrompuR framework and its components in detail.
3

---

Refrati
Transformer Block
Transformer Block bg
Skip connections
Input (1)
||
Restored (1)
PIM PGM
Interationn Madule
Transformer Block b
JK2C
2C
Transformer Block 1n
Prompt Block
Prompt
Components
Prompt Generation Module
4=M Transformer Block la
3 x
convalution
Transformer ia
MK4C
Prompt Block
Prompt Componentss
1 x
sanookzion
GAP
Avcrage
PIM: Prompt Interration Module
M2
Prompt Component Weight
Transformer Block 14
PGM
F P
Transformer
Fi
Wkieetwiee addiaan
Prompt Block
Channe webe consrtmetion
Comtanation
GAP
Upwampling
Devmmampling ng
PGM. Prompt
Componentts
Prompt
P.
Figure 3: Overview of the PromptR approach. We use a UNet-style network 668 with transformer blocks in the encoding and decoding stages. The primary component of the framework, i.e., the prompt block consists two modules, the Prompt Generation Module (PGM) and the Prompt Interaction Module (PIM). The prompt generation module generates the input-conditioned prompt P, using the input features F, and the Prompt Components. The prompt interaction module then dynamically adapts the input features using the generated prompt through the transformer block. The prompts interact with decoder features at multiple levels to enrich the degradation-specific context.
Overall pipeline. From a given degraded input image B#xWx3, PromptIR first extracts low-level features Fo R #xxC by applying convolution operation; where W is the spatial resolution and C² denotes the channels. Next, the feature embeddings Fo undergo a 4-level hierarchical encoder-decoder, transforming into deep features F, "xxxx2 x2C Each level of the encoder-decoder employs several Transformer blocks, with the number of blocks gradually increasing from the top level the bottom level maintain computational efficiency. Starting from the high-resolution input, the goal of the encoder is to progressively reduce the spatial resolution while increasing channel capacity, thereby yielding low-resolution latent representation F, R N W x8C From the cow-resouution latent features F,, the aim of the decoder is to gradually recover the high- resolution clean output. In order to assist the decoding process, incorporate prompt blocks in PromptIR framework. Prompt blocks are adapter modules that sequentially connect every two levels of the decoder. At each decoder level, the prompt block implicitly enriches the input features with information about the degradation type for guided recovery. Next, we describe the proposed prompt block and its core building modules detail.
3.1 Prompt Block
prompting-basee based techniques have been explored for parameter-efficient finetuning of large frozen models trained on source task S onto target task T. The effective performance of prompting-based techniques is attributed their ability to efficiently encode tusk-specific contextual information in prompt components. In the proposed PromptlR, prompt components are learnable parameters, that interact with the input features in order to enrich them with degradation type. Given N prompt-components P. E IR x and input features F, E pxxxxxxx the overall process of prompt block is defined as:
In NLP 48 21 34] and vision tasks 23
26
18
50
F = PIM(PGM Fi),
(I)
The prompt block consists of two key components: a prompt generation module (PGM) and prompt-interaction module (PIM), each of which we describe next.

---

3.1.1 Prompt Generation Module (PGM)
Prompt components Pe form a set of learnable parameters that interact with the incoming features to embed degradation information. One straightforward method for features-prone prompt interaction is to directly use the learned prompts to calibrate the features. However, such static approach may yield suboptimal results, as is agnostic to the input content. Therefore, we present PGM that dynamically predicts attention-basee weights from the imput features and apply them to prompt components to yield input-conditione prompts P. Furthermore, PGM creates a shared space to facilitate correlated knowledge sharing among prompt components.
To generate prompt-weii weights from the input features Fi, PGM first applies global average pooling (GAP) across spatial dimension to generate feature vector v E RRC Next, we pass v through a channel- downscaling convolution layer obtain compact feature vector, followed by the softmax operation, thus yielding prompt-weights w E IR- Finally, we use these weights to make adjustments in prompt components, followed by 3 convolution layer. Overall, the PGM process is summarized as:
== Convax3
Σ W.P.
Sof CootmaCoCoovipa (GAP(F:))
Since at inference time, it is necessary for the restoration network to be able handle images of different resolutions, we cannot use the prompt components P. with a fixed size. Therefore, we apply the bilinear upsampling operation to upscale the prompt components the same size as the incoming input features.
3.1.2 Prompt Interaction Module (PIM)
The primary goal of PIM enable interaction between the input features F1 and prompts P for
guided restoration.
In PIM, concatenate the generated prompts with the input features along the channel dimension. Next, we pass the concatenated representations through a Transformer block that exploits degradation information encoded in the prompts and transforms the input features.
The main contribution of this paper the prompt block, which a plug-in module, and architecture agnostic. Therefore, in the proposed PromptIR framework, we use an existing Transformer block [68], instead of developing new one. The Transformer block is composed sub-modules: Multi-Deonv head transposed attention (MDTA), and Gated-Deonv feedforward network (GDFN). MDTA applies self-attention operation across channels rather than the spatial dimension and has linear complexity. The goal of GDFN is to transform features in controlled manner, i.e., suppressing the less informative features and allowing only useful ones to propagate through the network. The overall process of PIM is:
two sequentially connected
F Conv3xa (MDFN FI; P))
where [: ] concatenation operation. MDTA is formulated as W Softmax (K Q/a) + X. Where X and Y are the input and output features. Q, K and V respectively represent query, key, and value projections that are obtained by applying Ix point-wise convolutions followed by 3x3 depth- wise convolutions
the layer normalized input feature maps. W,, is the point-wise convolution, a denotes a learnable scaling parameter, and represents dot-product interaction. The process of ((N((Y)) (LN(Y))) @ W W2 + Y. Where, ( is the 3x3
GDFN is defined as Z w (o(W) depth-wise convolution, o denotes element-wise multiplication, á is the GELU non-linearity, and LN is the layer normalization 22. The block diagram and additional details on the Transformer block are provided in the appendix.
4 Experiments
To demonstrate the effectiveness of the proposed PromptIR, we perform the evaluation on three representative image restoration tasks: image dehazing, image deraining, and image denoising. Following (29), we conduct experiments under two different experimental settings: (a) All-in-One, and (b) Single-task.
5

---

Table 1: Comparisons under All-in-one restoration setting: single model trained on a combined set
of images originating from different degradation types. When averaged across different tasks, our
PromptIR provides significant gain of 0.86 dB over the previous all-in-one method AirNet (29].
Method
Dehazing on SOTS [T]
Deraining on Rain100L
Denoising on BSD68 datasct 411)
Average
[6
15
0
25
50
BRDNet 52
23.23/0.895
27.42/0.88
32.26/0.898
29.76/0.836
26.34/0.836
27.80V0.
LPNet 117
20.84/0.828
24,88/0.78
26.47/0.7782
24.77/0.748
21.26/0.552
23.64/0.736
FDGAN [4
24.71/0.924
29.89/0.933
30.25/0.910
28.81/0.868
26.43/0.776
28.02/0.883
MPRNet
71
25.28/0.954
33.57/0.954
33.54/0.927
30.89/0.880
27.56/0.779
30.17/0.899
DI. D6
26.92/0.391
32.62/0.931
33.05/0.914
30.41/0.861
26.90/0.740
29.98/0.873
AirNet
29
27.94/0.962
34.90/09967
33.92/0.933
31.26/0.880
28.00/0.797
21,20/0.910
PromptIR (Ours)
30.58/0.974
36.37/0.972
33.98/0.933
31.31/0.888
28.06/0.799
32.06/0.913
Input
AirNet
PromptIR (Ours)
Figure 4: Dehazing comparisons for all-in-one methods on images from the SOTS dataset 31 The image quality of the results produced by PromptIR is visually better than the previous state-of-the-art approach A2rNet(29]
In the All-in-One setting, train unified model that recover images across all three degradation types. Whereas, for the Single-task task setting, we train separate models for different restoration tasks. The image quality scores for the best and second-best methods highlighted and underlined in the tables.
Implementation Details. Our PromptIR framework is end-to-end trainable and requires no pre- training of any individual component. The architecture of our PrompuR consists of 4-level encoder-decoder, with varying numbers of Transformer blocks at each level, specifically [4. 6, 6, 8] from level-1 to level-4.
We employ one prompt block between every two consecutive decoder levels, totaling 3 prompt blocks 5. The model is trained
in the overall PromptlR network. The total number of prompt components with batch size of 32 in the all-in-one setting, and with batch of 8 in the single-task setting. The network is optimized with an L, loss, and we use Adam optimizer (3, = 0.9, B32 = 0.999) with learning rate 2e 4 for 200 epochs. During training, we utilize cropped patches of size 128 128 as input, and to augment the training data, random horizontal and vertical flips are applied to the input images.
Datasets. We prepare datasets for different restoration tasks, following closely the prior work 29 For image denoising in the single-task setting, we use combined set of BSD400 [I] and WED 40 datasets for training. The BSD400 dataset contains 400 training images and the WED dataset has 4,744 images. From clean images of these datasets, we generate the noisy images by adding Gaussian noise with different noise levels (15, 25, 50). Testing is performed on BSD68 [4] and Urban 100 [22) datasets. For single-task image deraining, we use the 1001 (65 dataset, which consists of 200 clean-rainy image pairs for training, and 100 pairs for testing. Finally, for image
dehazing in the single-task setting, we utilize SOTS [3 dataset that contains 73.135 training images

---

Input
AirNet
PromptIR (Ours)
Figure 5: Image deraining comparisons for all-in-one methods on images from the Rain I00L dataset [16]. Our method effectively removes rain streaks to generate rain-free images.
Input
AirNet
PrompiI (Ours)
Figure 6: Denoising results for all-in-one methods.
and 500 testing images. Finally, to train unified model under the all-in-one setting, we combine all 4 aforementioned datasets and train single model that is later evaluated on multiple tasks.
4.1 Multiple Degradation All-in-One Results
We compare the proposed PromptIR with several general image restoration approaches |5
14 71] as well as with specialized all-in-one methods [16 29). Results are reported in Table
When averaged across different restoration tasks, algorithm yields 0.86 dB performance gain
over the previous best method AirNet [29). and 1.89 dB over the second best approach DL. [71]
Specifically, the proposed PromptIR significantly advances state-of-the-e by providing 2.64 dB PSNR improvement on the image dehazing task. The visual examples provided in Fig. 4 show that PrompuIR effectively removes haze from the input images, and generates cleaner results than
AirNet 29] Similarly on the image deraining task as shown in Table T the proposed PromptiR achieves substantial gain of 3.73 dB compared to DL. [16] and 1.47 dB over AirNet (29]. Visual comparisons in Fig. show that PromptIR is capable of removing rain streaks of various orientations and generates visually pleasant rain-free images. Finally, on denoising task, our method provides 1.16 db boost over the DI. algorithm [16] for high noise level of o-50. Qualitative examples are presented in Fig. 6 where our method reproduces noise-free images with better structural fidelity than the AirNet algorithm [29]

---

Table 2: Dehazing results the single-task setting on the SOTS benchmark dataset
31
Our
PromptIR achieves significant boost of 8.13 dB over AirNet 229
Method DehazeNet [ MSCNN [6
AODNet 30
EPDN 44
FDGAN 14
AinNet 29
Restormer 68
PromptI (Ours)
PSNR
22.46
22.06
20.29
22.57 0.863
23.15
23.18
30.87
31.31
SSIM
0.851
0.908
0.877
0.921
0.900
0.969
0.973
Table 3: Deraining results in the single-task setting on Rain100L [16] algorithm, the proposed method yields 2.13 dB PSNR improvement.
Compared to the AirNet 29
Method DIDMDN
UMR[66] SIRR(63
MSPFN(24]
LPNet[17]
AirNet 29
Restormer (88
PromptIR
PSNR
23.79
32.39
32.37
33.50
33.61
34.90
36.74
37.04
SSIM
0.773
0.921
0.926
0.948
0.958
0.977
0.978
0.979
Table 4: Denoising comparisons in the single-task setting on BSD68 [4] and Urban 22 datasets.
For the challenging noise level = 50 on Urban100 [22], our PromptIR obtains 0.51 dB gain compared to AirNet [29].
BSD68
Urban 100 (2
Method
== 15
0 = 25
0 = 50
== 15
d = 25
= 50
CBM3D
11
33,50/0.922
30.69/0.800 0..69/0.66
27.36/0.763
33.93/0.941
31. 51.36/0,909
27.93/0.840
DrCNN 75
33,89/0.930 31.23/0.883
27.92/0.789
32.98/0.931
30.81/0.902
27.59/0.833
IRCNN 76
33.87/0.929
31.18/0.882
27.88/0.790
27.59/0.833
31.20/0.909
27.70/0.840
FFDNet
77
33.87/0.929
31.21/0.882
27.96/0.789
33.83/0.942
31.40/0.912
28.05/0.844
BRDNet
522
34.10/0.92
31.43/0.885
28. 16/0.794
34.42/0.946
31.9910.919
28.56/0.858
AirNet
29
34. 14/0.936
31.48/0.893
28.23/0.806
34.40/0.949
32.10/0.924
28.88/0.87)
PromptlR(Ours)
34.34/0.938
31.71/0.897
28.49/0.813
34.77/0.952
32.49/0.929
29.39/0.881
4.2 Single Degradation One-by-One Results
In this section, we evaluate the performance of our PromptIR under the single-task setting, i.c., a separate model is trained for different restoration tasks. This is show that content-adaptive prompting via prompt block is also useful for tangle-task networks. Table Jableeeeseeentts dehazing results. shows that our PromptiR achieves 8.13 dB improvement over AirNet [29], and 0.44 dB gain over the baseline method Restormer [68]. Similar trends can be observed for deraining and denoising tasks. For instance, when compared to the AirNet (29), our method yields performance gains of 2.13 dB on the deraining task (Table 3 and 0.51 dB on denoising task for noise level a=50 on Urban (0
dataset 22
(see Table
4.3 Ablations Studies
We perform several ablation experiments demonstrate that contributions in PromptIR framework provides quality improvements.
Impact of PGM. We carry out this ablation experiment on Rain100L (65] for deraining task. Table 5 shows that the prompt block in our PromptIR network brings performance gains of 0.3 dB over the baseline (68) Further, it demonstrates that generating dynamic prompts conditioned on input content via PGM provides favorable gain of 0.19 dB over the fixed prompt components.
Position of prompt blocks. In the hicrarchical architecture of our PromptIR, we analyze where to place prompt blocks on the decoder side. T shows that using only one prompt block in the latent space degrades the network's 's performance. Whereas, incorporating prompt blocks between every consecutive level of the decoder performs the best.
Generalization to unseen degradation level. We take the model that is trained only on the noise levels (15, 25, 50} and test its performance the unseen noise level of a 100.
Table 7 shows that our PromptIR demonstrates significantly superior generalization capabilities compared to AirNet 229]. yicldingn ~ dB PSNR difference.

---

Table Impact of PGM. Results are reported on Rain 100L 655 dataset.
Table 6: Prompt blocks position. Results are
reported on Rain 100L [65] dataset.
Method
PSNR
Model
PSNR
Baseline [68]
36.74
level 4 (latent)
36.76
Fixed- Fixed-Prompt
36.85
levels 4+3
36.84
Jrnamic-Promp.
37.04
levels 4+3+2
37.04
Table 7: Denoising comparisons on unseen noise level of o 100.
Table 8: Evaluation on Spatially Variant
Degradation BSD68 [4] test set.
BSD68
41
Urban 100
22
Model
PSNR
Method
PSNR
PSNR
Aireet (2
31.42
Airnet
13.64
13.72
PromptIR (Ours)
31.65
PromptR (Ours)
21.03
20.50
Table Performance of the proposed PromptIR framework, when trained on different combinations of degradation types (tasks) i.e., removal of noise, rain and haze.
Degradation
Denoising on BSD68 dataset [4]
Deraining on
Dehazing on SOTS [3]
Noise
Rain
Haze
=
15
25
50
Rain [16
X
X
34.34/0.938
31.71/0.898
28.49/0.813
X
V
X
37.03/0.9786
X
V
31.31/0.929
V
x
34.26/0.937
31.61/0.895
28.37/0.810
39.32/0.986
V
X
V
33.69/0.928
31,03/0.880
27.74/0.777
30.09/0.975
V
V
35.12/0.969
30.36/0.973
V
V
33.98/0.933 31.31/0.888 28.06/0.799
36.37/0.972
30.58/0.974
Performance on spatially variant degradation. Here we evaluate PromptIR performance on images that are corrupted with varying degradations. For this, we follow closely the work of AirNet (29 and prepare a new test set from BSD68 [41] by applying Gaussian noise of varying levels [0, 15, 25, 50] at different spatial locations of the images. Results in Table @ show that our PromptIR framework more effective restoring these images than AirNet [29], providing 0.23 improvement.
Training model with different combinations of degradation. In Table D we report the results of training an all-in-one model on combined datasets from all three restoration tasks. Here, we evaluate the impact on PromptIR performance by different combinations of degradation types (tasks). Table 9 shows that with an increasing number of degradation types, it becomes increasingly difficult for the network to restore images, thereby leading a performance drop.
Specifically, the presence of hazy images in the combined dataset seems to negatively affect the model. Interestingly, model trained on the combination of rainy and noisy images achieves good performance, indicating a positive correlation between the deraining and denoising tasks. Such phenomenon is also observed in the AirNet work [29].
Impact of Prompt Block. To effectively evaluate the benefit of the proposed framework, we evaluate it against the Restormer model [68]. Restormer is scaled up match the parameter count of PromptIR. We perform this evaluation under the All-in-one setting. As shown in Table 10 the PromptIR framework provides average improvement of 0.38 dB over the Restormer model [68].
Number prompt components. We conduct an abalation study to understand effect of number of prompt components P. employed on the performance of the framework. As shown Table TT we find that utilizing more than components yields marginal improvements while incurring additional computational overhead. As result, choose to utilize five components

---

Table 10: Comparison of Restormer 68
against the proposed PromptIR framework.
Method
Dehazing on SOTS 33
Deraining
Denoising on BSD68 dataset
Average
on Rain 100.
16
15
50
Restormer 68
29.92/0.9703
35.56/0.9691
33.86/0.9327
31.20/0.8878
27. 37.90/0.7944
31.68/0.910
PromptIR (Ours)
30.58/0.974
36.37/0.972
33.98/0.933
31.31/0.888
28.06/0.799
32.06/0.913
Table 11: Evaluation on the number of Prompt Components on Rain 1001
Number of Prompt Components
6
7
PSNR
36.91
37.04
37.06
37.07
Conclusion
Existing image restoration models based on deep neural networks can work for specific degradation types and do not generalize well to other degradations. However, practical settings demand the ability to handle multiple degradation types with single unified model without resorting degradation- specific models that lack generalization and require apriori knowledge of specific degradation in the input. To this end, our work proposed drop-in prompt block that can interact with the input features to dynamically adjust the representations such that the restoration process is adapted for the relevant degradation. We demonstrated the utility of prompt block for all-in-one image restoration by integrating within a So'TA restoration model that leads to significant improvements on image denoising, deraining, and dehazing tasks. In the future, we will extend the model for broader of corruptions toward the goal of universal models for better generalization in image restoration tasks.
Acknowledgement
The computational resources were provided by the National Academic Infrastructure for Super- computing in Sweden (NAISS), partially funded by the Swedish Research Council through grant agreement no. 2022-06725, and by Berzelius resource, provided by the Knut and Alice Wallenberg Foundation at the National Supercomputer Centre.
References
[1] Arbelaez, P., M. Maire, C. Fowlkes. and J. Malik (2011). Contour detection and hierarchical image segmentation. TPAMI.
[2] Ba, J. J. R. Kiros, and G. E. Hinton (2016). Layer normalization. arXiv:1607.06455
[3] Brown, B., B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neolakantan, Shyam, G. Sastry,
A. Askell, et al. (2020). Language models are few-shot learners. arXiv:2005. 14166
[4] Cai, B., X. Xu, K. Jia, C. Qing, and D. Tao (2016). Dehazenet: An end-to-end system for single image haze removal. IEEE Transactions on Image Processing 25(1 5187-5198.
[5] Carion, N., F. Massa, Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko (2020). End-to-end object detection with transformers. In ECCV.
[6] Chen, H. Wang, Guo, C. Xu, Deng. Z. Liu, S. Ma. C. Xu, Xu, and W. Gao (2021a). Pre-trained image processing transformer. In CVPR, pp. 12299-12310.
[7] Chen, H., Y. Wang, Guo, C. Xu, Y. Deng. 7. Liu, S. Ma, C. Xu, C. Xu, and W. Gao (2021b). Pre-trained image processing transformer. In CVPR.
[8] Chen, L., X. Chu, X. Zhang, and J. Sun (2022). Simple baselines for image restoration. In ECCV.
[9] Chen, X., X. Wang, J. Zhou, and C. Dong. Activating more pixels in image super-resolution transformer. arxiv 2022. arXiv preprini 22X.:2205004333
Sorkine-Hornung, [10] Cornillere, V., A. Djelouah, W. Yifan, O. Sorkine- Hornung, and C. Schroers (2019). Blind image super- resolution with spatially variant degradations. ACM Transactions Graphics (TOG) 38(6). 1-13.
10

---

[11] Dabov, K., A. Foi, V. Katkovnik. and K. Egiazarian (2007), Color image denoising via sparse 3d collaborative filtering with grouping constraint in luninance-chrominance space. In 2007 IEEE International Conference on Image Processing, Volume 1, pp. 1-313, IEEE.
[12] Dong, H., J. Pan, L. Xiang, 7. Hu, X. Zhang, F. Wang, and M.-H. Yang (2020). Multi-scale boosted dehazing network with dense feature fusion. In CVPR.
[13] Dong, W., L. Zhang, G. Shi, and X. Wu (2011), Image deblurring and super-resolution by adaptive sparse domain selection and adaptive regularization. TIP.
[14] Dong, Y. Y. Liu, H. Zhang, S. Chen, and Y. Qiao (2020). Fd-gan: Generative adversarial networks with fusion-discriminann for single image dehazing. In Proceedings of the AAAI Conference Artificial Intelligence, Volume 34, pp. 10729-10736.
[15] Dosovitskiy, A., L. Beyer. A. Kolesnikov. D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Heigold. S. Gelly, al. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.
[16] Fan, Q.. D. Chen, L. Yuan, G. Hua, N. Yu, and B. Chen (2019). A general decoupled learning framework for paramcterized image operators. IEEE transactions on pattern analysis and machine intelligence 43(1), 33-47.
[17] Gao, H., X. Tao, X. Shen, and J. Jia (2019). Dynamic scene deblurring with parameter selective sharing and nested skip connections. CVPR, pp. 3848 3856.
[18] Gao, Y. X. Shi, Y. Zhu, H. Wang, Z. Tang, X. Zhou, M. Li, and D. N. Metaxas (2022). Visual prompt tuning for test-time domain adaptation. arXiv preprint arXiv:2210.04831.
[19] He, K., J. Sun, and X. Tang (2010). Single image haze removal using dark channel prior. TPAMI.
[20] He, S. Zheng, Y. Tay, J. Gupta, Y. Du, V. Aribandi, Z. Zhao, Y. Li, Z. Chen, D. Metzler, et al. (2022). Hyperprompt: Prompt- Prompt-based based task conditioning of transformers. In ICML, pp. 8678 8690, PMLR.
[21] Houlsby, N., A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly (2019). Parameter-efficient transfer leamming for nip. In ICML, pp. 2790-2799. PMLR.
[22] Huang, J.BB., Singh, and N. Ahuja (2015). Single image super-resolution from transformed self- exemplars. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5197- 5206.
[23] Jia, M., L. Tang, B.-C, Chen, C. Cardie, S. Belongie, B. Hariharan, and S.-N. Lim (2022). Visual prompt tuning. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIII, pp. 709-727. Springer.
[24] Jiang, K., Z. Wang. P. Yi, C. Chen, B. Huang, Y. Luo, J. Ma, and J. Jiang (2020). Multi-scale progressive fusion network for single image deraining. In CVPR, pp. 8346-8355.
[25] Khan, S., M. Nascer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah (2022). Transformers in vision: survey. ACM computing surveys (CSUR) 54(10s), 1-41.
[26] Khattak, M. U., H. Rasheed, M. Maaz, Khan, and F, S. Khan (2023). Maple: Multi-modal prompt learning. CVPR.
[27] Kim, K. I. and Kwon (2010), Single-image super-resolution using sparse regression and natural image prior. TPAMI.
[28] Kopf, B. Neubert, B. Chen, M. Cohen, D. Cohen-Or, O. Deussen, M. Uyttendacle, and D. Lischinski (2008). Deep photo: Model-based photograph enhancement and viewing. ACM TOG.
[29] Li. B., X. Liu, P. Hu, Z. Wu, J. Lv, and X. Peng (2022, June). All-in-one image restoration for unknown corruption. In CVPR, pp. 17452-17466
[30] Li, B., X. Peng, Z. Wang, J. Xu, and D. Feng (2017). Aod-net: All-in-one dchazing network. In ICCV, pp. 4770-4776
[31] Li, B., W. Ren, D. Fu, D. Tao, D. Feng, W. Zeng, and Z. Wang (2018). Benchmarking single-image dehazing and beyond. IEEE Transactions Image Processing 28(1), 492-505.
[32] Li. R.. R. T. Tan, and L.-F. Cheong (2020), All in one bad weather removal using architectural search. In CVPR, pp. 3185
11

---

[33] Li. W., X. Lu, J. Lu, X. Zhang, and J. Jia (2021). On efficient transformer and image pre-training for low-level vision. arXiv preprint 10115
[34] Xi, L. and P. Liang (2021). Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.01190
[35] Liang, J. J. Cao. G. Sun, K. Zhang, L. Van Gool, and R. Timofte (2021). SwinIR: Image restoration using swin transformer. In ICCV Workshops.
[36] Liu, J., H. Wu, Y. Xie, Y. Qu, and L. Ma (2020). Trident dehazing network. In CVPR Workshops.
[37] Liu, L.,, L.. Xie, X. Zhang, Yuan, Chen, W. Zhou, H. Li, and Q. Tian (2022). Tape: Task-agnostic prior embedding for image restoration. In Computer Vision-ECC 2022: 17th European Conference, Tel Aviv Israel, October 23-27, 2022, Proceedings, Part XVIII. pp. 447-464. Springer.
[38] Liu, Z., Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo (2021). Swin transformer: Hicrarchical vision transformer using shifted windows. arXiv:2103,
[39] Luo, Z., Y. Huang, S. Li, L. Wang, and T. Tan (2022). Learning the degradation distribution for blind image super-resolution. In CVPR, pp. 6063-6072.
[40] Ma, K., Z. Duanmu, Q. Wu, Wang, H. Yong, H. Li, and Zhang (2016). Waterloo exploration database: New challenges for image quality assessment models. TIP.
[41] Martin, D., C. Fowlkes, D. Tal, and Malik (2001). A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ccological statistics. In ICCV,
[42] Michaeli, T. and M. Irani (2013). Nonparametric blind super-resolution. In ICCV.
[43] Nah, S., S. Son, J. Lee, and K. M. Lee (2022). Clean images are hard to reblur: Exploiting the ill-posed inverse task for dynamic scene dehlurring. In ICLR.
[44] Qu, Y. Y. Chen, J. Huang, and Y. Xie (2019). Enhanced pix2pix dehazing network. In CVPR, pp. 8160-8168.
[45] Ren. C., X. He. C. Wang, and Z. Zhao (2021), Adaptive consistency prior based deep network for image denoising. In CVPR.
[46] Ren, W., S. Liu, H. Zhang, J. Pan, X. Cao, and M.-H. Yang (2016). Single image dehazing via multi-scale convolutional neural networks. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October l 11-4,, 2016, Proceedings, Part II 14, pp. 154-169. Springer.
[47] Ren, W., J. Pan, H. Zhang, X. Cao, and M.-HI Yang (2020). Single image dehazing via multi-scale convolutional neural networks with holistic edges. IJCV.
[48] Sanh, V., A. Webson, C. Raffel, S. H. Bach, L. Sutawika, Z. Alyafeai, A. Chaffin. A. Stiegler, T. L. Scao, A. Raja, al. (2021). Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:21 10.08207.
[49] Smith, J. S., I.. Karlinsky, V. Gutta, P. Cascante-Bonilla, D. Kim, A. Arbelle, R. Panda, R. Feris, and 7. Kira (2022). Coda-prompt: Continual decomposed attention-hased prompting for rehearsal-free continual learning. arXiv preprint 1.Xiv:2211..3/166
[50] Sohn, K., Y. Hao, J. Lezama, L. Polania, H. Chang, H. Zhang, I. Essa, and L. Jiang (2022). Visual prompt tuning for generative transfer learning. arXiv preprint arXiv:2210.00990.
[S1] Song, Y.. Z. He. H. Qian, and X. Du (2023). Vision transformers for single image dehazing. IEEE Transactions on Image Processing 32, 1927-1941.
[52] Tian, C., Y. Xu, and W. Zuo (2020). Image denoising using deep enn with batch renormalization. Neural Networks.
[53] Timofte, R., V. De Smet, and L. Van Gool (2013). Anchored neighborhood regression for fast example- based super-resolution. In ICCV.
[54] Touvron, H., M. Cord. M. Douze, F. Massa, A. Sablayrolles, and H. Jégou (2021). Training data-efficient image transformers & distillation through attention. In ICML.
[55] Tsai, F.J., Y.T. Peng, C.-C Tsai, Y.Y. Lin, and C.-W. Lin (2022). BANet: A blur-aware attention network for dynamic seene deblurring, IEEE Transactions on Image Processing.
12

---

[56] Tu, Z., H. Talebi, H. Zhang, F. Yang. P. Milanfar, A. Bovik, and Y. Li (2022). MAXIM: Multi-axis MLP for image processing. In CVPR, pp. 5769-5780.
[57] Valanarasu, J. M. J.. R. Yasarla, and V. M. Patel (2022). Transweather: Transformer-asedd restoration of images degraded by adverse weather conditions. In CVPR, pp. 2353 2363.
[58] Vaswani, A., N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, t. Kaiser, and Polosukhin (2017). Attention is all you need. In NewrIPS.
[59] Wang, Z., X. Cun, J. Bao, and J. Liu (2021). Uformer: A general u u-shaped transformer for image restoration. 1.X.v:2106.331
[60] Wang. Z., R. Panda, L. Karlinsky, R. Feris, H. Sun, and Y. Kim (2023). Multitask prompt tuning enables parameter-efficient transfer learning, arXiv preprint arXiv:2303.022 02861.
[61] Wang, Z., Z. Zhang, S. Ebrahimi, R. Sun, H. Zhang, C.-Y. Lee, X. Ren, G. Su, V. Perot, J. Dy, al. (2022). Dualprompt: Complementary prompting for rehearsal-fre free continual learning. In Computer Vision-ECCV ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23 27, 2022, Proceedings, Part XXVI. pp. 648. Springer.
[62] Wang, Z., Z. Zhang, C.-Y. Lee, H. Zhang, R. Sun, X. Ren, G. Su, V. Perot, J. Dy, and T. Pfister (2022). Learning to prompt for continual learning. In CVPR, pp. 139-149.
[63] Wei, W., D. Meng, Q Zhao, Z. Xu, and Y. Wu (2019). Semi-supervised transfer learning for image rain removal. In CVPR.
[64] Xie, E., W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo (2021). Segformer: Simple and efficient design for semantic segmentation with transformers. arXiv:2105. 152003.
[65] Yang, F., H. Yang, J. Fu, H. Lu, and B. Guo (2020). Learning texture transformer network for image resortresolution. In CVPR.
[66] Yasarla, R. and V. M. Patel (2019). Uncertainty guided multi-scale residual learning-using a cycle spinning cnn for single image de-raining. In CVPR.
[67] Yuan, L., Y. Chen, T. Wang. W. Yu, Y. Shi, Z. Jiang, F. E. Tay, J. Feng. and S. Yan (2021). Tokens-to-token vit: Training vision transformers from scratch on imagenet. arXiv:2101.11980
[68] Zamir, S. W., A. Arora, S. Khan, M. Hayat, F. S. Khan, and M.-H. Yang (2022). Restormer: Efficient transformer for high-resolution image restoration. In CVPR.
[69] Zamir, S. W., A. Arora, S. Khan, M. Hayat, F. S. Khan, M.-H. Yang, and I. Shao (2020a). CycleISP: Real image restoration via improved data synthesis. In CVPR.
[70] Zamir, S. W., A. Arora, S. Khan, M. Hayat, F. S. Khan, M.-H. Yang, and L. Shao (2020b). Learning enriched features for real image restoration and enhancement. In ECCV.
[71] Zamir, S. W., A. Arora, S. Khan, Hayat, F. S. Khan, M.-H. Yang, and L. Shao (2021). Multi-stage progressive image restoration. In CVPR.
[72] Zhang, H. and V. M. Patel (2018). Density-aware single image de-raining using a multi-stream dense network. In CVPR.
[73] Zhang, K. J. Liang, L. Van Gool, and R. Timofte (2021). Designing a practical degradation model for deep blind image super-resolution. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 4791-4800.
[74] Zhang, K., W. Luo, Y. Zhong, L. Ma, B. Stenger, W. Liu, and H. Li (2020). Deblurring by realistic blurring. In CVPR, pp. 2737-2746.
[75] Zhang, K., W. Zuo, Y. Chen, D. Meng, and Zhang (2017). Beyond a gaussian denoiser: Residual learning of deep can for image denoising. IEEE transactions on image processing 26(7), 3142-3144-315
[76] Zhang, K., W. Zuo, S. Gu, and L. Zhang (2017). Learning deep CNN denoiser prior for image restoration. In CVPR.
[77] Zhang, K., W. Zuo, and L. Zhang (2018). Fidnet: Toward fast and flexible solution for enn- an-based image denoising, IEEE Transactions on Image Processing 27(9), 4608 4622.
13

---

[78] Zheng, S., J. Lu, H. Zhao, X. Zhu, Z. Luo. Y. Wang. Y. Fu, J. Feng. T. Xiang, P. H. Torr et al. (2021). Rethinking semantic segmentation from sequennce-o-sequeeeee perspective with transformers. In CVPR.
[79] Zhou, K., J. Yang, C. C. Loy, and Z. Liu (2022). Learning to prompt for vision-language models. International Journal of Computer Vision (ICCV).
[80] Zhu, X., W. Su, L. Lu, B. Li, X. Wang, and J. Dai (2020). Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:0010.0455.
14

---

Appendix
A Additional Ablation Studies
We conduct further ablation studies to illustrate the effectiveness of various design choices of the PromptIR framework. We examine various key design choices like the usage of prompt tokens and plugging in prompt blocks only on the decoder branch of the network.
A.1 Contrastive learning- based Degradation Encoder embedding v/s Prompt Tokens
To strengthen the design rationale for incorporating prompts instead of following recent methods 29 that use embeddings learned through contrastive training. we replace the generated prompt from our PGM module with embeddings extracted from the Contrastive- learning based Degradation Encoder of the AirNet 29 model. We observed that the use of contrastive emheddings resulted in significantly weaker performance compared to prompt tokens. Moreover, achieving good performance with contrastive embeddings requires a desstnddeeigned restoration network, whereas our Prompt Blocks can be seamlessly integrated as plug-and-pla, modules into any restoration network.
Table A. Comparisons under all-in-one setting: between the usage of degradation embedding extracted from the Contrastive-larning Based Degradation Encoder (CBDE) of the Airnet [29 Model and the usage of prompt tokens in the PromptI framework.
Method
Dehazing on SOTS 31
Deraining on Rain [6
Denoising on BSD68 dataset (4T)
Average
a = 15
a 25
0
50
CBDE+PrompuR
23.92/0.881
32.03/0.0.972
32.96/0.910
30.36/0.860
26.93/0.732
29.24/0.875
PromptR (Ours)
30.58/0.974
36.37/0.972
33.98/0.933
31.31/0.888
28.06/0.799
32.06/0.913
A.2 Prompt Blocks on both Encoder branch and Decoder branch
We study the importance of decoder-only prompting by evaluating the usage of prompt blocks on both the encoder and decoder branches. We show that it is important the prompt block is only used on the decoder side.
Table A.2: Comparisons under the all-in-one setting: between the usage of the frompt-block on both the encoder branch and encoder branch with using the prompt block only the decoder branch. Method
Dehazing on SOTS [1
Deraining
Denoising on BSD68 dataset 4T)
Average
on Rain 1001. [6
Enc+Dec+PrompiR 28.52/0.927
35.43/0.965
33.59/0.927
30.85/0.878
27.35/0.732
31.14/0.885
PromptIR (Ours)
30.5%/0.974
36.37/0.972
33.98/0.933
31.31/0.888
28.06/0.799
32.06/0.913
B Transformer Block in PromptIR Framework
As mentioned in section 3.1.2 of the main manuscript, we present the block diagran B.1 of the transformer block and further, elaborate on the details of the transformer block employed in the PromptIR framework. The transformer block follows the design and syper-parameters outlined in (68)
To begin, the input features X E R XXXx are passed through the MDTA module. In this module, the features are initially normalized using Layer normalization. Subsequently, combination of 1 x convolutions followed by 3 x 3 depth-wise convolutions are applied to project the features into Query (Q), Key (K), and Value (V) tensors. An essential characteristic of the MDTA module is its computation of attention across the channel dimensions, rather than the spatial dimensions. This effectively reduces the computational overhead. To achieve this channel-wise attention calculation, the Q and K projections are reshaped from Hi x W, C₁ to H,W, x C₁ and C x H;W: respectively,
15

---

MDTA
Mi Miltt-Bron heed Tra reposed Attention
GDFN
Gated Duani Fine- Forward Network
Doonu
Norm
Normaltztion
Z
Z
x 3
3x3 depth-wise convolution(I
8, Mix
3x3
(ormelution((()))
operation
MDTA
3x3 DEOnV
K.G.
Bement-wise addition
X
X
Traropooed
Elementwise multiplicaiion
Attention Map (4)
Transformer Block
Nerr
3*3
K
G X.G.C
Multipization
Reshape
3x3 Doonv
R x Wix
GeLU Activetion
Figure B.I: Overview of the Transformer block used in the PromptI framework. The Transformer block composed of sub modules,the Multi Deonv head transposed attention modulc(MDTAA) and the Gated Dconv feed-forward (GDFN.
before calculating dot-product, hence the resulting transposed attention map with the shape C₁ x C₁- Bias-free convolutions utilized within this submodule. Furthermore, attention is computed in multi-head manner in parallel.
After MDTA Module the features are processed through the GDFN module. In the GDFN module, the input features are expanded by a factor yy using convolution and they are then passed through 3 x 3 convolutions. These operations are performed through two parallel paths and the output of one of the paths is activated using GeLU non-linearity. This activated feature map is then combined with the output of the other path using element-wise product.
16

---

C Qualitative results:
We present more qualitative results from single-task models further elucidate the effectiveness of prompt-block even when under the single-task setting.
C.1 Dehazing
Inpul
AicNet
PronptiR
(Ours)
Gronad Truth
Figure C.I: Image deraining comparisons under single task setting on images from the SOTS
dataset 31.
Our method effectively removes haze produce visually better images.
17

---

C.2 Deraining
Imput
AirNet
PromptiR
(Ours)
Ground Truth
Figure C.2: Image deraining comparisons under single task setting on images from the Rain 100L
dataset [6]. Our method effectively removes rain streaks to generate rain-free images.
18

---

C.3 Denoising
Input Image
Noisy Image
Ground truth
et
PromptR(Ours)
Figure C.3: Image deraining comparisons under single task setting images from the URBAN 100 dataset 22 with = 50. Our method produces visually better images as compared to previous methods. We show selected patches from the images.
19


Reviews:
[{"review_id": "IkMg3N5SUL", "reviewer": "Reviewer_Jp7P", "summary": "The paper proposes prompt based image restoration method that can handle different degradations. the proposed method uses prompts to encode degradation-specific information, which is then used to dynamically guide the restoration network. Given any instance, prompt generation module (PGM) and a prompt interaction module (PIM) are used generate input-conditioned prompts (via PGM) that are equipped with useful contextual information to guide the estoration network (with PIM) to effectively remove the corruption.", "questions": "Please refer weaknesses", "limitations": "Limitations of the proposed method were not discussed", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Paper is well organized \n- PGM, and PIM modules are used to generate the prompts and recover the image. Proposed method is clearly explained \n", "weaknesses": "- Can the proposed method method handle degradations like snow, night, blur, and turbulence conditions, since the proposed method mentions it is generic image restoration.\n- experiments are limited, since the experiments showed in the paper covered only haze, rain, and noise, specifically these the datasets used for comparison are synthetic and very small datasets. it is hard understand the limits of the proposed method from these experiments\n- Additionally the experiments performed contains only single degradation at any instance, can the proposed method handle multiple degradations in the single input image\n- Can authors performs the experiments showing if the wrong prompt was propagated (instead of the prompt from the PGM) to understand the importance of PGM module.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "`R1.1`:*Other Degradations*:\n\nWe opted for deraining, dehazing and denoising to demonstrate the effectiveness of our approach on a diverse range of low-level vision tasks. However, as per the reviewer suggestions, we retrained our PromptIR network on five distinct tasks, including two new tasks i.e, low-light enhancement and deblurring. We followed the recently published work [1] and supplemented their table with our obtained results. Table R1 shows that our method yields favorable performance. (Please note that these results are from our initial attempt in a limited time, which can further be improved with more careful choices of hyperparameters.)\n\n| Method      | Deraining | Dehazing | Denoising | Deblurring | Low-Light | Average PSNR | Param | FLOPs (G) | Runtime (ms) |\n|-------------|-----------|----------|-----------|------------|-----------|--------------|---------|-----------|--------------|\n| DL          | 21.96     | 20.54    | 23.09     | 19.86      | 19.83     | 21.05   | 2.09M    | -         | -            |\n| Transweather| 29.43     | 21.38    | 29.00     | 25.12      | 21.21     | 25.22  | 37.93M  |  4.684     | 14.0         |\n| Airnet      | 32.98     | 21.04    | 30.91     | 24.35      | 18.18     | 25.49  | 8.93M   |  311       | 141.0      |\n| Restormer   | 34.81     | 24.09    | 31.49     | 27.22      | 20.41     | 27.60  | 26.13M  |  155       | 87.9       |\n| IDR         | 35.63     | 25.24    | 31.60     | 27.87      | 21.34     | 28.34  | 15.34M  | -         | -            |\n| PromptIR    | 36.09     | 30.60    | 31.26     | 27.47      | 23.03     | 29.46 | 35.6M   |  173       | 95.5       |\n\n**Table R1**: FLOPs and Runtimes are computed on 256x256 images.\n\n\n`R1.2`: *Experiments*:\n\nIn this paper, we adhered to the experimental protocols, datasets, and\ntasks outlined by the previous best method AirNet (CVPR’22). It was\nto ensure fair comparisons as well as avoid retraining all previous works.\nNonetheless, it is important to highlight that the increase in dataset scale\nwill likely provide improved results, which is an expected outcome of deep\nlearning techniques. In this response, we have further provided additional\nresults as suggested by reviewers which demonstrate the scalability and\nefficacy of PromptIR (Table R1).\n\n\n\n`R1.3`: *Multiple degradations in single image*:\n\nThe proposed PromptIR is all-in-one method that can recover images with-\nout prior knowledge of input degradation type. It is important to clarify\nthat, as in AirNet, “multiple-degradation” indicates that an individual image contains only a single distinct type of degradation, while the dataset will encompass multiple such degradations. In order to handle multiple\ndegradations in the single input image, our model and other competitors\nwill need to be trained in a similar setting (i.e., multiple degradations per\nimage) which we are currently running, and we can include a comparison\nin the final version. In order to cater reviewer’s suggestion, we perform the\nfollowing experiment without any training on multiple degradations per\nimage. We add Gaussian noise to Rain100L images to create rainy+noisy\ndataset. Next, we take the pretrained AirNet and our PromptIR and di-\nrectly evaluate on this dataset. Results in Table R2 demonstrates that\nour algorithm fares better than the AirNet method.\n\n\n| Method   | Rain + σ = 15 | Rain + σ = 25 | Rain + σ = 50 |\n|----------|--------------|--------------|--------------|\n| Airnet   | 18.45        | 17.30        | 14.65        |\n| PromptIR | 18.85        | 17.88        | 15.76        |\n \n \n  \n\n\n**Table R2**: Evaluation on single-image corrupted with multiple degradations (Rain + Noise).\n\n\n`R1.4`: *Experiment with wrong prompt inputs*:\n\nIn ablation study (Table 5 main paper), we show the PGM contributes\nfavourably towards the final performance of our method. As per suggestion,\nwe added random noise to perturb the generated prompts at inference time\nand there is a significant drop in PSNR from 37.04 dB to 36.89 dB.\n\n\n[1] Zhang et al., Ingredient-oriented Multi-Degradation Learning for Image Restoration, CVPR 2023"}, {"review_id": "XbKXTe5lM5", "reviewer": "Reviewer_3Jtv", "summary": "This paper restores images from various types and levels of degradation. In detail, degradation information is encoded as prompts and used to guidance the restoration. Based on the classical UNet architecture for restoration, there are two added blocks: prompt generation module and prompt interaction module. With a single network, it achieves state-of-the-art performance on image denoising, deraining and dehazing.", "questions": "See weakness", "limitations": "See weakness", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1, A prompting-based single-model blind restoration framework.\n\n2, A plug-in module with prompt generation and prompt interaction, which guide the backbone model to remove corruptions effectively.\n\n3, State-of-the-art performance on multiple tasks with one single model", "weaknesses": "1, The ablation studies cannot prove this basic concern of the main contribution and intuition in this paper. Is the prompt (5 learned tensors as a dictionary) really working? What have the prompts learned? Do we really need the prompts? Since there are only 6 different types of degradations, I guess $softmax(Conv_{1\\times 1}(GAP(F_1)))$ learns a classification model to decide which degradation the input image has. If we directly inject this information to the UNet, it should work as well. This paper still follows the pipeline of \"degradation estimation + degradation-aware restoration\".\n\n2, Missing comparison on model size, runtime, FLOPs, etc. According to the model file sizes, I suspect there exists unfair comparison. The model size of PromptIR (file size: 388MB) is about 4 times of Restormer (file size: 99.9MB) and 11 times of Airnet (file size: 35.6MB). This means that the model might use 100MB for each sub-task and the rest 88MB for estimating the degradation type.\n\n3, For the single-task setting, for example, image denoising with sigma=50, what's the meaning of using prompt generation and interaction?\n\n4, Missing ablation studies on prompt design, such as number of prompts.\n\n5, Why adding prompt blocks at level 4 is nearly of no use (36.76 v.s. 36.74dB)? Note that it has more parameters in the prompt generation module and prompt interaction module. Besides, how about using level 2 only?\n\n6, In table 7 and 8, why PromptIR is significantly better than Airnet on unseen noise level, but is only slightly better on spatially variant degradation?\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "`R2.1`: *Ablation study (directly incorporating softmax outputs):*\n\nTo show the benefit and contribution coming from only prompts in our design, we perform an ablation experiment in which we adjust the baseline Restormer's capacity such that the params/FLOPs become similar to our PromptIR framework. This ensures that the performance gains are obtained due to the novel architecture design (prompt components) rather than the extra parameters. This baseline network yields PSNR of 36.82 dB, whereas our PromptIR provides 37.04 dB. This shows the contribution of prompts towards generic image restoration task. \n \nSecondly, PromptIR does not estimate degradations **explicitly**. Instead, it implicitly calculates these degradations, which subsequently help the main network in restoring images.\n As suggested, we remove prompt components and instead directly incorporate the information after the softmax into the main architecture. This newly trained network achieves 36.79 dB PSNR, lower by 0.25 dB compared to PromptIR's performance.\n\n`R2.2`: *Computational Comparison*:\n\nOur PromptIR is an all-in-one network, and does not hold separate parameters for separate tasks, i.e., incorporating additional tasks into the pipeline would not lead to increase in parameters. The public checkpoints\nfor Restormer and AirNet do not include the optimizer state dictionary,\nresulting in smaller file sizes. Similarly, if memory conservation is a priority, one can opt to exclude the optimizer dictionary from PromptIR. For computational comparisons, please refer to Table R1.\n\n`R2.3`: *Prompting for single task*:\n\nWe followed AirNet to further include single-task experiments in the paper. In our tests, we noticed that the use of prompts also contributes\nto the improvement of single-task restoration. These prompts provide\ncomplementary information to the main network, enabling it to better\ndistinguish various elements such as rain patterns and streaks of different\norientation and magnitude in the context of deraining.\n\n`R2.4`: *Ablation on prompt components*:\n\nAs suggested, we conduct an ablation study on the number of prompt\ncomponents, observing that employing more than 5 components yields\nmarginal improvements while incurring additional computational overhead. As a result, we choose to utilize five components, a decision supported by our findings. We will include this ablation Table R3 in the\nrevised paper.\n\n| # of prompt components | 3     | 5     | 6     | 7     |\n|-----------------------|-------|-------|-------|-------|\n| PSNR                  | 36.91 | 37.04 | 37.06 | 37.07 |\n\n**Table R3**:  Ablation study on the number of prompt components.\n\n`R2.5`: *Using prompts on only level 2 or level 4*:\n\nWhen employing the prompt block solely at level 2, the PSNR drops from 37.04 dB to 36.82 dB. We tried several combinations for placing prompt block, and empirically found that placing prompt blocks between every consecutive level of the decoder performs the best. \nAlthough the improvement only with level 4 is not much, keeping the block at this level is important since it feeds to later restoration stages and allows overall better performance working in tandem with other levels.\n\n`R2.6`: *Spatially variant degradation*:\n\nTo truly test the generalization capability of our PromptIR, we reported results on unseen noise level of 100 (Table 7), where our method shows a significant gain of ~7 dB over AirNet. Whereas in Table 8, the noise levels used in the spatially variant degradation experiment are seen by both AirNet and our PromptIR during their training phases. As a result, the performance gap between the two methods was narrower. Here, we perform a new spatially variant degradation experiment by including both seen and unseen noise levels $\\sigma$=[0, 50, 75, 100]. Our method yields a significant PSNR gain of 5 dB (i.e., 22.34 dB ours vs 17.35 dB Airnet).\n\n| Method   |  PSNR  | SSIM   |\n|----------|--------|--------|\n| Airnet   | 17.35  | 0.5420 |\n| PromptIR | 22.34  | 0.6137 |\n\n**Table R4**: Additional ablation on spatially-variant degradation.\n\n\n"}, {"review_id": "zr2xyd1zl6", "reviewer": "Reviewer_UJGQ", "summary": "In this paper, authors study to restore images with unknown degradation types with a unified framework. To achieve it, a prompt-based block is proposed which generates suitable prompt first and then interact them with original features. Experiments show that combining the proposed block with restormer network can achieve new SOTA performance.", "questions": "I think applying prompt technology into low-level image restoration task is very interesting, but this work seems suffer from many issues. Some of them are mentioned in the weaknesses. Still, there are some other questions as followed:\n\n1. As mentioned in the paper, the proposed prompt block can be used in various existing networks. However, no experiments are conducted. And the ablation studies in table 6 and table 2 (supplementary) show that different networks may require various block configurations.\n2. Why not use cross attention to interact learnable prompt with features? In [1], it seems work well. Thus, I suggest to compare this two ways in detail (the learnable parameters in [1] can be also regarded as prompt).\n[1] Transweather: Transformer-based restoration of images degraded by adverse weather conditions; \n\n", "limitations": "Authors have not discussed the limitations. In my opinions, one of the potential limitations is the unproved generalization ability in terms of networks (e.g., utilization on AirNet) and data (e.g., real world images).", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Prompt learning is recent a hot method in both language and vision domains. Different from previous works focus on high-level understanding tasks, this paper tempts to explore its application on low-level image restoration task.\n2. The writing is good and easy to understand.", "weaknesses": "1. Typos or confused explanations: \n     a. In Eq. 2, subscript of w should be the same with P in $\\sum_{c=1}^{N} w_{I}P_{c}$? And the subscript of w in the second formula should be removed?\n     b. Line 208, \"Table 1 Similarly on the image Deraining task, \".\n     c. Line 204, the second best approach is not DL in Table 1, is MRPNet. \n     d. Line 231-232, it mentions that using only one prompt block in the latent space (36.76 dB) degrades the performance. However, the baseline is 36.74 dB, which lower than 36.76 dB.\n2. Lack some comparisons. For example, the performance restormer is needed in Table 1. And other all-in-one weather image removing methods[1][2] are still very related to this work and needed to be compared. In addition, all results are in the synthesized images, how about its general performance on real world images (e.g., real rain images).\n3. Ablation study can be further improved. For example, the number of prompt is not studied. And the additional computational cost and model parameters should be discussed. Is the improvement gains come from the additional parameters?\n[1] Transweather: Transformer-based restoration of images degraded by adverse weather conditions;\n[2] Learning Multiple Adverse Weather Removal via Two-Stage Knowledge Learning and Multi-Contrastive Regularization: Toward a Unified Model;", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "`R3.1`: *Typos*:\n\nWe thank the reviewer for pointing out these issues. We will improve clarity and fix typos in the revised manuscript.\n\n`R3.2`: *Additional Comparisons*:\n\nAs suggested, we include Restormer and Transweather in comparisons. Please refer to Table R1. Our PromptIR provides favorable results. \n\n`R3.3`: *Ablation Study*:\n\nPlease refer to Table R3 for ablation on number of blocks, and Table R1 for the computational comparisons. These will be included in the revised manuscript.\nAs suggested, we perform an ablation experiment in which we adjust the baseline Restormer's capacity such that the params/FLOPs become similar to our PromptIR framework. This ensures that the performance gains are obtained due to the novel architecture design rather than the extra parameters. This baseline network yields PSNR of 36.82 dB, whereas our PromptIR provides 37.04 dB.\n\n`R3.4`: *Further Questions*:\n\nMajority of recent image restoration works (Restormer, HINet, NAFNet, MAXIM, MPRNet) employ a hierarchical architecture. In our PromptIR, we also use UNet design and demonstrate how to simply plug PGM and PIM into the main network. This can easily be extended to other networks as well.\nNote that our prompt block is already a kind of cross-attention because the interaction between features and prompts is taking place with a transformer block."}, {"review_id": "dQaigWhgoB", "reviewer": "Reviewer_vXsu", "summary": "This paper proposes an all-in-one image restoration method that can restore images from various types and levels of degradation. Specifically, it utilizes prompts to encode degradation-specific information, which is used to dynamically guide the restoration network. Experimental evaluations on image denoising, deraining, and dehazing are conducted to demonstrate the effectiveness of the proposed method. ", "questions": "1. What do the degradation embeddings used in Figure 2 refer to?\n2. What does the feature F_p in Figure 3 refer to?\n3. The key component of the proposed approach is the prompt block, which takes the prompt-components P_c and input features F_l as input. However, few information about P_c is provided. Please explain in detail how do the authors obtain the prompt-components P_c.\n4. Do the authors retrain all the competed methods using the same training dataset as the proposed method?\n5. For the single task setting, more recent and representative methods should be compared to demonstrate the effectiveness of the proposed method. For example,\n1) for image dehazing, \n      a) Song et al., Vision Transformers for Single Image Dehazing, IEEE TIP, 2023.\n      b) Tu et al., MAXIM: Multi-Axis MLP for Image Processing, CVPR 2022.\n2) for image deraining,\n      c) Chen et al., Pre-Trained Image Processing Transformer, CVPR 2021.\n6. To show the impact of PGM, it would be better to compare with a baseline that replaces all the prompt blocks in Figure 3 with the transformer blocks in Table 5.", "limitations": "This paper does not claim the limitations and potential negative societal impact of their work. It would be better to discuss about the limitations in real applications and potential negative societal impact of their work.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Prompts are used to encode degradation-specific information for all-in-one image restoration.\n\nExperimental evaluations and ablation studies are conducted to demonstrate the superior performance of the proposed method.", "weaknesses": "There are several unclear statements listed as follows.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "`R4.1`: *Degradation embeddings*:\n\nIn Figure 2, we plot the prompt P ,generated through the Prompt Generation Module within the Prompt block. Here, the term ”degradation embedding” is used to refer to the generated prompt P, as it provides\nadditional contextual information regarding the type of degradation observed within a corrupted image.\n\n`R4.2`: *Typo*:\n\nSorry for the typo. It should be $F_l$. We will fix it.\n\n`R4.3`: How prompt-components $P_c$ are obtained?\n\nPrompt components are learnable parameters that are initialized with random weights (for instance using nn.Parameter(*size) in Pytorch). \nSubsequently, these prompts undergo optimization throughout the training process to acquire specialized contextual information related to degradations. At inference time, these prompts guide the model in restoring degraded images. As recommended, we will provide a more detailed explanation of prompt components in the revised paper.\n\n`R4.4`: *Training setting*:\n\nWe follow the experimental protocols and datasets defined in the AirNet paper, and extended their tables with our results to avoid retraining any previous approach.\n\n`R4.5`: *Further comparisons*:\n\nWe thank reviewer for the references. We will cite the work of Song et al., while the others are already credited. Including these referenced methods would require retraining them. Our PromptIR approach is based on the Restormer baseline. Given that Restormer has demonstrated superior outcomes compared to MAXIM and IPT, we are confident that our approach will likely exhibit a similar advantageous trend.\n\n`R4.6`: *Baseline*:\n\nAs per reviewer suggestion, we perform this ablation experiment by replacing prompt blocks with restormer blocks, making the params/FLOPs similar to our PromptIR framework. The new baseline network yields PSNR of 36.82 dB, whereas our PromptIR provides 37.04 dB."}, {"review_id": "AWiBVZUXcA", "reviewer": "Reviewer_4MQM", "summary": "The paper proposes PromptIR, a prompt-based learning approach for all-in-one image restoration. The method utilizes prompts to encode degradation-specific information and dynamically guide the restoration network. It achieves state-of-the-art results on various image restoration tasks.", "questions": "(1) Could you please provide more details on the sensitivity of the results to the number of prompts used and how to choose the optimal number of prompts?\n\n(2) How does the Restorformer perform in the setting of Table 1?\n\n(3) How about the perceptual metric comparison between this method and the baselines?\n", "limitations": "See the weakness and question part.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The proposed method offers a generic and efficient plugin module with a few lightweight prompts that can be used to restore images of various types and levels of degradation with no prior information of corruption.\n\n(2) The method achieves state-of-the-art results in image denoising, deblurring, and dehazing.\n\n(3) The paper is well-written and easy to follow.\n", "weaknesses": "(1) The paper lacks discussions on the limitations of the proposed method, such as the sensitivity of the results to the number of prompts used and how the method performs on combined types of degradation, such as noise + rain.\n\n(2) The paper lacks the visualization of the learned prompt and an explanation of how and why it could work.\n\n(3) The paper lacks comparisons on complexity, including runtime and parameter number.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "`R5.1`: *Ablations on number of prompt blocks and multi-degraded images*:\n\nThe number of prompt components is chosen empirically. Please refer to response 2.4 (Table R3). And for a detailed response on images corrupted with multiple degradations, please see R1.3.\n\n`R5.2`: *Learned prompts*:\n\nWe perform an ablation experiment by replacing prompt blocks with restormer blocks, making the params/FLOPs similar to our PromptIR framework. The new baseline network yields PSNR of 36.82 dB, whereas our PromptIR provides 37.04 dB.\n\n`R5.3`: *Computational comparisons*:\n\nThe parameters, Flops and runtimes are reported in Table R1.\n\n`R5.4`: *Number of prompts*:\n\nWe empirically choose number of prompt components. Please refer to R2.4 for more details\n\n`R5.5`: *Results of Restormer*:\n\nWhen averaged across all three tasks, our method provides 0.38 dB performance gain over Restormer, as shown in Table R5.\n\n| Method    | Dehazing(SOTS) | Deraining(Rain100L)        |     |  Denoising(BSD68)   |      | Average       |\n|------------|-------------------|------------------|----------------------------|------|----|----------------|\n|                 |                           |                         | $\\sigma = 15$          | $\\sigma = 25$ | $\\sigma = 50$ |               |\n| Restormer | 29.92/0.9703      | 35.56/0.9691      | 33.86/0.9327           | 31.20/0.8878 | 27.90/0.7944  | 31.68/0.910   |\n| PromptIR  | **30.58**/**0.974** | **36.37**/**0.972** | **33.98**/**0.933** | **31.31**/**0.888** | **28.06**/**0.799** | **32.06**/**0.913** |\n\n\n**Table R5**: Add Restomer results in Table 1 of the main paper\n\n`R5.6`: *LPIPs metric*:\n\nOur PromptIR provides better image fidelity scores using LPIPs than AirNet, as shown in Table R6.\n\n| Method    | Denoising on BSD68 | Deraining on RAIN100L | Dehazing on SOTS |\n|-----------|---------------------|-----------------------|------------------|\n| AirNet    | 0.1961              | 0.0525                | 0.0431           |\n| PromptIR  | 0.1958              | 0.0408                | 0.0168           |\n\n**Table R6**: LPIPs scores\n\n"}]

Author rebuttal:

Meta-review: The paper received 1xBorderline Reject and 4xBorderline Accept. The authors provided responses.

After carefully reading the paper, the reviews, as well as the authors' responses and discussions, the ACs agree with the majority of the reviewers that the work has merits and can be accepted for publication.

The authors are asked to further refine their paper for camera ready with the information provided in their responses and by using the received feedback from the reviewers.
Decision: Accept (poster)