SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2: Dehazing results in the single-task setting on the SOTS benchmark dataset [31]. Our PromptIR achieves a significant boost of  $8.13\mathrm{dB}$  over AirNet [29].
Method	DehazeNet[4]	MSCNN[46]	AODNet[30]	EPDN[44]	FDGAN[14]	AirNet[29]	Restormer[68]	PromptIR (Ours)
PSNR	22.46	22.06	20.29	22.57	23.15	23.18	30.87	31.31
SSIM	0.851	0.908	0.877	0.863	0.921	0.900	0.969	0.973
Table 3: Deraining results in the single-task setting on Rain100L [16]. Compared to the AirNet [29] algorithm, the proposed method yields 2.13 dB PSNR improvement.
Method	DIDMDN[72]	UMR[66]	SIRR[63]	MSPFN[24]	LPNet[17]	AirNet[29]	Restormer[68]	PromptIR(Ours)
PSNR	23.79	32.39	32.37	33.50	33.61	34.90	36.74	37.04
SSIM	0.773	0.921	0.926	0.948	0.958	0.977	0.978	0.979
Table 4: Denoising comparisons in the single-task setting on BSD68 [41] and Urban100 [22] datasets. For the challenging noise level of  $\sigma = 50$  on Urban100 [22], our PromptIR obtains 0.51 dB gain compared to AirNet [29].
Method	BSD68 [41]	Urban100 [22]
σ = 15	σ = 25	σ = 50	σ = 15	σ = 25	σ = 50
CBM3D [11]	33.50/0.922	30.69/0.868	27.36/0.763	33.93/0.941	31.36/0.909	27.93/0.840
DnCNN [75]	33.89/0.930	31.23/0.883	27.92/0.789	32.98/0.931	30.81/0.902	27.59/0.833
IRCNN [76]	33.87/0.929	31.18/0.882	27.88/0.790	27.59/0.833	31.20/0.909	27.70/0.840
FFDNet [77]	33.87/0.929	31.21/0.882	27.96/0.789	33.83/0.942	31.40/0.912	28.05/0.848
BRDNet [52]	34.10/0.929	31.43/0.885	28.16/0.794	34.42/0.946	31.99/0.919	28.56/0.858
AirNet [29]	34.14/0.936	31.48/0.893	28.23/0.806	34.40/0.949	32.10/0.924	28.88/0.871
PromptIR(Ours)	34.34/0.938	31.71/0.897	28.49/0.813	34.77/0.952	32.49/0.929	29.39/0.881
4.2 Single Degradation One-by-One Results
In this section, we evaluate the performance of our PromptIR under the single-task setting, i.e., a separate model is trained for different restoration tasks. This is to show that content-adaptive prompting via prompt block is also useful for single-task networks. Table 2 presents dehazing results. It shows that our PromptIR achieves 8.13 dB improvement over AirNet [29], and 0.44 dB gain over the baseline method Restormer [68]. Similar trends can be observed for deraining and denoising tasks. For instance, when compared to the AirNet [29], our method yields performance gains of 2.13 dB on the deraining task (Table 3) and 0.51 dB on denoising task for noise level  $\sigma = 50$  on Urban100 dataset [22] (see Table 4).
4.3 Ablations Studies
We perform several ablation experiments to demonstrate that our contributions in PromptIR framework provide quality improvements.
Impact of PGM. We carry out this ablation experiment on Rain100L [65] for deraining task. Table 5 shows that the prompt block in our PromptIR network brings performance gains of  $0.3\mathrm{dB}$  over the baseline [68]. Further, it demonstrates that generating dynamic prompts conditioned on input content via PGM provides a favorable gain of  $0.19\mathrm{dB}$  over the fixed prompt components.
Position of prompt blocks. In the hierarchical architecture of our PromptIR, we analyze where to place prompt blocks on the decoder side. Table 6 shows that using only one prompt block in the latent space degrades the network's performance. Whereas, incorporating prompt blocks between every consecutive level of the decoder performs the best.
Generalization to unseen degradation level. We take the model that is trained only on the noise levels  $\sigma \in \{15, 25, 50\}$  and test its performance on the unseen noise level of  $\sigma = 100$ .
Table 7 shows that our PromptIR demonstrates significantly superior generalization capabilities compared to AirNet [29], yielding  $\sim 7$  dB PSNR difference.
Table 5: Impact of PGM. Results are reported on Rain100L [65] dataset.
Method	PSNR
Baseline [68]	36.74
Fixed-Prompt	36.85
Dynamic-Prompt	37.04
Table 6: Prompt blocks position. Results are reported on Rain100L [65] dataset.
Model	PSNR
level 4 (latent)	36.76
levels 4+3	36.84
levels 4+3+2	37.04
Table 7: Denoising comparisons on unseen noise level of  $\sigma  = {100}$  .
BSD68 [41]	Urban100 [22]
Method	PSNR	PSNR
AIRnet [29]	13.64	13.72
PromptIR (Ours)	21.03	20.50
Table 8: Evaluation on Spatially Variant Degradation on BSD68 [41] test set.
Model	PSNR
Airnet [29]	31.42
PromptIR (Ours)	31.65
Table 9: Performance of the proposed PromptIR framework, when trained on different combinations of degradation types (tasks) i.e., removal of noise, rain and haze.
Degradation	Denoising on BSD68 dataset [41]	Deraining on Rain100L [16]	Dehazing on SOTS [31]
Noise	Rain	Haze	σ = 15	σ = 25	σ = 50
✓	X	X	34.34/0.938	31.71/0.898	28.49/0.813	-	-
X	✓	X	-	-	-	37.03/0.9786	-
X	X	✓	-	-	-	-	31.31/0.929
✓	✓	X	34.26/0.937	31.61/0.895	28.37/0.810	39.32/0.986	-
✓	X	✓	33.69/0.928	31.03/0.880	27.74/0.777	-	30.09/0.975
X	✓	✓	-	-	-	35.12/0.969	30.36/0.973
✓	✓	✓	33.98/0.933	31.31/0.888	28.06/0.799	36.37/0.972	30.58/0.974
Performance on spatially variant degradation. Here we evaluate PromptIR performance on images that are corrupted with varying degradations. For this, we follow closely the work of AirNet [29], and prepare a new test set from BSD68 [41] by applying Gaussian noise of varying levels  $\sigma = [0, 15, 25, 50]$  at different spatial locations of the images. Results in Table 8 show that our PromptIR framework is more effective in restoring these images than AirNet [29], providing  $0.23\mathrm{dB}$  improvement.
Training model with different combinations of degradation. In Table 1, we report the results of training an all-in-one model on combined datasets from all three restoration tasks. Here, we evaluate the impact on PromptIR performance by different combinations of degradation types (tasks). Table 9) shows that with an increasing number of degradation types, it becomes increasingly difficult for the network to restore images, thereby leading to a performance drop.
Specifically, the presence of hazy images in the combined dataset seems to negatively affect the model. Interestingly, a model trained on the combination of rainy and noisy images achieves good performance, indicating a positive correlation between the deraining and denoising tasks. Such phenomenon is also observed in the AirNet work [29].
Impact of Prompt Block. To effectively evaluate the benefit of the proposed framework, we evaluate it against the Restormer model [68]. Restormer is scaled up to match the parameter count of PromptIR. We perform this evaluation under the All-in-one setting. As shown in Table 10, the PromptIR framework provides an average improvement of  $0.38\mathrm{dB}$  over the Restormer model [68].
Number of prompt components. We conduct an abalation study to understand the effect of the number of prompt components  $P_{c}$  employed on the performance of the framework. As shown in Table 11, we find that utilizing more than 5 components yields marginal improvements while incurring additional computational overhead. As a result, we choose to utilize five components
Table 10: Comparison of Restormer [68] against the proposed PromptIR framework.
Method	Dehazing on SOTS [31]	Deraining on Rain100L [16]	Denoising on BSD68 dataset [41])	Average
σ = 15	σ = 25	σ = 50
Restormer[68]	29.92/0.9703	35.56/0.9691	33.86/0.9327	31.20/0.8878	27.90/0.7944	31.68/0.910
PromptIR (Ours)	30.58/0.974	36.37/0.972	33.98/0.933	31.31/0.888	28.06/0.799	32.06/0.913
Table 11: Evaluation on the number of Prompt Components on Rain100L[16]
Number of Prompt Components	3	5	6	7
PSNR	36.91	37.04	37.06	37.07
Existing image restoration models based on deep neural networks can work for specific degradation types and do not generalize well to other degradations. However, practical settings demand the ability to handle multiple degradation types with a single unified model without resorting to degradation-specific models that lack generalization and require apriori knowledge of specific degradation in the input. To this end, our work proposed a drop-in prompt block that can interact with the input features to dynamically adjust the representations such that the restoration process is adapted for the relevant degradation. We demonstrated the utility of prompt block for all-in-one image restoration by integrating it within a SoTA restoration model that leads to significant improvements on image denoising, deraining, and dehazing tasks. In the future, we will extend the model for a broader set of corruptions toward the goal of universal models for better generalization in image restoration tasks.
The computational resources were provided by the National Academic Infrastructure for Supercomputing in Sweden (NAISS), partially funded by the Swedish Research Council through grant agreement no. 2022-06725, and by Berzelius resource, provided by the Knut and Alice Wallenberg Foundation at the National Supercomputer Centre.
[1] Arbelaez, P., M. Maire, C. Fowlkes, and J. Malik (2011). Contour detection and hierarchical image segmentation. TPAMI.
[2] Ba, J. L., J. R. Kiros, and G. E. Hinton (2016). Layer normalization. arXiv:1607.06450.
[3] Brown, T. B., B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. (2020). Language models are few-shot learners. arXiv:2005.14165.
[4] Cai, B., X. Xu, K. Jia, C. Qing, and D. Tao (2016). Dehazenet: An end-to-end system for single image haze removal. IEEE Transactions on Image Processing 25(11), 5187-5198.
[5] Carion, N., F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko (2020). End-to-end object detection with transformers. In ECCV.
[6] Chen, H., Y. Wang, T. Guo, C. Xu, Y. Deng, Z. Liu, S. Ma, C. Xu, C. Xu, and W. Gao (2021a). Pre-trained image processing transformer. In CVPR, pp. 12299-12310.
[7] Chen, H., Y. Wang, T. Guo, C. Xu, Y. Deng, Z. Liu, S. Ma, C. Xu, C. Xu, and W. Gao (2021b). Pre-trained image processing transformer. In CVPR.
[8] Chen, L., X. Chu, X. Zhang, and J. Sun (2022). Simple baselines for image restoration. In ECCV.
[9] Chen, X., X. Wang, J. Zhou, and C. Dong. Activating more pixels in image super-resolution transformer. arxiv 2022. arXiv preprint arXiv:2205.04437.
[10] Cornillere, V., A. Djerouah, W. Yifan, O. Sorkine-Hornung, and C. Schroers (2019). Blind image superresolution with spatially variant degradations. ACM Transactions on Graphics (TOG) 38(6), 1-13.
[78] Zheng, S., J. Lu, H. Zhao, X. Zhu, Z. Luo, Y. Wang, Y. Fu, J. Feng, T. Xiang, P. H. Torr, et al. (2021). Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In CVPR.
[79] Zhou, K., J. Yang, C. C. Loy, and Z. Liu (2022). Learning to prompt for vision-language models. International Journal of Computer Vision (IJCV).
[80] Zhu, X., W. Su, L. Lu, B. Li, X. Wang, and J. Dai (2020). Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159.
A Additional Ablation Studies
We conduct further ablation studies to illustrate the effectiveness of various design choices of the PromptIR framework. We examine various key design choices like the usage of prompt tokens and plugging in prompt blocks only on the decoder branch of the network.
A.1 Contrastive learning-based Degradation Encoder embedding v/s Prompt Tokens
To strengthen the design rationale for incorporating prompts instead of following recent methods [29] that use embeddings learned through contrastive training, we replace the generated prompt from our PGM module with embeddings extracted from the Contrastive- learning based Degradation Encoder of the AirNet [29] model. We observed that the use of contrastive embeddings resulted in significantly weaker performance compared to prompt tokens. Moreover, achieving good performance with contrastive embeddings requires a custom-designed restoration network, whereas our Prompt Blocks can be seamlessly integrated as plug-and-play modules into any restoration network.
Table A.1: Comparisons under all-in-one setting: between the usage of degradation embedding extracted from the Contrastive-learning Based Degradation Encoder (CBDE) of the Airnet [29] Model and the usage of prompt tokens in the PromptIR framework.
Method	Dehazing on SOTS [31]	Deraining on Rain100L [16]	Denoising on BSD68 dataset [41])	Average
σ = 15	σ = 25	σ = 50
CBDE+PromptIR	23.92/0.881	32.03/0.0.972	32.96/0.910	30.36/0.860	26.93/0.732	29.24/0.875
PromptIR (Ours)	30.58/0.974	36.37/0.972	33.98/0.933	31.31/0.888	28.06/0.799	32.06/0.913
A.2 Prompt Blocks on both Encoder branch and Decoder branch
We study the importance of decoder-only prompting by evaluating the usage of prompt blocks on both the encoder and decoder branches. We show that it is important the prompt block is only used on the decoder side.
Table A.2: Comparisons under the all-in-one setting: between the usage of the Prompt-block on both the encoder branch and encoder branch with using the prompt block only on the decoder branch.
Method	Dehazing on SOTS [31]	Deraining on Rain100L [16]	Denoising on BSD68 dataset [41])	Average
σ = 15	σ = 25	σ = 50
Enc+Dec+PromptIR	28.52/0.927	35.43/0.965	33.59/0.927	30.85/0.878	27.35/0.732	31.14/0.885
PromptIR (Ours)	30.58/0.974	36.37/0.972	33.98/0.933	31.31/0.888	28.06/0.799	32.06/0.913
B Transformer Block in PromptIR Framework
As mentioned in section 3.1.2 of the main manuscript, we present the block diagramB.1 of the transformer block and further, elaborate on the details of the transformer block employed in the PromptIR framework. The transformer block follows the design and hyper-parameters outlined in [68]
To begin, the input features  $\mathbf{X} \in \mathbb{R}^{H_l \times W_l \times C_l}$  are passed through the MDTA module. In this module, the features are initially normalized using Layer normalization. Subsequently, a combination of  $1 \times 1$  convolutions followed by  $3 \times 3$  depth-wise convolutions are applied to project the features into Query (Q), Key (K), and Value (V) tensors. An essential characteristic of the MDTA module is its computation of attention across the channel dimensions, rather than the spatial dimensions. This effectively reduces the computational overhead. To achieve this channel-wise attention calculation, the  $Q$  and  $K$  projections are reshaped from  $H_l \times W_l \times C_l$  to  $H_lW_l \times C_l$  and  $C_l \times H_lW_l$  respectively,
Multi-Dconv head Transposed Attention
Gated Dconv Feedforward Network
$3\times 3$  depth-wise convolution  $(W_{d})$
1x1 convolution  $(W_{p})$
Softmax operation
Element-wise
GeLU Activation
Figure B.1: Overview of the Transformer block used in the PromptIR framework. The Transformer block is composed of two sub modules, the Multi Dconv head transposed attention module (MDTA) and the Gated Dconv feed-forward network (GDFN).
before calculating dot-product, hence the resulting transposed attention map with the shape of  $C_l \times C_l$ . Bias-free convolutions are utilized within this submodule. Furthermore, attention is computed in a multi-head manner in parallel.
After MDTA Module the features are processed through the GDFN module. In the GDFN module, the input features are expanded by a factor  $\gamma$  using  $1\times 1$  convolution and they are then passed through  $3\times 3$  convolutions. These operations are performed through two parallel paths and the output of one of the paths is activated using GeLU non-linearity. This activated feature map is then combined with the output of the other path using element-wise product.
C Qualitative results:
We present more qualitative results from single-task models to further elucidate the effectiveness of prompt-block even when under the single-task setting.
C.1 Dehazing
Figure C.1: Image deraining comparisons under single task setting on images from the SOTS dataset [31]. Our method effectively removes haze to produce visually better images.
C.2 Deraining
Figure C.2: Image deraining comparisons under single task setting on images from the Rain100L dataset [16]. Our method effectively removes rain streaks to generate rain-free images.
C.3 Denoising
Figure C.3: Image deraining comparisons under single task setting on images from the URBAN100 dataset [22] with  $\sigma = 50$ . Our method produces visually better images as compared to previous methods. We show selected patches from the images.