SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We evaluate PromptRestorer on benchmarks for 4 image restoration tasks: (a) deraining, (b) deblurring, (c) desnowing, and (d) dehazing. We train separate models for different image restoration tasks. Our PromptRestorer employs a 3-level encoder-decoder. From level-1 to level-3, the number of CGT is [2, 3, 6], attention heads are [2, 4, 8], and number of channels is [48, 96, 192]. The expanding channel capacity factor  $\beta$  is 4. For downsampling and upsampling, we adopt pixel-unshuffle and pixel-shuffle [77], respectively. We train models with AdamW optimizer with the initial learning rate  $3e^{-4}$  gradually reduced to  $1e^{-6}$  with the cosine annealing [63]. The patch size is set as  $256\times 256$ .
Table 1: Image deraining results. Our PromptRestorer advances recent 14 state-of-the-arts on average.
Method	Test100 [107]	Rain100H [97]	Rain100L [97]	Test2800 [27]	Test1200 [106]	Average
PSNR ↑	SSIM ↑	PSNR ↑	SSIM ↑	PSNR ↑	SSIM ↑	PSNR ↑	SSIM ↑	PSNR ↑	SSIM ↑	PSNR ↑	SSIM ↑
DerainNet [26]	22.77	0.810	14.92	0.592	27.03	0.884	24.31	0.861	23.38	0.835	22.48	0.796
SEMI [95]	22.35	0.788	16.56	0.486	25.03	0.842	24.43	0.782	26.05	0.822	22.88	0.744
DIDMDN [106]	22.56	0.818	17.35	0.524	25.23	0.741	28.13	0.867	29.65	0.901	24.58	0.770
UMRL [99]	24.41	0.829	26.01	0.832	29.18	0.923	29.97	0.905	30.55	0.910	28.02	0.880
RESCAN [55]	25.00	0.835	26.36	0.786	29.80	0.881	31.29	0.904	30.51	0.882	28.59	0.857
PreNet [72]	24.81	0.851	26.77	0.858	32.44	0.950	31.75	0.916	31.36	0.911	29.42	0.897
MSPFN [44]	27.50	0.876	28.66	0.860	32.40	0.933	32.82	0.930	32.39	0.916	30.75	0.903
DCSFN [90]	27.46	0.887	28.98	0.887	34.70	0.961	30.96	0.903	32.92	0.937	31.00	0.915
MPRNet [103]	30.27	0.897	30.41	0.890	36.40	0.965	33.64	0.938	32.91	0.916	32.73	0.921
SPAIR [69]	30.35	0.909	30.95	0.892	36.93	0.969	33.34	0.936	33.04	0.922	32.91	0.926
Uformer [93]	29.17	0.880	30.06	0.884	36.34	0.966	33.36	0.935	31.98	0.909	32.18	0.915
MAXIM-2S [84]	31.17	0.922	30.81	0.903	38.06	0.977	33.80	0.943	32.37	0.922	33.24	0.933
Restorer [101]	32.00	0.923	31.46	0.904	38.99	0.978	34.18	0.944	33.19	0.926	33.96	0.935
SFNet [20]	31.47	0.919	31.90	0.908	38.21	0.974	33.69	0.937	32.55	0.911	33.56	0.929
PromptRestorer	31.84	0.920	31.72	0.908	39.04	0.977	34.40	0.947	33.27	0.928	34.05	0.936
(a) Input
20.83 dB
(b) RESCAN
Figure 4: Image deraining example on Rain100H [97].
21.89 dB
(c) DCSFN
21.51 dB
(d) MPRNet
22.41 dB
(e) Restormer
(f) PromptRestorer
22.46 dB
$\infty$
(g) GT
Table 2: Image deblurring results. Our PromptRestorer is trained only on the GoPro dataset [65] and directly applied to the HIDE [76] and RealBlur [73] benchmark datasets.
Benchmark	Metrics	Nah et al. [65]	SRN [81]	DBGAN [109]	MT-RNN [68]	DMPHN [105]	Suin et al. [80]	SPAIR [69]	MIMO-UNet+ [18]	MPRNet [103]	Restomer [101]	PromptRestorer
GoPro [65]	PSNR ↑	21.00	30.26	31.10	31.15	31.20	31.85	32.06	32.45	32.66	32.92	33.06
SSIM ↑	0.914	0.934	0.942	0.945	0.940	0.948	0.953	0.957	0.959	0.961	0.962
HIDE [76]	PSNR ↑	25.73	28.36	28.94	29.15	29.09	29.98	30.29	29.99	30.96	31.22	31.36
SSIM ↑	0.874	0.915	0.915	0.918	0.924	0.930	0.931	0.930	0.939	0.942	0.944
RealBlur-R [73]	PSNR ↑	32.51	35.66	33.78	35.79	35.70	-	-	35.54	35.99	36.19	36.06
SSIM ↑	0.841	0.947	0.909	0.951	0.948	-	-	0.947	0.952	0.957	0.954
RealBlur-J [73]	PSNR ↑	27.87	28.56	24.93	28.44	28.42	-	28.81	27.63	28.70	28.96	28.82
SSIM ↑	0.827	0.867	0.745	0.862	0.860	-	0.875	0.837	0.873	0.879	0.873
PSNR
(a) Degraded Image (b) Degraded Patch
24.83 dB
Figure 5: Image deblurring example on GoPro [65].
30.40 dB
(c) Nah et al.
31.04 dB
(d) SRN
31.07 dB
(e) MPRNet
(f) PromptRestorer
31.68 dB
Image Deraining Results. Similar to existing methods [44, 103, 69], we report PSNR/SSIM scores using Y channel in YCbCr color. Tab. 1 shows that our PromptRestorer outperforms current state-of-the-art approaches when averaged across all five datasets. Compared to the recent best method Restormer [101], PromptRestorer achieves  $0.09\mathrm{dB}$  improvement on average. On individual datasets, the gain can be as large as  $0.22\mathrm{dB}$ , e.g., Test2800 [27]. In Fig. 4, we present a challenging visual deraining example, where our PromptRestorer is able to generate a clearer result with finer details.
Image Deblurring Results. We evaluate deblurring results on both synthetic datasets (GoPro [65], HIDE [76]) and real-world datasets (RealBlur-R [73], RealBlur-J [73]). Tab. 2 summarises the results, where our PromptRestorer advances current state-of-the-art approaches on GoPro [65] and HIDE [76]. Compared with MPRNet [103], our PromptRestorer obtains a performance 0.12 dB gains. Fig. 5 provides a visual deblurring example. Our PromptRestorer produces a sharper result with fewer artifacts.
Table 3: Image dehazing results on SOTS-Indoor [52] and real-world benchmarks Dense-Haze [2] and NH-Haze [3]. Our PromptRestorer significantly advances state-of-the-arts on SOTS-Indoor [52].
Benchmark	Metrics	DCP [37]	DehazeNet [9]	AODNet [51]	GridNet [58]	FFANet [70]	MSBDN [21]	UHD [113]	MAXIM [84]	DeHamer [33]	PromptRestorer
SOTS-Indoor [52]	PSNR ↑	16.61	19.82	20.51	32.16	36.39	32.77	21.75	38.11	36.63	42.54
SSIM ↑	0.8546	0.8209	0.8162	0.9836	0.9886	0.9812	0.8786	0.9910	0.9881	0.9945
Dense-Haze [2]	PSNR ↑	11.01	9.48	12.82	14.96	12.22	15.13	12.16	-	16.62	15.86
SSIM ↑	0.4165	0.4383	0.4683	0.5326	0.4440	0.5551	0.4594	-	0.5602	0.5680
NH-Haze [3]	PSNR ↑	12.72	11.76	15.69	18.33	18.13	17.97	16.05	-	20.66	20.36
SSIM ↑	0.4419	0.3988	0.5728	0.6667	0.6473	0.6591	0.4612	-	0.6844	0.7203
PSNR
(a) Input
26.18 dB
(b) GridNet
Figure 6: Image dehazing example on SOTS-Indoor [52].
29.14 dB
(c) MSBDN
15.52 dB
(d) UHD
32.77 dB
(e) DeHamer
(f) PromptRestorer
36.86 dB
Table 4: Image desnowing results on CSD (2000) [65], SRRS (2000) [76], and Snow100K (2000) [73]. Our PromptRestorer achieves the best metrics on all datasets on the image desnowing problem.
Benchmark	Metrics	DesnowNet [61]	JSTASR [14]	HDCW-Net [15]	TransWeather [85]	MSP-Former [13]	Uformer [94]	Restomer [101]	PromptRestorer
CSD (2000) [15]	PSNR ↑	20.13	27.96	29.06	31.76	33.75	33.80	35.43	37.48
SSIM ↑	0.81	0.88	0.91	0.93	0.96	0.96	0.97	0.99
SRRS (2000) [14]	PSNR ↑	20.38	25.82	27.78	28.29	30.76	30.12	32.24	33.99
SSIM ↑	0.84	0.89	0.92	0.92	0.95	0.96	0.96	0.99
Snow100K (2000) [61]	PSNR ↑	30.50	23.12	31.54	31.82	33.43	33.81	34.67	36.02
SSIM ↑	0.94	0.86	0.95	0.95	0.96	0.94	0.95	0.97
24.18 dB
(b) JSTASR
Figure 7: Image desnowing example on CSD (2000) [15].
26.81 dB
(c) HDCWNet
29.07 dB
(d) Uformer
29.72 dB
(e) Restormer
(f) PromptRestorer
34.70 dB
Image Dehazing Results. We perform the image dehazing experiments on both synthetic benchmark RESIDE SOTS-Indoor [52], and real-world hazy benchmarks Dense-Haze [2] and NH-Haze [3]. Tab. 3 summarise the quantitative results. Compared to the recent works DeHamer [33] and MAXIM [84], our method receives 4.33 dB and 5.91 dB PSNR gains on the SOTS-Indoor, respectively. On the real-world benchmark NH-Haze [3], our PromptRestorer can achieve 0.7203 of the SSIM result, which is a new record and significantly outperforms current state-of-the-art approaches DeHamer [33]. The results on both synthetic and real-world benchmarks have demonstrated the effectiveness of our PromptRestorer on the image dehazing task. Fig. 6 shows the visual results, where our PromptRestorer is more effective in removing haze than other methods.
Image Desnoing Results. For the image desnoing task, we compare our PromptRestorer on the CSD [15], SRRS [14], and Snow100K [61] datasets with existing state-of-the-art methods [61, 14, 15, 13, 85]. We also compare recent Transformer-based general image restoration approaches Restormer [101] and Uformer [93]. As shown in Tab. 4, our PromptRestorer yields a  $2.05\mathrm{dB}$  PSNR improvement over the state-of-the-art approach [101] on the CSD benchmark [15]. The visual results in Fig. 7 show that our PromptRestorer is able to remove spatially varying snowflakes than competitors.
(a) Effect on L2P. Both Res-InBan and Deg-InBan play positive roles for image restoration.
Table 6: Ablation experiments on PromptDPM. Each component in L2P and G2P is effective.
Experiment	PSNR	FLOPs (G)	Params (M)
w/o L2P	30.819	148.34	12.60
w/o Res-InBan	30.952	153.39	12.97
w/o Deg-InBan	30.964	153.39	12.97
Full (Ours)	31.015	157.04	13.24
(b) Effect on G2P. Both Q-InAtt and KV-InAtt play a positive effect on high-quality image restoration.
Experiment	PSNR	FLOPs (G)	Params (M)
w/o G2P	30.697	123.44	10.69
w/o Q-InAtt	30.914	148.93	12.65
w/o KV-InAtt	30.906	147.26	12.52
Full (Ours)	31.015	157.04	13.24
4.2 Analysis and Discussion
For ablation experiments, following [84, 20], we train the image deblurring model on GoPro dataset [65] for 1000 epochs only and set the number of Transformer in each CGT is 1. Params mean the number of learnable parameters. Testing is performed on the GoPro testing dataset [65]. FLOPs are computed on image size  $256 \times 256$ . Next, we describe the influence of each component individually.
Effect on Prompting. The core design of our PromptRestorer is the 'prompting', which exploits a pre-trained model to extract raw degradation features from the degraded observations and then generate perceived content to guide the restoration branch (i.e., Case 3 in Fig. 1).
Compared to existing frameworks such as Cases 1-2 in Fig. 1, our proposed prompting strategy shows superior performance, as demonstrated in Tab. 5. Our method achieved  $0.877\mathrm{dB}$  gains compared to Case 1, and  $0.369\mathrm{dB}$  higher than Case 2. Interestingly, the learnable condition branch in Case  $2^{2}$ , despite consuming more FLOPs and Params, results in worse performance than ours. Our approach directly exploits raw degradation features to prompt restoration with persistent degradation priors to facilitate
Table 5: Effect on prompting. Our method that directly exploits the raw degradation to prompt restoration performs better.
Case in Fig. 1	PSNR	FLOPs (G)	Params (M)
1	30.138	105.58	10.16
2	30.646	157.04	16.77
3 (Ours)	31.015	157.04	13.24
better recovery. Fig. 1(c) shows two examples, where our model that exploits raw degradation features as prompting generates sharper and clearer images.
Effect on PromptDPM. We analyze the impact of PromptDPM on restoration quality in Tab. 6 by disabling one component at a time. Each model in L2P and G2P consumes similar Params and FLOPs, while our full model achieves the best performance. Disabling L2P or G2P results in a decrease in performance by  $0.196\mathrm{dB}$  and  $0.318\mathrm{dB}$ , respectively. These experiments conclusively demonstrate the effectiveness of each component in L2P and G2P for restoration.
Effect on GDP. To understand the impact of GDP, we disable it to compare with full model in Tab. 7.
Note that the computational cost of the GDP is negligible compared to disabling it as it only involves a  $1 \times 1$  convolution and sigmoid function for the gating mechanism, while it leads to a gain of  $0.091\mathrm{dB}$ . This finding highlights the significance of controlling the propagation of the perceived degradation features.
Table 7: Effect on GDP. Our GDP which controls the degradation propagation is effective.
Experiment	PSNR	FLOPs (G) Params (M)
w/o GDP	30.924	154.60	12.95
w/ GDP (Ours)	31.015	157.04	13.24
4.3 Visualization Understanding for Degradation Vanishing
To emphasize the understanding of degradation vanishing, we visualize the features learned in the condition/prompting branches to better comprehend the learned status of these branches in Fig. 8. Notably, both Cases 1-2 exhibit sharper results in later iterations compared to earlier ones, which fail to provide the restoration branch with sufficient degraded information and cause the restoration
2We ensure fairness in the comparison by using the same network architecture in the condition branch of Case 2 and the encoder of the pre-trained model in our network.
models to not perceive the degradation well, thereby hindering the model capacity. In contrast, as the restoration branch needs to adapt perceived features from the PromptDPM which is to perceive the raw degradation features from inputs, our model (Case 3) initially exhibits inferior performance (around 20K iterations) as shown in Fig. 1(b). However, with better adaptation to the degradation information after more iterations, the prompting branch can better prompt the restoration branch consistently with more reliable perceived content learned from the raw degradation, enabling our restoration branch to overcome degradation vanishing and improve restoration quality, as shown in Fig. 1(c).
Figure 8: Visualization. We show the average features over the channel dimension in the condition/prompting branches for the second example in Fig. 1(c). We obtain GT/degraded features by inputting GT/degraded images into the pre-trained VQGAN. As single-branch models (Case 1 in Fig. 1) do not have condition branches, we visualize the last layer in the 1-level encoder for reference.
In this paper, we investigate the degradation vanishing in the learning process for image restoration. To solve this problem, we have proposed the PromptRestorer which explores the raw degradation features extracted by a pre-trained model from the given degraded observations to guide the restoration process to facilitate better recovery. Extensive experiments have demonstrated that our PromptRestorer favors against state-of-the-art approaches on 4 restoration tasks, including image deraining, deblurring, dehazing, and desnowing.
[1] A. Abuolaim and M. S. Brown. Defocus deblurring using dual-pixel data. In ECCV, 2020.
[2] C. O. Ancuti, C. Ancuti, M. Sbert, and R. Timofte. Dense-haze: A benchmark for image dehazing with dense-haze and haze-free images. In ICIP, pages 1014-1018, 2019.
[3] C. O. Ancuti, C. Ancuti, and R. Timofte. Nh-haze: An image dehazing benchmark with non-homogeneous hazy and haze-free images. In CVPR workshops, pages 444-445, 2020.
[4] S. Anwar and N. Barnes. Densely residual laplacian super-resolution. TPAMI, 2020.
[5] S. Anwar, S. Khan, and N. Barnes. A deep journey into super-resolution: A survey. ACM Computing Surveys, 2019.
[6] J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv:1607.06450, 2016.
[7] D. Berman, T. Treibitz, and S. Avidan. Non-local image dehazing. In CVPR, pages 1674-1682, 2016.
[8] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. arXiv:2005.14165, 2020.
[9] B. Cai, X. Xu, K. Jia, C. Qing, and D. Tao. Dehazenet: An end-to-end system for single image haze removal. IEEE TIP, 25(11):5187-5198, 2016.
[10] H. Cai, J. He, Y. Qiao, and C. Dong. Toward interactive modulation for photo-realistic image restoration. In CVPR Workshops, pages 294–303, 2021.
[11] C. Chen, X. Shi, Y. Qin, X. Li, X. Han, T. Yang, and S. Guo. Real-world blind super-resolution via feature matching with implicit high-resolution priors. In ACM MM, pages 1329-1338, 2022.
[12] H. Chen, Y. Wang, T. Guo, C. Xu, Y. Deng, Z. Liu, S. Ma, C. Xu, C. Xu, and W. Gao. Pre-trained image processing transformer. In CVPR, 2021.
[105] H. Zhang, Y. Dai, H. Li, and P. Koniusz. Deep stacked hierarchical multi-patch network for image deblurring. In CVPR, 2019.
[106] H. Zhang and V. M. Patel. Density-aware single image de-raining using a multi-stream dense network. In CVPR, 2018.
[107] H. Zhang, V. Sindagi, and V. M. Patel. Image de-raining using a conditional generative adversarial network. TCSVT, 2019.
[108] K. Zhang, Y. Li, W. Zuo, L. Zhang, L. Van Gool, and R. Timofte. Plug-and-play image restoration with deep denoiser prior. TPAMI, 2021.
[109] K. Zhang, W. Luo, Y. Zhong, L. Ma, B. Stenger, W. Liu, and H. Li. Deblurring by realistic blurring. In CVPR, 2020.
[110] Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong, and Y. Fu. Image super-resolution using very deep residual channel attention networks. In ECCV, 2018.
[111] Y. Zhang, K. Li, K. Li, B. Zhong, and Y. Fu. Residual non-local attention networks for image restoration. In ICLR, 2019.
[112] Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu. Residual dense network for image restoration. TPAMI, 2020.
[113] Z. Zheng, W. Ren, X. Cao, X. Hu, T. Wang, F. Song, and X. Jia. Ultra-high-definition image dehazing via multi-guided bilateral learning. In CVPR, pages 16185-16194, 2021.
[114] Z. Zheng, X. Yue, K. Wang, and Y. You. Prompt vision transformer for domain generalization. CoRR, abs/2208.08914, 2022.
[115] K. Zhou, J. Yang, C. C. Loy, and Z. Liu. Conditional prompt learning for vision-language models. In CVPR, pages 16795-16804, 2022.
[116] S. Zhou, K. C. K. Chan, C. Li, and C. C. Loy. Towards robust blind face restoration with codebook lookup transformer. In NeurIPS, 2022.
[117] H. Zhu, C. Wang, Y. Zhang, Z. Su, and G. Zhao. Physical model guided deep image deraining. In ICME, 2020.