SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Cong Wang $^{1}$ , Jinshan Pan $^{2*}$ , Wei Wang $^{3*}$ , Jiangxin Dong $^{2}$ , Mengzhu Wang $^{4}$ , Yakun Ju $^{1}$ , Junyang Chen $^{5}$
<sup>1</sup>The Hong Kong Polytechnic University, <sup>2</sup>Nanjing University of Science and Technology,
$^{3}$ Dalian University of Technology,  $^{4}$ Hebei University of Technology,  $^{5}$ Shenzhen University
We show that raw degradation features can effectively guide deep restoration models, providing accurate degradation priors to facilitate better restoration. While networks that do not consider them for restoration forget gradually degradation during the learning process, model capacity is severely hindered. To address this, we propose a Prompt image Restorer, termed as PromptRestorer. Specifically, PromptRestorer contains two branches: a restoration branch and a prompting branch. The former is used to restore images, while the latter perceives degradation priors to prompt the restoration branch with reliable perceived content to guide the restoration process for better recovery. To better perceive the degradation which is extracted by a pre-trained model from given degradation observations, we propose a prompting degradation perception modulator, which adequately considers the characters of the self-attention mechanism and pixel-wise modulation, to better perceive the degradation priors from global and local perspectives. To control the propagation of the perceived content for the restoration branch, we propose gated degradation perception propagation, enabling the restoration branch to adaptively learn more useful features for better recovery. Extensive experimental results show that our PromptRestorer achieves state-of-the-art results on 4 image restoration tasks, including image deraining, deblurring, dehazing, and desnowing.
Image restoration aims to recover clear high-quality images from given degraded ones. It is highly ill-posed since only degraded images can be exploited, statistical observations are thus required to well-pose the problems [37, 67, 66, 41, 42]. Although conventional approaches can recover images to some extent, they typically involve solving optimization algorithms that are difficult due to the non-convexity and non-smooth problems. Additionally, the observations may not always hold, which can cause algorithms to fail.
With the emergence of convolutional neural networks (CNNs) [48] and Transformers [22, 45], which perform well at implicitly learning the priors from large-scale data, learning-based methods have dominated recent image restoration tasks and achieved impressive performance [81, 51, 103, 72, 61, 93, 33, 13]. However, these methods are usually built without explicitly considering the specific degradation information, which accordingly limits model capacity (Case 1 in Fig. 1). An alternative approach is to design a conditional branch to learn additional information to provide the restoration network with useful content for modulation [30, 17, 34, 35] (Case 2 in Fig. 1). However, we note that while conditional branches in these models are learnable, they may not effectively provide degradation information, as the estimizable parameters result in gradually clearer features during the learning process, leading to the degradation vanishing which accordingly limits model performance.
*These authors equally contribute to this work.
Figure 1: (a) compares different restoration frameworks. Unlike existing approaches that are built within the architectures such as Cases 1-2, which are unable to memorize the degradation well during the learning process, we propose a prompting method (Case 3) that directly exploits raw degradation features extracted by a pre-trained model from the given degradation observations to guide restoration. In (b), we observe that both Cases 1-2 outperform our method in early iterations, as they effectively memorize degraded information. However, both Cases 1-2 experience degradation vanishing with further iterations (better demonstrated in Sec. 4.3), while our prompting method persists in guiding the restoration network with accurate degradation priors, accordingly producing better restoration quality. In (c), visual performance demonstrates that our prompting method recovers sharper images. Quantitative results are reported in Tab. 5.
(b) Learning Curves
(c) Visual Performance
Recently, prompt learning has been shown an effective tool to improve model performance by designing various prompts [115, 98, 104, 23, 114, 46, 53, 43]. The prompt usually serves as the guidance tool to correct the networks toward better results [28]. However, prompt learning still keeps a margin for image restoration, and existing prompts may not be suitable for image restoration since they cannot effectively model degradation priors well. Hence, we ask: Is there a reasonable prompting manner to correct degraded image restoration networks to facilitate better recovery?
The answer is in the riddle. This paper proposes the PromptRestorer, a Prompting image Restorer, to overcome degradation vanishing in image restoration via promoting by exploring degradation input itself for better restoration (Case 3 in Fig. 1). Our idea is simple: we directly exploit the raw degraded features extracted by a pre-trained model from the degraded inputs to generate more reliable prompting content to guide image restoration. Raw degraded features preserve accurately degraded information, which can consistently prompt the restoration network with accurate degraded priors, enabling the restoration network to perceive the degradation for better restoration. Hence, we design the PromptRestorer, which consists of two branches: (a) the restoration branch and (b) the prompting branch. The former is used to restore images and the latter is used to generate reliable prompting features to guide the restoration network for better restoration. To better perceive the degradation, we propose a Prompting Degradation Perception Modulator (PromptDPM), which consists of Global Prompting Perceptor (G2P) and Local Prompting Perceptor (L2P). The G2P adequately exploits the self-attention mechanism to form global prompting attention, while the L2P considers the pixel-level perception to build local prompting content. To control the propagation of perceived features in the restoration branch, we propose Gated Degradation Perception Propagation (GDP), enabling the restoration network to adaptively learn more useful features to facilitate better restoration.
The main contributions of this work are summarized below:
- We propose PromptRestorer, which is the first approach to our knowledge that takes advantage of the prompting learning for general image restoration by considering raw degradation features in restoration, enabling the restoration model to overcome degradation vanishing while consistently retaining the degradation priors to facilitate better restoration.
- We propose a prompting degradation perception modulator that is used to perceive degradation from global and local perspectives, which is able to provide the restoration network
with more reliable perceived content learned from the degradation priors, enabling it to better guide the restoration process.
- We propose gated degradation perception propagation that exploits a gating mechanism to control the propagation of the perceived features, enabling the model to adaptively learn more useful features for better image restoration.
Fig. 1 summarises framework comparisons, and their learning curves and visual performance. Deeper analysis and discussion about them are provided in Sec. 4.2.
In this section, we review image restoration, conditional modulation, and prompt learning.
Image Restoration. Recently, CNN-based architectures [110, 112, 103, 4, 24, 102, 89, 91, 87, 19, 88, 117] and Transformer-based models [96, 56, 49, 12, 93, 93] have been shown to outperform conventional restoration approaches [37, 83, 64, 47, 7, 79]. These learning-based methods usually adopt U-Net architectures [50, 18, 103, 100, 1, 93, 108, 101], which have been demonstrated the effectiveness because of hierarchical multi-scale representation and effective learning between shallow and deeper layers by skip connection [111, 59, 102, 31]. We refer the readers to recent excellent literature reviews on image restoration [5, 54, 82], which summarise the main designs in deep image restoration models.
Although these models have achieved promising performance, they do not explicitly take degradation into consideration for model design which is vital for restoration, limiting the model capacity.
Conditional Modulation. Conditional modulation usually involves implicitly modulating the additional content to guide the restoration [30, 10, 16, 17, 36, 35, 34, 60, 57, 92]. These approaches usually contain two branches: a basic network and a conditional network. The conditional network provides additional information to guide the basic network for restoration via spatial feature transform (SFT) [92]. Among these methods, blur kernel [30], semantics [92], and degraded input [57, 16] which serve as the additional conditions are broadly known.
The learnable nature of the conditional network in these models does not effectively provide degradation information for the basic network. As parameters are optimized in the learning process, features become gradually clear.
Prompt Learning. Prompt learning methods have been studied broadly in natural language processing (NLP) [75, 78, 8]. Due to high effectiveness, prompt learning is recently used in vision-related tasks [115, 98, 104, 23, 114, 46, 53, 43, 71, 29, 40, 86, 28]. In vision prompt learning, many works seek useful prompts to correct task networks toward better performance [28].
Although prompt learning has shown promise in various vision tasks, it still keeps a margin in general image restoration. This paper proposes an effective prompting method, enabling the restoration model to overcome the degradation vanishing in the learning process for better restoration.
3 PromptRestorer
Our goal aims to overcome degradation vanishing and better perceive degradation in deep restoration models to improve image recovery quality. To achieve this, we introduce a prompting strategy that helps the model consistently memorize degradation information, enabling it to prompt restoration with better degradation for better restoration. To better perceive degradation, we propose the Prompting Degradation Perception Modulator (PromptDPM), which can provide more reliable perceived content to guide the restoration network. To control the propagation of the perceived content, we propose the Gated Degradation Perception Propagation (GDP), enabling the restoration network to adaptively learn more useful features for better restoration.
3.1 Overall Pipeline
Fig. 2 shows the framework of our PromptRestorer, which contains two branches: (a) the restoration branch and (b) the prompting branch. The restoration branch is used to restore images, where each block is prompted by the prompting branch. The prompting branch first generates the accurate
Figure 2: Overall pipeline of our PromptRestorer. PromptRestorer contains two branches: (a) the restoration branch and (b) the prompting branch. The restoration branch is used to restore images, where each block (c) in CGT is prompted by the prompting branch. The prompting branch first generates precise degradation features extracted by a pre-trained model from degradation observations, then these features prompt the restoration branch to facilitate better restoration via PromptDPM (d).
degradation feature extracted by a pre-trained model, and then the feature is to prompt the restoration branch, enabling the restoration branch to better perceive the degradation prior for better recovery.
Restoration Branch. Given a degraded input image  $\mathbf{I} \in \mathbb{R}^{H \times W \times 3}$ , we first apply a  $3 \times 3$  convolution as the feature extraction to obtain low-level embeddings  $\mathbf{X}_0 \in \mathbb{R}^{H \times W \times C}$ ; where  $H \times W$  denotes the spatial dimension and  $C$  is the number of channels. Next, the shallow features  $\mathbf{X}_0$  gradually are hierarchically encoded into deep features  $\mathbf{X}_l \in \mathbb{R}^{\frac{H}{l} \times \frac{W}{l} \times lC}$ . After encoding the degraded input into low-resolution latent features  $\mathbf{X}_3 \in \mathbb{R}^{\frac{H}{3} \times \frac{W}{3} \times 3C}$ , the decoder progressively recovers the high-resolution representations. Finally, a reconstruction layer which contains 4 Transformer blocks as the refinement followed by a  $3 \times 3$  convolution is applied to decoded features to generate residual image  $\mathbf{S} \in \mathbb{R}^{H \times W \times 3}$  to which degraded image is added to obtain the restored output image:  $\hat{\mathbf{H}} = \mathbf{I} + \mathbf{S}$ . Both encoder and decoder at  $l$ -level consist of multiple Continuous Gated Transformers (CGT) with expanding channel capacity. To help better recovery, the encoder features are concatenated with the decoder features via skip connections [74] by  $1 \times 1$  convolutions.
Prompting Branch. The prompting branch, as shown in Fig. 2(b), aims to generate and perceive degradation features and then provide useful guidance content for the restoration branch. We note VQGAN [25] has been demonstrated that it can generate high-quality images while representing the features of input images. However, it tends to damage image structure after vector quantization [116, 11, 32]. To avoid this problem, we only exploit the encoder of pre-trained VQGAN to represent the deep features of the degraded inputs. We first use the pretrained encoder to extract degraded features  $\mathbf{Y}_l\in \mathbb{R}^{\frac{H}{l}\times \frac{W}{l}\times lC}$ ; where  $l$  denotes the  $l$ -level layer in the pre-trained encoder. Then, the degraded features are exploited to generate reliable prompting content to prompt the restoration branch by PromptDPM (see Sec. 3.2). The generated prompting content is transmitted to each CGT to guide the restoration branch.
Continuous Gated Transformers. CGT exploits the perceived features from PromptDPM to provide the Transformer block with more reliable content to overcome degradation vanishing to facilitate better restoration. Each CGT consists of three Transformer blocks (Fig. 2(c)) with residual connections [38] and the input in each block is gated by GDP (see Sec. 3.3) to control the propagation of perceived features. Let  $\mathcal{P}$ ,  $\mathcal{G}$ , and  $\mathcal{T}$  respectively denote the operations of PromptDPM (expressed in (2)), GDP (expressed in (7)), and Transformer, the features flow in  $k^{th}$  block in one CGT at  $l$ -level encoder/decoder, which can be expressed as:
$$
\mathbf {X} _ {k} = \mathcal {T} \left(\mathbf {G} _ {k - 1}\right); \mathbf {G} _ {k - 1} = \mathcal {G} \left(\mathbf {X} _ {k - 1}, \mathbf {P} _ {l}\right); \mathbf {P} _ {l} = \mathcal {P} \left(\mathbf {X} _ {k - 1}, \mathbf {Y} _ {l}\right), \tag {1}
$$
Figure 3: (a) Global Prompting Perceptor (G2P); (b) Local Prompting Perceptor (L2P).
where  $\mathbf{X}_k$  means the output of  $k^{th}$  Transformer block in one CGT, especially  $\mathbf{X}_0$  is the downsampled/upsampled features at  $(l - 1)$ -level encoder/decoder;  $\mathbf{P}_l$  refers to the generated features of PromptDPM at  $l$ -level layer;  $\mathbf{G}_{k - 1}$  means the gated features between  $\mathbf{X}_{k - 1}$  and  $\mathbf{P}_l$ , which serves as the input of  $k^{th}$  Transformer block. Each Transformer block consists of multi-head attention [101] followed by an improved ConvNeXt [62] as the feed-forward network (see Fig. 2(c)).
3.2 Prompting Degradation Perception Modulator
To better perceive the degradation to prompt the restoration network with more reliable perceived content from the degradation priors, we propose the PromptDPM (see Fig. 2(d)). The PromptDPM consists of 1) Global Prompting Perceptor (G2P, introduced in Sec. 3.2.1) and 2) Local Prompting Perceptor (L2P, introduced in Sec. 3.2.2) to respectively perceive the degradation from global and local perspectives, enabling to generate more useful content to guide the restoration branch. From a restoration tensor  $\mathbf{X} \in \mathbb{R}^{\hat{H} \times \hat{W} \times \hat{C}}$  and a degradation tensor  $\mathbf{Y} \in \mathbb{R}^{\hat{H} \times \hat{W} \times \hat{C}}$ , we prompt  $\mathbf{X}$  with  $\mathbf{Y}$ :
$$
\mathcal {P} (\mathbf {X}, \mathbf {Y}) = W _ {p} \left(\mathcal {C} \left[ \Psi^ {\text {g l o b a l}} (\mathbf {X}, \mathbf {Y}), \Psi^ {\text {l o c a l}} (\mathbf {X}, \mathbf {Y}) \right]\right) + \mathbf {X}, \tag {2}
$$
where  $\Psi^{\mathrm{global}}(\cdot ,\cdot)$  and  $\Psi^{\mathrm{local}}(\cdot ,\cdot)$  respectively denote the operations of G2P and L2P;  $\mathcal{C}[\cdot ,\cdot ]$  means the concatenation at channel dimension;  $W_{p}(\cdot)$  refers to the  $1\times 1$  point-wise convolution.
3.2.1 Global Prompting Perceptor
The G2P, shown in Fig. 3(a), fully exploits the self-attention mechanism to form the global prompting attention induced by the degraded features. The G2P contains the global perception attention followed by an improved ConvNeXt [62]. Our global perception attention consists of 1) Query-Induced Attention (Q-InAtt) and 2) Key-Value-Induced Attention (KV-InAtt). The Q-InAtt considers re-forming the query vector induced by degradation features to build a representative query to perform attention, while the KV-InAtt re-considers key and value vectors induced by other degradation counterparts to search for more similar content with the restoration query. From a layer normalized restoration tensor  $\mathbf{X} \in \mathbb{R}^{\hat{H} \times \hat{W} \times \hat{C}}$ , our G2P first generates restoration query (Q), key (K), and value (V) projections from the restoration features. It is achieved by applying  $1 \times 1$  convolutions to aggregate pixel-wise cross-channel context followed by  $3 \times 3$  depth-wise convolutions  $W_{d}(\cdot)$  to encode channel-wise spatial context, yielding  $\mathbf{Q} = W_{d}W_{p}\mathbf{X}$ ,  $\mathbf{K} = W_{d}W_{p}\mathbf{X}$ , and  $\mathbf{V} = W_{d}W_{p}\mathbf{X}$ . Meanwhile, we similarly convert the degradation tensor  $\mathbf{Y} \in \mathbb{R}^{\hat{H} \times \hat{W} \times \hat{C}}$  into degradation query ( $\widetilde{\mathbf{Q}}$ ), key ( $\widetilde{\mathbf{K}}$ ), and value ( $\widetilde{\mathbf{V}}$ ) projections:  $\widetilde{\mathbf{Q}} = W_{d}W_{p}\mathbf{Y}$ ,  $\widetilde{\mathbf{K}} = W_{d}W_{p}\mathbf{Y}$ , and  $\widetilde{\mathbf{V}} = W_{d}W_{p}\mathbf{Y}$ . Then, we respectively conduct Q-InAtt and KV-InAtt:
$$
\mathbf {A} _ {\mathrm {Q} - \text {I n A t t}} = \mathcal {A} _ {\mathrm {Q} - \text {I n A t t}} \left(W _ {p} \left(\mathcal {C} [ \mathbf {Q}, \widetilde {\mathbf {Q}} ]\right), \mathbf {K}, \mathbf {V}\right); \mathbf {A} _ {\mathrm {K V} - \text {I n A t t}} = \mathcal {A} _ {\mathrm {K V} - \text {I n A t t}} \left(\mathbf {Q}, W _ {p} \left(\mathcal {C} [ \mathbf {K}, \widetilde {\mathbf {K}} ]\right), W _ {p} \left(\mathcal {C} [ \mathbf {V}, \widetilde {\mathbf {V}} ]\right)\right), \tag {3}
$$
where  $\mathcal{A}_{(\cdot)}\left(\hat{\mathbf{Q}},\hat{\mathbf{K}},\hat{\mathbf{V}}\right) = \hat{\mathbf{V}}\cdot \mathrm{Softmax}\left(\hat{\mathbf{K}}\cdot \hat{\mathbf{Q}} /\alpha\right)$ ; Here,  $\alpha$  is a learnable scaling parameter to control
the magnitude of the dot product of  $\hat{\mathbf{K}}$  and  $\hat{\mathbf{Q}}$  before applying the softmax function. Similar to the conventional multi-head SA [22], we divide the number of channels into 'heads' and learn separate attention maps. Then two induced attentions are fused and followed by an improved ConvNeXt:
$$
\mathbf {A} ^ {\prime} = W _ {p} \left(\mathcal {C} \left[ \mathbf {A} _ {\mathrm {Q} - \text {I n A t t}}, \mathbf {A} _ {\mathrm {K V} - \text {I n A t t}} \right]\right) + \mathbf {X}; \mathbf {A} = W _ {p} W _ {d} \phi W _ {p} W _ {d} \left(L N \left(\mathbf {A} ^ {\prime}\right)\right) + \mathbf {A} ^ {\prime}, \tag {4}
$$
where the  $W_{p}W_{d}\phi W_{p}W_{d}(\cdot)$  means the improved ConvNeXt shown in the latter of Fig. 2(c);  $LN(\cdot)$  means the operation of layer normalization [6].
3.2.2 Local Prompting Perceptor
The L2P, as shown in Fig. 3(b), adequately considers the pixel-level degradation perception, enabling to better perceive degradation from spatially neighboring pixel positions. The L2P consists of a local perception modulator followed by a separable depth-level convolution. The local perception modulator contains two core components: 1) Degradation-Induced Band (Deg-InBan) and 2) Restoration-Induced Band (Res-InBan). The former is achieved by exploiting the degradation features to induce spatially useful content from restoration content to guide restoration gating fusion, while the latter utilizes the deep restoration features to induce more useful features from another degradation counterpart to form the degradation gating. Given the degradation tensor  $\mathbf{Y} \in \mathbb{R}^{\hat{H} \times \hat{W} \times \hat{C}}$ , we first exploit the point-wise convolution and  $3 \times 3$  depth-wise convolution to encode two degradation projections, yielding  $\widetilde{\mathbf{Q}} = W_{d}^{Q}W_{p}^{Q}\mathbf{Y}$  and  $\widetilde{\mathbf{K}} = W_{d}^{K}W_{p}^{K}\mathbf{Y}$ . Meanwhile, the restoration tensor  $\mathbf{X} \in \mathbb{R}^{\hat{H} \times \hat{W} \times \hat{C}}$  are also encoded into two restoration projections:  $\mathbf{Q} = W_{d}^{Q}W_{p}^{Q}\mathbf{X}$  and  $\mathbf{K} = W_{d}^{K}W_{p}^{K}\mathbf{X}$ . Then, we respectively conduct Deg-InBan and Res-InBan:
$$
\mathbf {Z} _ {\text {D e g - I n B a n}} = \sigma \left(W _ {p} \phi W _ {d} \left(\mathcal {C} [ \widetilde {\mathbf {Q}}, \mathbf {Q} ]\right)\right) \odot \mathbf {K}; \mathbf {Z} _ {\text {R e s - I n B a n}} = \widetilde {\mathbf {Q}} \odot \sigma \left(W _ {\phi} W _ {d} \left(\mathcal {C} [ \widetilde {\mathbf {K}}, \mathbf {K} ]\right)\right), \tag {5}
$$
where  $\sigma (\cdot)$  denotes the sigmoid function that controls the gating level. Then, the perceived features in the two bands are fused via concatenation and  $1\times 1$  convolution and followed by a depth-level separable convolution  $W_{p}\phi W_{d}(\cdot)$ :
$$
\mathbf {Z} ^ {\prime} = W _ {p} \left(\mathcal {C} \left[ \mathbf {Z} _ {\text {D e g - I n B a n}}, \mathbf {Z} _ {\text {R e s - I n B a n}} \right]\right) + \mathbf {X}; \mathbf {Z} = W _ {p} \phi W _ {d} \left(\mathbf {Z} ^ {\prime}\right) + \mathbf {Z} ^ {\prime}. \tag {6}
$$
3.3 Gated Degradation Perception Propagation
The GDP aims to control the propagation of the perceived degradation, enabling to adaptively learn more useful features in Transformer blocks to facilitate better restoration. Given the output restoration tensor  $\mathbf{X}_{k - 1}\in \mathbb{R}^{\hat{H}\times \hat{W}\times \hat{C}}$  of  $(k - 1)^{th}$  Transformer block in one CGT and the perceived tensor  $\mathbf{P}_l\in \mathbb{R}^{\hat{H}\times \hat{W}\times \hat{C}}$  which is the output feature of one PromptDPM at  $l$ -level, the input of  $k^{th}$  Transformer block can be obtained by gating the  $\mathbf{X}_{k - 1}$  with  $\mathbf{P}_l$  by  $1\times 1$  convolution and gated control function sigmoid  $\sigma (\cdot)$  with residual learning [39]:
$$
\mathcal {G} \left(\mathbf {X} _ {k - 1}, \mathbf {P} _ {l}\right) = \sigma \left(W _ {p} \mathbf {P} _ {l}\right) \odot \mathbf {X} _ {k - 1} + \mathbf {X} _ {k - 1}. \tag {7}
$$
3.4 Learning Strategy
To train the network, two objective loss functions are adopted, including image reconstruction loss  $(\mathcal{L}_i)$  for pixel recovery and frequency loss  $(\mathcal{L}_f)$  for detail enhancement [18]:
$$
\mathcal {L} = \mathcal {L} _ {i} + \lambda \mathcal {L} _ {f}, \text {w h e r e} \mathcal {L} _ {i} = \| \hat {\mathbf {H}} - \mathbf {H} \| _ {1}; \mathcal {L} _ {f} = \| \mathcal {F} (\hat {\mathbf {H}}) - \mathcal {F} (\mathbf {H}) \| _ {1}, \tag {8}
$$
where  $\mathbf{H}$  denotes the ground truth image;  $\mathcal{F}$  denotes the Fast Fourier transform;  $\lambda$  is a weight that is empirically set to be 0.1.