Decoder Pretraining with only Text for Scene Text Recognition

Authors: shuai  zhao
yongkun  du
zhineng  chen
yu-gang  jiang
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 19:59:01

Abstract: Scene text recognition (STR) pre-training methods have achieved remarkable progress, primarily relying on synthetic datasets. However, the domain gap between synthetic and real images poses a challenge in acquiring feature representations that align well with images on real scenes, thereby limiting the performance of these methods. We note that vision-language models like CLIP, pre-trained on extensive real image-text pairs, effectively align images and text in a unified embedding space, suggesting the potential to derive the representations of real images from text alone. Building upon this premise, we introduce a novel method named Decoder Pre-training with only text for STR (DPTR). DPTR treats text embeddings produced by the CLIP text encoder as pseudo visual embeddings and uses them to pre-train the decoder. An Offline Randomized Perturbation (ORP) strategy is introduced. It enriches the diversity of text embeddings by incorporating natural image embeddings extracted from the CLIP image encoder, effectively directing the decoder to acquire the potential representations of real images. In addition, we introduce a Feature Merge Unit (FMU) that guides the extracted visual embeddings focusing on the character foreground within the text image, thereby enabling the pre-trained decoder to work more efficiently and accurately. Extensive experiments across various STR decoders and language recognition tasks underscore the broad applicability and remarkable performance of DPTR, providing a novel insight for STR pre-training. Code is available at https://github.com/Topdu/OpenOCR.

Main paper: 
 introduction: 
 Recognizing text in natural scenes, known as scene text recognition (STR), is regarded as a core task of optical character recognition (OCR). Despite significant strides in recognizing printed text images through OCR, STR encounters persistent challenges in deciphering natural text images due to complexities such as intricate background, diverse fonts and imaging conditions, etc.To confront these challenges, many studies have been dedicated to pre-training STR models, usually employing the encoder-decoder architecture on synthetic or real text images. STR encoder pretraining methods, exemplified by CCD [24] and DiG [61] , employ Masked Autoencoders [26] or contrastive learning [12] on unlabeled real text images, which drive the encoder to learn visual representations from real images, and enhancing the model's adaptability in real scenes. On the other hand, recent studies like MaskOCR [41] and TrOCR [37] train their models via two stages. For example, TrOCR is first pre-trained using hundreds of millions of printed text images, then followed by a fine-tuning on synthetic MJSynth [29] and SynthText [25] datasets. They get improved recognition results compared to the widely employed one-stage training pipeline [21, 46, 50, 64, 68, 69] . Note that both encoder and decoder are updated in these approaches. However, these approaches do not address the domain gap between synthetic and real text images. STR models trained on synthetic data, when tested on real text images, exhibit worse accuracy compared to models trained on real images [5, 17, 30] , suggesting that synthetic-trained models still struggle to capture feature representations that align well with real images. The lack of large-scale labelled real text images becomes a major obstacle for building more accurate STR models. Although some progress has been achieved in English [30] , this obstacle still exists for Chinese and many minority languages, which are even difficult to collect many unlabeled real images. Hence, it is imperative to explore novel STR pre-training methods that are less demanding on large-scale labelled real text images.Recently, we observe that visual-language models like CLIP [48] , trained on nearly 400 million real image-text pairs, adopt a multitask learning approach to simultaneously optimize image and text representations, aligning them more closely in feature space. As illustrated in Fig. 1 (a), the prompt text exhibits higher similarity to real images compared to synthetic images. To further substantiate this hypothesis, we collect 49,425 samples from both SynthText [25] and real datasets (introduced in Sec. 4.1). Each sample comprises several synthetic and real text images with the same label. We compute the similarity between these images and the prompt text one-by-one following the template "a photo of a 'label'". The sum of similarities from real images serves as the real similarity of this sample and vice versa. After inspecting all the samples and images, the similarity (after Softmax) distribution is depicted in Fig. 1 There are 29,584 samples with a real similarity higher than 0.5, constituting 60% of the total samples. The result indicates that the CLIP text features are statistically more similar to real image features rather than synthetic image features. This suggests the feasibility of deriving potential representations of real images solely from text embeddings. In other words, performing pre-training at the decoder side by leveraging the readily available CLIP. Building upon this premise, we introduce a novel pre-training method, named Decoder Pre-training with only text for STR (DPTR). Concretely, we utilize the CLIP text encoder to encode the prompt text, treating the resulting text embeddings as the pseudo image embeddings for decoder pre-training. However, as the text encoder is frozen, a fixed mapping relationship from the text to its embeddings is established. The lack of feature diversity may lead to overfitting of the pre-trained decoder. To mitigate this issue, we introduce an Offline Random Perturbation (ORP) strategy. This involves encoding natural images with the CLIP image encoder. The resulting image features are randomly cropped and added to the original text embeddings as background noise at a specified ratio. Subsequently, the decoder enjoys rich and diverse features for effective pre-training.With the pre-trained decoder, we then use it to substitute the existing STR decoder, and conduct fine-tuning with synthetic or labelled real images. After this fine-tuning, the visualization of attention maps indicates that the model's attention is not chiefly directed towards the character foreground. This phenomenon indicates that image embeddings extracted by the STR encoder contain redundant features. To remedy this issue, we introduce a Feature Merge Unit (FMU) behind the encoder. FMU employs the cross-attention mechanism to search for character features in image embeddings, and filters out redundant background features through a learnable query. This enhancement directs the model's visual attention towards the character foreground, making it easier for the decoder to decipher the character sequence.To validate the effectiveness of DPTR, we pre-trained the decoders of three typical STR models, i.e., PARSeq [5] , ABINet [21] , and NRTR [50] using DPTR. The models are then applied to English, Chinese and multi-language mixed recognition tasks. All the models get improved experimental results and PARSeq reaches state-of-the-art (SOTA) accuracy. In addition, extensive ablation experiments and visualizations also verify the effectiveness of DPTR. Contributions of this paper can be summarized as follows: the-art performance on English, Chinese and multi-language mixed datasets, showcasing its remarkable performance and great universality in a wide range of STR tasks. 
related work: 
 Scene Text Recognition. Scene text recognition (STR) has been extensively studied and existing methods [5, 15, 19, 21, 28, 46, 49, 58, 59, 67, 68, 70] can be classified into two categories: languagefree and language-aware methods. Language-free methods predict characters directly from image features, with examples including CTC-based [22] methods like CRNN [51] , SVTR [18] and Rosetta [6] , ViT-based methods like ViTSTR [3] , and methods that consider scene text recognition as an image classification problem [8, 29] .On the other hand, language-aware methods leverage external or internal-learned language representations to aid recognition. Methods in this category include using RNN or Transformer blocks for training semantic models. Typical examples are SRN [62] using a groundtruth-based pre-decoding, ABINet [21] refining predictions with contextual semantics via a cloze mask, NRTR [50] employing a left-to-right autoregressive decoding, and PARSeq [5] utilizing different attention masks for more nuanced semantic modeling.Pre-training for STR. In order to improve the performance of STR methods, some STR pre-training studies are proposed [12, 24, 41, 61, 63] . They usually include two categories: encoder and the whole model pre-training. The encoder pre-training uses massive unlabelled real images to instruct the encoder to learn real image representations, usually through self-supervised learning such as Masked Autoencoders (MAE) [26] or contrastive learning [12] . The trained encoder can be better generalized to different  
method 3.1 decoder pre-training: 
 As illustrated in the left part of Fig. 2 , decoder pre-training comprises a pre-trained CLIP text encoder and a randomly initialized decoder. It aims to effectively pre-train the decoder by using prompt text. To this end, the text encoder extracts features from the prompt text. We add perturbation to these features using an Offline Random Perturbation (ORP) module. Subsequently, the decoder learns potential representations of real images from the perturbed features and models them jointly with the prompt text.Text Encoder. In English task, we adopt the text encoder of CLIP-B and use the "a photo of a 'label'" template to generate prompt text.The prompt text undergoes encoding into a discrete text sequence using the lower-cased byte pair encoding (BPE) [48] with a coding dictionary of size 49,152. Subsequently, the text sequence is fed into the Transformer [35] to obtain the text features. Different from CLIP, which exclusively considers features from the [EOS] token, we capture features from all the tokens. Similarly, for Chinese and multi-language mixed tasks, we employ the text encoder of Multilingual CLIP-B [9] with the same template but their own language as the label, e.g., "a photo of a '标签'" in Chinese. For an input text label ŷ, the text features 𝐹 𝑡 can be:EQUATIONwhere T (•) denotes the CLIP text encoder, and 𝐿 𝑡 = 78 denotes the token length outputted by the text encoder. We directly concatenate the [EOS] token with the original 77 tokens after text projection. 𝐷 = 512 denotes the feature dimension.Offline Random Perturbation (ORP). Since the text encoder is frozen during pre-training, the obtained features are also fixed given the same prompt text, as the decoder has a fixed mapping between them. To resolve this problem, we randomly encode 10,000 natural images from COCO2017 [38] dataset using the CLIP image encoder. Subsequently, we randomly select the features of one image, and add them as background noise to the text features. The obtained image features are saved locally for facilitating the subsequent pretraining. Through this straightforward implantation, different text features can be obtained given the same prompt text, thus largely enriching the diversity of features and effectively preventing model overfitting. The perturbed features 𝐹 𝑝 can be written as:𝐹 𝑝 = 𝐹 𝑡 + 𝜆 • C(I ( x)) ∈ R 𝐿 𝑡 ×𝐷 (2)where x is the randomly selected natural image, I (•) denotes the CLIP image encoder, C(•) denotes the crop strategy that randomly selects 𝐿 𝑡 tokens from the CLIP image features, and 𝜆 is a hyperparameter controlling the weight of background noise. Decoder. For decoder pre-training, typically, the objective is to enable the decoder to search for potential real image representations from perturbed features 𝐹 𝑝 , integrate them with contextual information, and facilitate text recognition ultimately. To this end, we have chosen three language-aware STR models, ABINet [21] , NRTR [50] and PARSeq [5] , for text recognition. Their decoders are different thereby evaluations on them can reveal the universality of our pre-training. For the input text ŷ, the decoder predictions 𝑦 𝑚 can be uniformly expressed as:EQUATIONwhere 𝐷𝑒𝑐 (•) denotes the decoder, 𝑚 is an attention mask. It is a permutation-derived autoregressive (AR) mask dor PARSeq, a fixed left-to-right causal mask for NRTR, and a cloze mask for ABINet.𝑇 denotes the text length, and 𝑇 + 1 is because the [BOS] token is added to the text. 𝑆 is the size of character set, and 𝑆 + 1 is because we use [EOS] to mark the end of the sequence. Loss Function. For the given text label ŷ and the prediction 𝑦 𝑚 , the loss function can be uniformly expressed as:EQUATIONwhere L 𝑑𝑒𝑐 (•) denotes the decoder loss function. It is the arithmetic mean of the cross-entropy losses obtained from the K-attention masks for PARSeq, the cross-entropy loss of 𝑦 𝑚 and ŷ for NRTR, and the weighted average of the three losses in [21] for ABINet.With the pre-trained decoder, we then employ a fine-tuning stage as illustrated in Fig. 2 to improve the performance of existing STR models. The model comprises a randomly initialized encoder, a randomly initialized feature merge unit (FMU), and a pre-trained decoder. The image encoder extracts visual features from the input image, which are processed by FMU and then fed into the pretrained decoder for joint semantic modeling.Visual Encoder. For an input image 𝑋 ∈ R 𝑊 ×𝐻 and the patch size (𝑃 𝑤 , 𝑃 ℎ ), the image features 𝐹 𝑖 can be represented as:𝐹 𝑖 = 𝐸𝑛𝑐 (𝑋 ) ∈ R 𝑊 𝐻 𝑝𝑤 𝑝 ℎ ×𝐷 (5)where 𝐸𝑛𝑐 (•) denotes the encoder, where ABINet employs ResNet [27] and Transformer units [62] , while PARSeq and NRTR utilize Vision Transformer (ViT) [16] .Feature Merge Unit (FMU). FMU serves as an adapter to transfer the features extracted by the STR encoder to features that are more compatible with the pre-trained decoder. We first directly fine-tune the whole model without FMU. When converged, we visualize attention maps on image features 𝐹 𝑖 , it is observed that the encoder does not focus on foreground characters (see Sec. 4.3), indicating that redundant features are included. To address this issue, we introduced an FMU behind the image encoder. The FMU employs the cross-attention mechanism to select 𝐹 𝑖 features focusing on character foreground through a learnable query 𝑞, and the resulting condensed features 𝐹 𝑢 can be represented as:EQUATIONwhere 𝑀𝐻𝐴(•) denotes the Multi-head Attention, 𝐹 𝐹 𝑁 is the feed forward network, and 𝐿 𝑢 is a hyperparameter that controls the number of tokens outputted by FMU. By employing the cross-attention mechanism above, FMU adaptively selects features that can enhance the recognition accuracy. Note that smaller 𝐿 𝑢 means a more condensed representation of visual features.Decoder. As shown in Fig. 2 , we start fine-tuning by using exactly the pre-trained decoder, which is the same as the existing STR decoders in architecture. For instance, in case of PARSeq, the decoder includes the decoding layer, head, text embedding, and position query. During fine-tuning, the pre-trained decoder updates parameters according to the fused image features 𝐹 𝑢 and contextual features. The prediction 𝑦 𝑚 can be formulated as:EQUATIONNote that the decoder employs a cross-attention-based decoding scheme, where the text features are the query and 𝐹 𝑢 is the key and value. The text features are extracted following their STR methods.Loss Function. The loss function for fine-tuning is the same as that of the pre-training stage and is omitted here. 
experiment 4.1 datasets: 
 Pre-training dataset. To facilitate a fair comparison with existing methods, we generate text prompts by extracting labels from the synthetic datasets MJSynth (MJ) [29] and SynthText (ST) [25] . After de-duplication, we obtain approximately 380,000 English labels for pre-training on English. Similarly, we extract labels from the Chinese text recognition benchmark (BCTR) [11] and acquire around 700,000 Chinese labels for pre-training on Chinese. For multi-language mixed task, we obtain 150,000 labels from the synthetic datasets SynthMLT [7] , which encompasses 9 languages including Chinese, Japanese, Korean, Bangla, Arabic, Italian, English, French, and German. The labels are encoded to text embeddings using the CLIP text encoder.Fine-tuning dataset. Similar to prior research [5, 63] , for English task, we utilize MJ and ST as the synthetic data. The two datasets have approximately 17 million synthetic text images in total. The real data employed include COCO-Text (COCO) [56] , RCTW [53] , Uber-Text (Uber) [66] , ArT [14] , LSVT [55] , MLT19 [43] , ReCTS [65] , TextOCR [54] , and Open Images [33] annotations from the OpenVINO toolkit [34] , encompassing around 3 million text images depicting real scenes. For Chinese task, we adopt BCTR as the dataset, which aggregates four types of Chinese text recognition subsets: Scene, Web, Document, and Handwriting. The dataset contains about 1 million Chinese text images in total. For multilanguage mixed task, we adopt MLT17 [44] and MLT19 [43] as the datasets. They together contain about 150,000 text images, covering 10 languages including Arabic, Bengali, Chinese, Devanagari, English, French, German, Italian, Japanese, and Korean. Test benchmark. We recruit the following test sets for English task: IIIT 5k-word (IIIT5k) [42] , CUTE80 (CUTE) [2] , Street View Text (SVT) [57] , SVT-Perspective (SVTP) [45] , ICDAR 2013 (IC13) [32] , and ICDAR 2015 (IC15) [31] .For Chinese task, we utilize the test sets of BCTR, which are also further categorized into four subsets: Scene, Web, Document, and Handwriting. For multi-language mixed task, we use the validation set of MLT17 for test only due to the unavailability of MLT19 test data. This set encompasses 6 subsets covering 9 languages: Chinese, Japanese, Korean, Bangla, Arabic, and Latin (Italian, English, French, and German).The input image is resized to 32 × 128 for both English and multilanguage mixed tasks. For Chinese task, we resize the input image to 32 × 256. The patch size is set to 4 × 8 for all languages. The maximum text length is restricted to 25 characters. We pre-train the model on 2 NVIDIA RTX A6000 GPUs with a batch size of 512, and then fine-tune it with a batch size of 384. Hyperparameters include an initial learning rate of 7e-4 without weight decay.We conduct ablations to verify the effectiveness of the proposed decoder pre-training, ORP and FMU. For brevity, 𝑺𝒚𝒏𝒕𝒉 denotes the method pre-trained with synthetic images.The effectiveness of decoder pre-training. We conduct a comparative experiment with 𝐵𝑎𝑠𝑒, 𝑆𝑦𝑛𝑡ℎ, and 𝐷𝑃𝑇 𝑅 sharing the same model structure and experimental setup. The primary distinction of the three methods lies in: 𝐵𝑎𝑠𝑒 trains directly on the real datasets without pre-training, 𝑆𝑦𝑛𝑡ℎ undergoes pre-training with synthetic images before fine-tuning on real images, and 𝐷𝑃𝑇 𝑅 pre-trains with a photo of a 'EXATAS' a photo of a 'basted' text only before fine-tuning on real images. Experimental results presented in Tab. 1 show that 𝐷𝑃𝑇 𝑅 improves average accuracy by 0.9% compared to 𝐵𝑎𝑠𝑒 and by 0.6% over 𝑆𝑦𝑛𝑡ℎ. Furthermore, when the pre-trained decoder is frozen during fine-tuning, the model experiences only a marginal accuracy decrease of 0.7%, indicating the effectiveness of the text pre-trained decoder.As shown in Fig. 3 , we compare the pre-trained decoder attention maps between 𝑆𝑦𝑛𝑡ℎ and 𝐷𝑃𝑇 𝑅. The results reveal that 𝑆𝑦𝑛𝑡ℎ exhibits more pronounced attention drift, suggesting a higher susceptibility to interference from intricate backgrounds in real images. In contrast, attention of 𝐷𝑃𝑇 𝑅 is mostly located on the corresponding characters, indicating a more accurate alignment between image embeddings and text embeddings. We also visualize the character distribution of 𝑆𝑦𝑛𝑡ℎ and 𝐷𝑃𝑇 𝑅, as shown in Fig. 5 , where each circle is a character and its color represents the character category. The symbol '+' denotes 'n' in the image labelled 'nVIDIA', while 'x' represents 't' in the image labelled 'tO'. Due to overlapping with the background, 𝑆𝑦𝑛𝑡ℎ incorrectly predicts 'n' as '2'. Similarly, character 't' is obscured such that 𝑆𝑦𝑛𝑡ℎ misses it. Meanwhile, it incorrectly identifies 'O' as '0'. In contrast, for 𝐷𝑃𝑇 𝑅 the two misidentified characters fall into the correct character categories. Since 𝑆𝑦𝑛𝑡ℎ and 𝐷𝑃𝑇 𝑅 only differ in the pre-training step, the results in Tab. 1, Fig. 3 and Fig. 5 clearly indicate the effectiveness of the proposed decoder pre-training.The effectiveness of ORP. To investigate the impact of adding random perturbation to decoder pre-training, we ablate the noise ratio that balances the weight of text features and randomly selected visual features. The results are presented in Tab. 2. The model's accuracy first experiences an increase when the noise ratio is small, and a decrease when the noise ratio further goes up. This phenomenon is attributed to that excessive noise alters the text features too much, leading the decoder to acquire incorrect representations.In contrast, introducing a small ratio of noise effectively prevents model overfitting, meanwhile without significantly altering the distribution of text features. As depicted in Fig. 4 , the distributions of text features are significantly altered when setting 𝜆 = 0.5 or 𝜆 = 1, which display quite different shapes compared to the raw distribution (𝜆 = 0). Conversely, for 𝜆 = 0.01, the added noise is minor, resulting in little deviation from the raw distribution. For 𝜆 = 0.1, although the distribution changes a lot from the raw, it still holds a certain geometric shape. This analysis vividly illustrates that introducing a small noise ratio through ORP enriches the diversity of pre-training features, thereby improving the performance.The effectiveness of FMU. It is not surprising that the features from the CLIP text encoder are different from those extracted by the image encoder employed in fine-tuning. FMU serves as an adapter to bridge this gap. To gain insights into the role of FMU, we first omit the FMU module and fine-tune the model. Visualizations of the last self-attention layer of the image encoder are shown in Fig. 6 (a), it is apparent that the attention is not solely focused on the character foreground. Instead, a portion of attention is directed towards the background. This observation suggests that not all the extracted features are useful and positively contribute to the recognition. Some of them are redundant and may hinder the recognition. With this observation, we propose FMU to mitigate the feature redundancy and address the issue of attention not focusing. As depicted in Fig. 6(d) , when FMU is equipped, hotspots of the attention maps are mostly concentrated on character foreground, which vividly indicates that FMU successfully extracts useful image features and discards redundant ones, explaining why accuracy gains are obtained. In addition to using the proposed FMU to distinct features, there also are other ways to fuse these features. We ablate two typical of them. The first is 𝐶𝑢𝑡 that directly truncates the first 𝐿 𝑢 tokens from the image encoder. While the second is 𝑃𝑜𝑜𝑙 that denotes pooling all the tokens into 𝐿 𝑢 tokens using AdaptiveAvg-Pool1d. Their comparing results (𝐿 𝑢 =26) are given in Tab. 3. 𝐶𝑢𝑡 exhibits the worst performance and 𝑃𝑜𝑜𝑙 also reports an accuracy To further validate the performance of DPTR, we conduct evaluations on Art, COCO, and Uber datasets, which are typically more challenging compared to previous benchmarks. As depicted in Tab. 6, compared to PAPSeq, 𝑃𝐴𝑅𝑆𝑒𝑞 + 𝐷𝑃𝑇 𝑅 trained on synthetic data achieves accuracy improvements of 1.7%, 5.0%, and 1.3%, and when trained on real datasets, the improvements are 0.5%, 1.5%, and 3.2%, respectively. It achieves the best accuracy among five of the six comparisons. Similarly, 𝑁 𝑅𝑇 𝑅 + 𝐷𝑃𝑇 𝑅 and 𝐴𝐵𝐼 𝑁 𝑒𝑡 + 𝐷𝑃𝑇 𝑅 both report improvements compared to their raw counterparts.Chinese Benchmarks. We also train a DPTR for Chinese through Multilingual CLIP and evaluate it on BCTR. As depicted in Tab. 7, compared with MaskOCR-L, the previous SOTA method 𝑃𝐴𝑅𝑆𝑒𝑞 + 𝐷𝑃𝑇 𝑅 reports an average accuracy of 80.7%. It is also the new state of the art. 𝑃𝐴𝑅𝑆𝑒𝑞 + 𝐷𝑃𝑇 𝑅 gains accuracy improvements of 3.8% and 2.8% on Scene and Web, respectively. However, it is worse than MaskOCR-L on Doc and Hand. This is because Doc is synthesized using a text rendering tool, and the Handwriting style in Hand is more similar to synthetic text images. Both are less similar to real scene text. The observation indicates that 𝑃𝐴𝑅𝑆𝑒𝑞 + 𝐷𝑃𝑇 𝑅 still exhibits advantages in Chinese STR. Meanwhile, similar accuracy variants are also observed when 𝑁 𝑅𝑇 𝑅 + 𝐷𝑃𝑇 𝑅 v.s. NRTR, and 𝐴𝐵𝐼 𝑁 𝑒𝑡 + 𝐷𝑃𝑇 𝑅 v.s. ABINet. These results demonstrate the effectiveness of DPTR in Chinese recognition.Multi-language Mixed Dataset. Similarly, a multi-language mixed DPTR is trained using Multilingual CLIP and tested on MLT17. The results are given in Tab. 8 (each language is abbreviated using its first three characters). Compared with the raw implementation of NRTR, ABINet, and PARSeq, 𝑁 𝑅𝑇 𝑅 + 𝐷𝑃𝑇 𝑅, 𝐴𝐵𝐼 𝑁 𝑒𝑡 + 𝐷𝑃𝑇 𝑅, and 𝑃𝐴𝑅𝑆𝑒𝑞 + 𝐷𝑃𝑇 𝑅 gain accuracy improvements of 1.9%, 1.1%, and 1.6%, respectively. Note that 𝑁 𝑅𝑇 𝑅 + 𝐷𝑃𝑇 𝑅 and 𝑃𝐴𝑅𝑆𝑒𝑞 + 𝐷𝑃𝑇 𝑅 report improvements on all the evaluated languages, while 𝐴𝐵𝐼 𝑁 𝑒𝑡 + 𝐷𝑃𝑇 𝑅 reports slightly lower accuracy on Ara and Kor. This is because ABINet relies on external language models, which are not readily available for minority languages thus the side affection may be more apparent. Nevertheless, the experiment validates that DPTR can still take effect for a recognition  
conclusion: 
 In this study, we have presented DPTR, a novel decoder pre-training approach for STR. We have observed that embeddings extracted from the CLIP text encoder are more similar to embeddings of real text images rather than commonly employed synthetic text images. Therefore, DPTR is featured by leveraging CLIP text embeddings to pre-train the decoder, offering a new paradigm for STR pretraining. We have developed ORP, a dedicated data augmentation to generate rich and diverse embeddings and make our decoder pre-training effective, and FMU to condense the embeddings of real text images and make them better aligned with the pre-trained decoder. Extensive experiments across various decoders and languages demonstrate the effectiveness of DPTR. Our exploration above basically validates that CLIP can be utilized to enhance the training of STR models. In future, we plan to investigate the more thorough utilization of large pre-trained models like CLIP, and activate the rich knowledge contained by them to further improve the accuracy of STR as well as other OCR-related tasks. 
 
 Appendix: ACM MM, 2024, Melbourne, Australia Shuai Zhao, Yongkun Du, Zhineng Chen, and Yu-Gang Jiang 


Decision: The authors did a good rebuttal. The reviewers unanimously recommend acceptance in the final rating. After checking the rebuttal, the review, and the paper, the AC agrees with this assessment.

Reviews: review: paper_summary: paper_summary: This paper introduces Decoder Pretraining with Only Text for Scene Text Recognition (DPTR), leveraging CLIP's text embeddings to bridge the gap between synthetic and real images. DPTR uses text embeddings as visual proxies during pre-training and incorporates an Offline Randomized Perturbation (ORP) strategy to enhance these embeddings with image information. In the fine-tuning stage, a Feature Merge Unit (FMU) focuses on character regions, resulting in improved accuracy and efficiency, with extensive experiments demonstrating DPTR's superior performance across various decoders and languages.
strength_weakness: strength_weakness: The paper showed some improvement compared with baselines.
limitations: limitations: 1. The improvement in performance compared to the baselines is relatively modest.

2. The novelty of the approach appears limited. Previous works such as ABINet and SEED have already utilized external pretrained language models to leverage linguistic priors, which are trained with text-only data and enhance scene text recognition models. Pretraining the decoder with text, in this context, seems like an incremental advancement.
paper_summary: paper_summary: The paper introduces a novel pre-training method named DPTR (Decoder Pretraining with only text for STR) for scene text recognition (STR). Unlike traditional methods that rely on synthetic datasets or real images, DPTR leverages text embeddings produced by the CLIP text encoder as visual embeddings to train the decoder. The proposed method includes an Offline Randomized Perturbation (ORP) strategy, which enriches text embeddings with image embeddings to prevent overfitting. Additionally, a Feature Merge Unit (FMU) is introduced during fine-tuning to enhance the decoder's focus on character regions within text images. The experimental results demonstrate that DPTR achieves state-of-the-art performance across various decoders and multi-language STR tasks.
strength_weakness: strength_weakness: 1. The paper proposes an interesting approach to STR pre-training by using text embeddings from CLIP, eliminating the need for text images during pre-training. This is a significant departure from traditional methods that rely heavily on synthetic or real text images. By using text embeddings as visual embeddings, the method effectively addresses the domain gap between synthetic and real images

2. The introduction of the Offline Randomized Perturbation (ORP) strategy helps in enriching text embeddings and prevents the model from overfitting, which enhances the robustness of the pre-training process. The Feature Merge Unit (FMU) helps focus the model’s attention on the character regions during fine-tuning, thereby improving accuracy and efficiency.

3. The experiments and ablations in this paper are detailed and comprehensive, showing improved performance and achieving state-of-the-art results.
limitations: limitations: 1. With the innovative training approach, the design of the model structure is relatively simple.

2. Some additional evaluation should be included. Since the CLIP model is trained on large-scale image-text pair datasets, the proposed method in this paper may not be robust to noise. The authors can compare the performance on occlude datasets such as HOST, WOST, and WordArt to enhance persuasiveness.

3. Discussion about the computation complexity of the proposed ORP and FMU should be included.
paper_summary: paper_summary: This paper focuses on the pre-training of scene text recognition and proposes a pre-training method for the decoder where the decoder tries to predict the word based on the text embedding from the CLIP. Furthermore, the Offline Random Perturbation and Feature Merge Unit are designed to improve the robustness of the model.
strength_weakness: strength_weakness: 1. Pre-training the decoder is novel for the scene text recognition;
2. Using the text embedding from the CLIP is neat since the visual and semantic features are aligned in the CLIP.
limitations: limitations: 1. More details about the usage of F_p can be provided, will it work as the key and value of the attention mechanism?
2. The proposed method seems can not be applied to the CTC-based decoder;
3. I wonder if initializing the image encoder with the CLIP can achieve a better performance;
4. The compared methods in Table 6 is not up-to-date;
5. Since the paper focuses on the pre-training of text recognition, treating the FMU as a contribution seems not appropriate, since it only be adopted for fine-tuning;
6. Lacks a comparison between the proposed method with other unsupervised pre-training of STR.
paper_summary: paper_summary: This paper proposes a novel method (DPTR) for pre-training the decoder of text recognition models relying only on text. DPTR employs the embedding generated by CLIP's text encoder as the visual embedding of the scene text, which is fed into the decoder.  In order to enhance the diversity of the embedding produced by the text encoder, the Offline Randomized Perturbation (ORP) strategy is proposed, which incorporates the embedding of images produced by the image encoder of CLIP. DPTR pre-training of the existing sota method shows a major performance improvement.
strength_weakness: strength_weakness: 1. The proposed DPTR method for decoder pretraining with only text is overall insightful. DPTR exploits CLIP's alignment of visual and text, and thus simulates visual embedding using a text encoder.
2. The experiments are comprehensive, including performance in English, Chinese, and multilingualism, as well as detailed experiments on the ablation of individual modules.
limitations: limitations: 1. The proposed FMU module is a cross attention module, I can't quite figure out why it's called a Feature Merge Unit as it doesn't seem to have a feature fusion function. 
2.  What are the advantages of FMU over other approaches such as linear, mlp, convolution layers? Comparative experiments are missing.
3.  Missing references to Scene text recognition related articles in CVPR 2024. Such as  "OTE: Exploring Accurate Scene Text Recognition Using One Token", "Multi-modal In-Context Learning Makes an Ego-evolving Scene Text Recognizer" and "An Empirical Study of Scaling Law for OCR ".