MaskOCR: Scene Text Recognition with Masked Vision-Language Pre-training
Authors: Pengyuan Lyu
Chengquan Zhang
Shanshan Liu
Meina Qiao
Yangliu Xu
Liang Wu
Kun Yao
Junyu Han
Errui Ding
Jingdong Wang
Venue: Accepted by TMLR ()

Abstract: Text images contain both visual and linguistic information. However, existing pre-training techniques for text recognition mainly focus on either visual representation learning or linguistic knowledge learning. In this paper, we propose a novel approach to unify vision and language pre-training in the classical encoder-decoder recognition framework. We adopt the masked image modeling approach to pre-train the feature encoder using a large set of unlabeled real text images, which allows us to learn strong visual representations. In contrast to introducing linguistic knowledge with an additional language model, we directly pre-train the sequence decoder. Specifically, we transform text data into synthesized text images to unify the data modalities of vision and language, and enhance the language modeling capability of the sequence decoder using a proposed masked image-language modeling scheme.
Significantly, the encoder is frozen during the pre-training phase of the sequence decoder. Experimental results demonstrate that our proposed method achieves superior performance on benchmark datasets, including Chinese and English text images. The code for our approach will be made available.

Published in Transactions on Machine Learning Research (05/2024)
MaskOCR: Scene Text Recognition with Masked Vision- Language Pre-training
Pengyuan Lyu, Chengquan Zhang, ShanShan Liu, Meina Qiao, Yangliu Xu, Liang Wu, Kun Yao, Junyu Han, Errui Ding, Jingdong Wang VIS, Baida Inc.
lepyuan @mmall com
(zhangchengguan, Bushanshan07, giaomeina, zugangliu, uuliang11 yaokun01, hanjunyu, dingerrui] @boida. com wangjingdong Sengiinggoongnttlooo.okkk Woutlook.coo com
Reviewed on OpenReview: https: // opwpr//veeee.e.....mmmmm net/ per/forum? id- KNAWoKKps3
Abstract
Text images contain both visual and linguistic information. However, existing pre-training techniques for text recognition mainly focus on either visual representation learning or lin- guistic knowledge learning. In this paper, we propose a novel approach MaskOCR to unify vision and language pre-training the classical encoder-decoder recognition framework. We adopt the masked image modeling approach to pre-train the feature encoder using large set of unlabeled real text images, which allows to learn strong visual representations. In contrast to introducing linguistic knowledge with additional language model, we directly pre-train the sequence decoder. Specifically, we transform text data into synthesized text images to unify the data modalities of vision and language, and enhance the language mod- eling capability of the sequence decoder using a proposed masked langual modeling seeeeeeeee
scheme. Siguificantly, the encoder is frozen during the pre-training phase of the sequence decoder. Experimental results demonstrate that our proposed method achieves superior performance benchmark datasets, including Chinese and English text images.
1 Introduction
Text recognition, which involves transcribing visual information into text, is a crucial technology for bridging vision and language. It has a wide range of applications, including visual search, document digitization, and more. Many significant. progres has been made in the field of text. recognition. However, remains a challenging task due to the difficulty of recognizing fine-grained categories of text, which can vary in fonts, colors, and other factors, coupled with the relative scarcity of labeled real-world data.
As an alternative
approach, synthetic data has been used in previous studies (e.g.. Jaderberg et al.
(014a)): Max Jaderberg
and Karen Simonyan and Andrea Vedaldi and Andrew Zisserman ; (Shi et al.
(201 7a 2019): Yu et al.
(2 Fang et al. (2021]), and meaningful results have been obtained. Nonetheless, recognition performance is still restricted by the domain gap between synthetic and real-world data.
Efforts have been made reduce the need for real labeled data through pre-training, which can be broadly divided into two categories: strengthening visual representation learning with unlabeled real images, and introducing language priors with a language model. In previous studies (e.g., Liu et al. 2002 Yang et al. (2022)), contrastive learning and masked image modeling technologies were employed for pre-training using large amount of unlabcled image data to obtain better visual representations. In other studies (o.g., Qiao el al. ) Fang et al. (2021)), ), a pre-trained language model was used to guide recognition model learning or correct recognition model predictions. These methods have achieved promising results, thanks to their incorporation of vision or language priors. However, there are two limitations these approaches. First, they tend focus solely on either visual representation learning or linguistic knowledge learning, while text images inherently contain both visual and linguistic information. Neglecting either type of prior knowledge may

---

Published in Transactions on Machine Learning Research (05/2024)
sraiingg
6
TExt
Longu-age Fre tarning
011 Imsze
Pretum Encockr
Secauerci Decoder
Modil
-Ted
Visio Fre-
Language Fre
Ted
Figure 1: Comparison of previous pre-training methods. A classical recognition model consists of a feature encoder and a sequence decoder, responsible for extracting visual representations and decoding text sequences from those representations, respectively. Previous pre-training methods have primarily focused on pre- training the feature encoder for better visual representation learning (a) or equipping a separate language model to introduce language priors (b). In this paper, we take different approach by simultancously and naturally integrating vision and language pre-training into the encoder-decoder recognition model (c).
result in loss of accuracy. Sccond, previous studics (e.g., Qiao et al. 2020 Fang et al. (022) introduced language priors into the recognition model with delached language model that blocked gradient flow between the recognition and language models, potentially leading to suboptimal performance.
In this paper, we explore the utilization of both visual and language priors through pre-training to enhance text. recognition performance. Our approach unifies vision and language pre-training within the classical encoder-decoder recognition framework. Specifically, we pre-train the feature encoder on a large set of unlabeled real text images using masked image modcling, which enables the extraction of better visual representations through self-supervised mechanisms. Additionally, to improve language modeling capabilities. To achieve this, we first transform the text corpus into hesized text images to unify the data modalities and then use a proposed masked image-langusee modeling technology to learn the linguistic representation. During language pre- teettraining,, we fix the pre-trained feature encoder and only update the sequence decoder. This strategy benefits from the language pre- vre-training task, which explores language rules while the pro-trained encoder is not affected by the synthesized text image.
directly pre-train the sequence decoder
We validate the effectiveness of proposed approach on Chinese and English text images through exten- sive experiments and detailed discussion. Our proposed method achieves state-of-the-arr performance and significantly surpasses previous methods, particularly on Chinese benchmarks.
The main contributions of this paper can be summarized as follows:
We propose a masked vision-language pre-training method that unifies vision and language repre- sentation learning within the classical encoder-decoder recognition framework.
Our vision-language pre-training approach contributes to a significant. improvement in text recog- nition. Specifically, with the proposed pre-training technology, we observe a 5% performance gain on the challenging Chinese benchmark BCTR. Furthermore, compared to previous tiio--oftthh-rrtt methods, we achieve superior results on both Chinese and English datasets.
2 Related Work
Text recognition. Text recognition be achieved through various approaches, including character- based Wang et al. (2011): : Jaderberg et al. (2014b); Lyu et al. : Liao et al. (2019), word-based Max Jaderberg and Karen Simonyan and Andrea Vedaldi and Andrew Zissorman (2016), and sequence-based Shi
el al. (2017a 2019); Yu et al. (2020); Fang et al. (202);; Xue el al. (2021) methods. Character-based methods recognize text images by performing character localization, classification, and grouping. Word- basod methods treat each word as a category and recognize text through image classification. Sequence-
based methods treat text recognition as a sequence labeling problem. This approach uses methods such
CTC Graves et. al. (2006) and attention mechanism Shi et. al. (2019); Lyu et al. (2019): Yu et. al. (2020):
2

---

Published in Transactions on Machine Learning Research (05/2024)
Fang et al. (2021) to align the input image patch sequence with the output character sequence. Recently, the sequence- solution has been the most widely studied framework due to its flexibility and ease of ground-truth labeling. The requence-based architecture consists of two main modules: feature encoder and sequence decoder. The feature encoder learns visual representations for text images, while the sequence decoder maps the representations into character sequences, with or without the aid of a language module.
Pro-training. Representation pre-training has been shown to be beneficial for improving downstream tasks, such supervised or self-supervised pre-training on ImageNet for computer vision, and self-superviser pre- training for natural language processing. Self-supervised pre-training, which does not require labeled data, has gained significant attention in recent years, Several approaches have been proposed to achieve self- supervised pre-training, such as siamese networks trained using contrastive learning in He et al. (2020):
Chen et al. (2020 2021ep; and masked autoencoders for both vision and NLPP models in He et al. (2022) Bao al. 2022):: Devlin al. (2019), where the models improve their representations by predicting masked content. In the method proposed by Rust et al. (2023), pre-training is conducted
text images using Masked Image Modeling to enhance the feature extraction capabilities of the visual encoder, which is similar to the first stage of our method. However, key divergence from Rust et al. (2023) lios in our method's 's incorporation of Masked Image-Laguuage Modeling for additional pre-training of the decoder. This addition aims to enhance the semantic reasoning ability of the decoder when interpreting character sequences.
In recent years, there has been significant interest the field of vision-language pre-training, whereby models are trained to jointly understand both visual and textual information. Various approaches have been proposed for aligning visual and language representations, including image-text contrastive learning Radford
et al. 2021);; Jia et al. (2021), ,, image-text matching Li et al. (2021a); Wang et al. (2021b), masked language
| modeling Zhou et al. 20200; Wang et al. (2021a), and language modeling Li et al. (2022); Yu et al. (2022).
These pre-trained models can then be fine-tuned for specific rewnst tasks, such as visual question answering, image captioning, image retrieval, and text-to-image generation.
Pre-training for text recognition. Several methods have been studied to employ pre-training for text recognition. In Jaaeeberg et al. (2014a); Max Jaderberg and Karen Simonyan and Andrea Vedaldi and Andrew Zisscrman (2016); Shi et al. (2017a] 2019); Yu et al. (2020); Fang et al. (2021); Li et al. (2021b). the recognition model is pre-trained on synthesized data using supervised learning. However, despite this approach, the recognition performance is still limited by the domain gap between hesthesize and real data. To improve the recogmition accuracy, some methods use unlabeled real images or text corpus. For instance, Liu et al. (2022) introduces self-supervised contrastive pro-training to learn representations from the input real images, while Yang et al. (2022) integrates contrastive learning and masked image modeling to pre-train the recognition model. Notably, the aforementioned methods are mainly focused visual representation learning. In Fang et al. (2021) and Qiao et al. (2020). separate pre-trained language model is used to enhance the language modeling capability.
Compared to previous techniques that concentrate solely on either visual or language representation learning, our proposed approach combines vision and language pretraining into a single, integrated model. This ap- proach naturally integrates with the encoder-decoder recognition model, leading to improved representation quality for both vision and language.
Vision-language pre-training for document understanding. Some document understanding meth- ods utilize vision-language pre-training to model both visual and linguistic information. For instance, the MVLM module in LayoutLM Xu et al. (0200) is proposed to align visual and textual representations, while VLPT Song et al. (2022) uses image-Lext contrastive learning to enhance visual representations.
Our proposed vision-language pre-training approach differs from LayoutLM and VLPT in several ways. Specifically, LayoutLM takes multi-modal information (visual tokens and textual tokens) input to the network and learns to process both image and text information. In contrast, our approach only lakes the image as input and utilizes language to assist in learning the decoder. Moreover, our approach differs from VLPT in that VLPT mainly uses masked language modeling and cross-model contrastive schemes to learn
3

---

Published in Transactions on Machine Learning Research (05/2024)
Character queries
Encoder
Decoder
Linear layer
R RONALDO
Figure 2: Encoder-decoder transformer for text recognition. The encoder extracts sequence of patch representations, and the decoder maps the patch representations to sequence of representations, followed by a linear layer to recognize the sequence of characters.
the image encoder, while our approach employs masked image modeling to learn the text-image encoder and language pre-training with masking for the decoder.
3 Approach
In this section, we will first present our encoder-decoder recognition model which is equipped with trans- former. After that, the visual pre-training of encoder and the language pre-training of decoder will be described respectively.
3.1 Encoder-Decoder Transformer for Text Recognition
We adopt the encoder-decoder transformer architecture for text recognition. The encoder extracts sequence of patch representations and the decoder maps the patch representations to text representations. Figure 2 illustrates the encoder-decoder transformer architecture.
Encoder. The encoder receives an image, E R as the input. We partition the image vertically into of M vertical patches.(p1. ....... Pax]. The size of each patch H W/M. We process the flattened patches using linear projection to get patch embeddings. We add the 1D positional embeddings, which is enough as partition the images vertically. We use the VIT Dosovitskiy et al. (2021), consisting of a sequence of multi--ead self-aliention and FFN units, as the encoder and learn the patch-level representations, for the text image.
F
Decoder. We form the text recognition decoder by following the decoder style of the DETR Carion et al. 2020 that originally designed for object detection. The decoder transforms the N input embeddings, C1, c2. Cx), called character queries, into output embeddings in parallel, which then independently mapped into characters through linear classifier.
Loss. The labels of the text recognition task are ordered strings. So unlike the DETR which uses the matching mechanism to assign ground truth for each output, we directly allocate the label of each character prediction in order. We denote the character predictions by Y = [y1 Y2 ... yv]. Assuming N larger than the number of characters in the text image. We consider the ground truth Y [yī y2 ... yy] padded with end-of-sentence symbol [EOS]. The loss function is formulated follows,
L+
1
(Y,
CE(y,.r),,
(1)
L
4

---

Published in Transactions on Machine Learning Research (05/2024)
Mask queries
Visible patches
Latent contextual regressor
Encoder
Image decoder
Input
Aligmment
Encoder
1
Masked patches
Visual pre-training encoder. We adopt masked image modeling approach to pre-train the encoder for text image representation learning, In this example, six image patches (top) are visible patches, and the other four (bottom) to be predicted are masked patches.
Figure 3:
where CE(.,)) is the cross-entropy loss. L is the number of characters in the text image. To balance the number of [EOS] and other characters, we only employ the loss function on the characters as well as the first EOS).
3.2 Masked Image Modeling on Encoder for Visual Pre-training
We follow the masked image modeling framework, which is recently studied for general image representation pre-training, and adopt the context autoencoder-style method Chen et al. (2022) to pre-train the encoder for text image representation learning. The context autoencoder separates the decoding role from the encoder and drives the encoder to focus representation learning, which better choice for our visual representation learning.
The encoder pre-training process is given as follows. The input text image is randomly masked with given masking ratio and the remaining parts, which are termed visible patches are sent to the encoder, generating the representations of visible patches. Then, the representations visible patches are fed into a latent contextual regressor with mask queries, predicting the representations for masked patches Zn which is expected be close to the representations Zü of masked patches directly computed from the encoder. Last, the representations of the predicted masked patches go into the image decoder, predieting the targets Tm. Figure 3 illustrates the encoder pre-training architecture.
We use the patch RGB values, processed with layer normalization (Gaussian normalization), to form the targets. The loss function for encoder pre-training is a combination of representation alignment. loss and prediction loss and given as follows,
(T m. Tú + l2 (z Zin).
(2)
Here, both losses l(ss)) and .))))) are the MSE loss. is the trade-off parameter and set to 0.05 in our implementation.
3.3 Masked Image-Language Modeling on Decoder for Language Pre-training
Unifying the vision and language pre-training in a single model is not easy, since the granularity of image and text are different and the semantic of vision and language are not aligned. The core designs of our method overcome this problem are that: 1) we transfer the text image via text to image generation, thus the modality difference between image and text is eliminated. 2) We propose masked image-language modeling,
5

---

Published in Transactions on Machine Learning Research (05/2024)
Character queries
Encoder
#
Decoder
Linear layer
u
Figure 4: Language pre-training on decoder. The whole pipeline similar with the one in Figure 2 The difference is that the input to the encoder are the visible patches. The visible patches are formed by masking the image patches that correspond to the target characters (the patch number may be greater than the character number). The input representations to the decoder and zero representations added to the positions (gray boxes) where the masked patches are. The prediction targets are the characters that are masked. The encoder is fixed when performing decoder pre-training.
combination of encoded representations
which randomly masks some characters of the input image, and prodicts the masked characters via the unmasked part. 3) We also design a sequential pre-training mechanism by freezing the pre-trained encoder to handle the domain gap between real image and synthesized image, that the language presentation is enhanced while the pre-trained encoder which learns better visual representation is not affected.
In detail, we transform the text. data to images via a public synthesis tool Text Render Given a text corpus, some fonts, and some background images, the synthesized text images as well as the location annotation of characters can be generated.
To further enhance the language modeling capability of the decoder, we adopt the idea of masked language modeling and introduce masked image-language modeling scheme. As shown in Figure 4 randomly mask some characters and accordingly the patches, and send the remaining visible patches to the encoder, obtain the representations of visible patches, F.. Then, we insert the zero representations Fm to the positions corresponding to the masked patches, and feed the combined representations the corresponding positional embeddings added into the decoder, predicting the text: Y [y1 Y2 yN].
F, F..] with
The loss function similar BERT Devlin et al. 2019 and is merely about masked characters. We formulate it as follows,
(Y Yi) 1 Σ CE(yn,
where L is the number of masked characters, and (ni, ..,..., , nz} are the positions of the masked characters.
Considering the style of synthesized lext images might be different from the real text images, keep the encoder (learned from encoder pre-training) not updated and only optimize the decoder, that the pre-training stage of the decoder does not affect the representation quality of the pre-trained encoder.
4 Experiment
4.1
Datasets
Chinese text line images. The pre-training set consists of 100 million unlabeled text line images collected from practical scenarios for visual pre-training, and 100 million synthetic text line images for language pre- training. The real images are collected from documents and street view, and the text in them almost in Chineso. We collect text corpus from Chinese corpus / and generate 100 million images with 64 commonly used fonts using Text Render. Specifically, for each synthetic sample, the text transcription as well as the character bounding boxes are given.
httpp/////shuuubccc/oaaaaaerr/es renderer https://github.

---

Published in Transactions on Machine Learning Research (05/2024)
We first pre-train the encoder and decoder serially on the collected real images and the synthetic images, and then finetune our model on a large-scale Chinese text image benchmark BCTR Chen et al. (2021b). BCTR consists of four subsets (scene, web, document, and handwriting) and provides 1.4 million fully labeled
images in total. The scene subset (Sen) is derived from some scene text datascts, including RCTW Shi et al. (2077b) ReCTS Zhang el al. (2019), LSVT "Sun et al. (2019) ArT Ching el al. (2019), and CTW Yuan el al. (2019h resulting in 636,455 images. The web subset (Web) is constructed based on the MTWI He et al.
(2018) dataset and contains 140589 text images. The document subset (Doc) is composed of 500000 synthetic
text images generated by Text Render in document style. The handwriting subset (Hand) is collected from
handwriting dataset SCUT-HCCDoc Zhang et al. 2020), and 116643 text images are included.
English text word images. We follow Yang et al. 2022 and collect about 15.8 million unlabeled English word images from CC-OCR Yang et al. (2021) for visual pre-training. In addition, we also synthesize 100 million English word images for language pre-training Similarly, we collect corpus from Wikifext 103 Merity
ct al. 2017 and generate synthetic images with Text Render and 10 commonly used English fonts.
Following Shi et al. (2019] : Yu et al. (2020); ; Fang et al. (2021] Wang et al. (2021c); c Zhang et al. (0022)
two synthetic datasets MJSynth Jaderberg et al. (2014a) and SynthTText Gupta et al. (2016) are used for
the training of downstream recognition lasks. Besides, we also collect 2.78 million real labeled images from TextOCR, (iighh et al. (2021) and Open Images Dataset v5 3 as Yang et al. (2022) to verify the effectiveness of pre-training when finetuned on real imagos. We evaluate our model on six public scene text datasets:
ICDAR 2013 (IC13) Karatzas al.] (2013), Street View Text (SVT) Wang et al. (2011), IIIT5K Woods (IITT5K) Mishra et al. (2012)), ICDAR 2015 (IC15) Karatzas et al. (2015), Street View
(SVTP) Phan et al. (2013), and CUTES0 (CUTE) Rismumawan et al. (2014)). The samples in the first three datasets are all regular text images and the remaining datasets may contain perspective or curved text images.
4.2 Implementation Details
Encoder-decoder transformer. The image patches are fed into a linear projection layer and then sent to the ViT. Two ViT structures are studied: VIT-S (12 transformer blocks with dimension 384), and VIT-B (12 transformer blocks with dimension 768). The decoder consists of four decoder layers, each of which includes self-attention unit, cross-attention unit, and an FFN unit. Each attention module is 12-head attention with dimension 384.
We train the encoder-decoder transformer with AdamW optimizer Loshchilov & Hutter 2019 cosine learn- ing rate docay Dya Loshchilov and Frank Hutter (0177) a weight decay of 0.05, a drop path ratio of 0.1, and a batch size of 512. When the model is trained from scratch, the learning rate set to le - 3. Otherwise, the model is optimized with an initial learning rate of le 4. We set the training epochs as 120 and 20 for the Chinese text line recognition model and the English word recognition model with warm-up of 5 epochs and 0.5 epochs respectively.
Visual pre-training on encoder. The latent contextual regressor consists of four regressor layers. Each layer includes a cross-attention unit and an FFN unit. The image decoder consists of four layers, and each layer includes self-attention unit and an FFN unit. Each attention module is also 12-head attention with dimension 384. Following He al. (0022), we use the normalized pixel values of each masked patch as task.
We optimize the model with AdamW optimizer and set the learning rate with the linear learning rate scaling rule Goyal et al. (2017): : Ir = base_h, Ir x batchsize/ /256. By default, the base Ir is set to 1.5e- 15--4 4 with cosine learning rate decay and a 0.5 epoch warm-up. We train the encoder for 10 epochs and 30 epochs for Chinese data and English data pre-training, with the batch size being 4096.
Language pre-training on decoder. We mask some characters with a ratio of 0.15 and accordingly mask the patches that contain the characters. This might lead to that different mumber of patches are masked for
Laaeeeinnnnnneeeeeee sacees_____te___ text
7

---

Published in Transactions on Machine Learning Research (05/2024)
Table 1: Ablation about lision-language pre-training. "Scratch" means the model is trained from scratch. "V" and "L" mean visual pre-training and language pro-training respectively.
Sce
Web
Doc
Hand
Avg
Scratch
68.8
70.7
98.6
49.4
75.8
V
72.3
73.7
99.2
62.5
79.8
L
71.0
72.4
98.8
54.5
77.7
V -
L
73.9
74.8
99.3
63.7
80.8
different text images as one character may correspond to different number of patches. We adopt masked attention to replace the original attention in the encoder with the parameters unchanged.
We pre-train the decoder for epochs with batch size of 512, an initial learning rate of le- 4, 0.5 epochs warmup, and a cosine learning rate decay.
Data preprocessing. Since the Chinese text line images vary greatly in width, we resize the height of the input image to 32 with the aspect ratio kept and pad the width of the input images to 400. For the English word samples, we directly resize all input images to 32 x 128. We set the width of the split vertical patch to for all datasets by default. During the training of downstream recognition, some data augmentations like rotation, distortion, and color jitter are also used.
Evaluation We evaluate BCTR by first processing the predictions and ground truth with the following rules as Chen et al.] (202Tb): (i) convert the full-width characters to half-width characters; (ii) convert all traditional Chinese characters to simplified characters; (iii) convert all English characters to lowercase; (iv) remove all spaces. After that, we compute the accuracy in sentence level over each subsct and the whole dataset (Avg).
To evaluate the six scene English text datasets, we follow Shi et al. (2019) Yu et al. (2020) Fang et al.) (2021b ;; Wang et al. (2021c) Zhang et al. (2022) and evaluate the recognition performance of our model with case- isseiisenssttve word accuracy, We also report the average accuracy (Avg) over all samples.
4.3 Ablation Studies
In this section, conduct ablation studies on the BCTR dataset to verify the effectiveness of our proposed pre-training method. All experiments are conducted on 8 A100 GPUs with the ViT-B as the encoder.
The effectiveness of vision-language pre-training. To verify the effectiveness vision- laioun-anguuae pre- training, four models are trained. (i) The model is randomly initialized. (ii) The encoder initialized with visual pre-training, and the parts are randomly initialized. (iii) The model is initialized with language pre-training which pre-trained on synthetic data. (iv) The model is initialized with vision-language pre-training.
The results are presented in Table I which demonstrate the effectiveness of vision-language pre-training, Specifically, the superiority of the "V" model over the "Seratch' model indicates that visual pre-training of the encoder can lead to an improvement of up to 4%. In addition, language pre-training also yields better performance, achicving 1.9% improvement over the 'Scratch' model. Morcover, the visual pre-training and language pre-training are complementary, evidenced by the 1% improvement achieved by the 'V L' model over the "V" model. These results provide strong evidence that language pre-training is valuable approach.
Evaluation of representation quality. To evaluate the quality of representations learned by our pre- trained model, we conducted lincar probing experiments and the results are shown in Table 2 We fixed the pre-trained encoder and decoder, which was pre-trained using vision-language pre-training, and only updated the parameters of the remaining linear layer. Surprisingly, the model achieved average accuracy
8

---

Published in Transactions on Machine Learning Research (05/2024)
Table 2: Evaluation of representation quality. We consider two cases (1) to evaluate the representation quality of the encoder which with vision pre-training, we fix the pre-trained encoder and finctune the remain part of the recognition model. we termed this setting "V Linear probing'. (ii) We conduct "V L linear probing' that fixes the pre-trained encoder and decoder to evaluate the representation quality of vision-language pre-training. Only the remaining lincar layer is updated in this sctting.
Sce
Web
Doc
Hand
Avg
V Linear probing 62.8
67.5
98.1
54.3
73.6
V L Linear probing
39.1
54.7
66.3
28.1
47.8
Table 3: Studying if the pre-trained encoder is simultancously retrained during decoder pro-training. (i) The model is trained from scratch. (ii) the pre-trained encoder is retneined during the decoder pre-training. (ii) The pre-trained encoder is fixed when conducting decoder pre-training.
See
Web
Doc
Hand
Avg
Scratch
68.8
70.7
98.6
49.4
75.8
Retrain encoder
69.0
71.4
99.0
53.3
76.7
Fix encoder
73.9
74.8
99.3
63.7
80.8
of 47.8%, indicating that with lasion-language pre-training, the encoder and decoder are already capable of learning meaningful representations.
We also examined the quality of representations learned through visual pre-training, which fixes the pre- trained encoder and fine-tunes the decoder and linear layer of the recognition model. The model achieved an average accuracy of 73.6%, demonstrating that visual pre-training also enhances the representation quality of the encoder.
The effectiveness of our serially pre-training mechanism. To address the large domain gap between real and synthesized images, we fixed the pre-trained encoder when conducting language pre-training on the decoder. A comparison of our serially pre-trained mechanism with the encoder retrained from pre- trained weights is shown in Table which demonstrates the effectiveness of our approach. Specifically, when retraining the encoder from pre-trained weights during decoder pro-training, an average accuracy of 76.7% was achieved, which outperformed the scratch model (75.8%). However, this approach yielded 4.1% drop in accuracy compared to our serially pre-trained model (80.8)), suggesting that the pre-trained encoder would be impacted by the synthesized text images if it were retrained during decoder pre-training. This emphasizes the large domain gap between synthetic and real data.
Masking ratio visual pre-training We conducted an exploration into the effect of different masking ratios, namely 0.30, 0.45, and 0.60, for visual pre-training. The results have been presented in Table A and found that the optimal masking ratio for downstream recognition lask was 0.45. It is worth noting that our optimal masking ratio is lower than that reported in MAE He et al. (2022) (0.75). This discrepancy could be attributed to the higher information density of the text images used in our experiments.
Table 4: Ablation about the masking ratio of visual pre-training. In our experiments, the optimal masking ratio for dovnstream recognition task was 0.45.
Masking ratio
Sce
Web
Doc
Hand
Avg
0.30
71.5
73.1
99.1
61.8
79.3
0.45
72.3
73.7
99.2
62.5
79.8
0.60
72.0
73.6
99.1
60.7
79.4
0.75
72.2
73.4
99.1
59.1
79.2
9

---

Published in Transactions on Machine Learning Research (05/2024)
Table 5: Ablation about the masking strategy in language pre-training. It can be seen that masking helps the performance, particularly for challenging scenarios such scenes and handwritten text recognition, where accurate recognition is more difficult to achieve. "M" means trained with the masking strategy and "V" means trained with the visual pre-training.
M
Sce
Web
Doc
Hand
Avg
69.3
71.4
98.6
49.3
76.1
71.0
72.4
98.8
54.5
77.7
V
73.1
74.6
99.3
63.6
80.5
V
73.9
74.8
99.3
63.7
80.8
Table 6: Comparison with supervised pre-training, (i) The model is trained without pre-training model. (ii) The pre-trained model is trained on synthetic data with supervised pre-training, (iii) The model is pre-trained with our vision-language pre-training.
Sce
Web
Doc
Hand
Avg
Scratch Supervised pre-training
68.8
70.7
98.6
49.4
75.8
69.3
71.4
98.6
49.3
76.1
Ours
73.9
74.8
99.3
63.7
80.8
Masking strategy of language pre-training We have examined the impact of masking strategy on language pre-training in Table [ The results show that the language models trained with the masking strategy outperformed those without both the scenarios of only conducting language pre-training and integrating visual pre-training. The results highlight the efficacy of masking strategy in capturing the underlying linguistic rules during pre-training, particularly for challenging scenarios such as scenes and handwritten text recognition.
Comparison with supervised pre-training. To demonstrate the superiority of our vision-languaag pre- training, conducted a comparison with supervised pre-training, the results of which are presented in Table 6 Specifically, we pre-trained the recognition model on 100 million synthetic images and fine-tuned it on the BCTR dataset. Our findings show that, compared to the model trained from scratch, the model with supervised pre-training only achieved marginal improvement, indicating that supervised pre-training with synthetic images unnecessary when there is an abundance of real labeled data. However, our proposed vision-language pre-training resulted a 5% improvement, highlighting its effectiveness.
Vertical patch size. In our study, evaluated the performance of two different patch sizes without pre-training, the results of which are presented in Table 7 We found that the larger patch size led to worse performance, dropping the accuracy by 3.6%. We hypothesize that this may be due to the higher information density of the embedded token in the larger patch size, making it more difficult to learn.
Table 7: Ablation about the vertical patch size.
Patch size
Sce
Web
Doc
Hand
Avg
32 x 4
68.8
70.7
98.6
49.4
75.8
32 x 8
64.0
67.3
97.5
43.3
72.2
Generalizability. To demonstrate the effectiveness and generalizability of our proposed vision-language pre- training, we conducted experiments on a widely used text recognition decoder, the Connectionist Temporal Classification (CTC) decoder. We utilized multiple transformer encoder layers the sequence decoder and trained the model using CTC loss. During language pre-training, we randomly masked some characters and predicted the entire text sequence using CTC loss.
10

---

Published in Transactions on Machine Learning Research (05/2024)
ROSPA ZERI KInd
1E
ROSPK ZERI Kind スE
IN
Figure 5: Example results of masked image modeling. The first row the input images, the remaining two rows masked images and reconstructed images.
Osea
oscr
oscar
Villa
vvilla
villa
T
pizza
pizzza
pizza
K
DiSNEy
disnep
disney
Sthawberry
strawbering
strawberry
I
Figure 6: Visualization of text recognition results. The images in the first column and the fourth column are the input images. The results in the second column and the fifth column are predicted by the models trained from scratch. The results in column 3 and column 6 are output by the models with vision-language pre-training.
To validate the effectiveness of our approach, we trained two models: one trained from scratch and the other initialized with vision-language pre-training. Our results in Table demonstrate that the model initialized with vision-language pre-training achieved higher accuracy, with a 3.4% performance gain over the scratch model. Specifically, we achieved accuracies of 76.7% and 80.1%, respectively. These findings support the robustness and generalizability of our proposed model in the encoder-decoder recognition framework.
Table 8: Ablation about the different decoders.
Decoders
Scratch languual Pre-training
CTC-style 76.7
80.1
DETR-style 75.8
80.8
Qualitative analysis. We visualize some examples result of masked image modeling in Figure 5
The input
images are randomly masked with a masking ratio of 0.45. As a result, a part of a character or a whole character may be not visible for the encoder. The semantically plausible reconstructed results in the third row show that meaningful visual representation are learned by the encoder.
We also show some results of the models without and with languaee pre-training in Figure 6 As shown, when trained from scratch, the model is not robust to artistic fonts, occluded text, and visually similar characters. Benefiting from the visual and linguistic prior knowledge of real images and texi corpus, the model with vision-language pre-training can handle the above-mentioned scenarios well.
4.4 Comparison with Hate-of-the-an Methods
Comparison with statc-of-the-art pre-training method. We compare our method with previous state- of-the-art visual pre-training method DiG Yang et al. (2022) fairly with the same data setting. Specifically,
11

---

Published in Transactions on Machine Learning Research (05/2024)
Table 9: Comparison with state-of-the-ary pre-training method DiG. "Scratch' denotes the model is trained from scratch. "V" mcans finetuned with visual pro-training, "V LL' means finetuned with vision-language pre-training. The percentages 1,, 10,, and 100% refer to the proportions of real data used during the fine-tuning stage.
1%
10%
100%
# Params
CRNN Scratch
63.5
79.6
89.4
25M
DiG (ViT-S) Scratch
9.2
73.4
91.4
36M
DiG (ViT-S) V
84.6
92.0
94.6
36M
Ours (VTT-S) Scratch
72.3
90.6
95.2
31M
Ours (VIT-S) V
87.7
92.6
95.3
31M
Ours (ViT-S) V L
90.9
93.8
95.6
31M
we first pre-train our model on CC-OCR, MJSynth, and SenthText, following Yang et al. (022) and finetune the pre-trained model on varying percentages of real labeled samples (1%, 10,, and 100%) sourced from TextOCR and Open Images Dataset v5, Our model's performance was thoroughly evaluated and compared against DiG, as presented in Table
We discovered that pre-training DiG led to significant improvement in performance compared to the DiG baseline results obtained without pre-training. However, it is unclear whether this improvement is solely due to DiG pre- training or differences in the training settings. To reduce uncertainty, we trained commonly used recognition model, CRNN Shi et al. (2017a) using the same data settings as our model, and achieved results of 63.5%, 79.6%, and 89.4%. Notably, CRNN was published in 2015 and its performance is generally not competilive with most recent recognition methods. The results of CRNN in Table g are significantly higher than the DiG baseline, suggesting that the observed performance gains may be primarily due differences in the training scttings used for the DiG bascline rather than the DiG pro-training.
Due to the aforementioned reasons, directly compared the performance of pre-training- Remarkably, our proposed approach significantly outperformed DiG in all the cases. Most notably, our method achieved an exceptional accuracy of 90.9% with only 27.8K (1%) real labeled images during finetuning, providing substantial evidence of the significant reduction in the demand for labeled data offered by our approach. Furthermore, the performance gains observed our model, equipped with pre-trained encoder and decoder, further underscore the effectiveness of our pre-training mechanism.
Chinese Text Line Recognition. We evaluate the ability of our model to recognize Chinese text lines on the BCTR dataset. We set the number of character queries N to 40 since most of the samples in BCTR have less than 40 characters. We show the results of our method and some representative methods on the BCTR dataset in Table I0 When training from scratch, our method with ViT-S as encoder outperforms all the previous methods while with the smallest model size. Specifically, our method is better than the previous best method TransOCR Chen et al. (2021a) by 2.8% (72.8% vs. 75.6%). When training with the pre-trained encoder and decoder, our models surpass the previous best results by large margins. In detail, our method shows steady improvement with the increase of the model size and improves over the state-of-the-art by 5.3% and 8.0% respectively.
English scene text recognition. Following Shi et al. (2017a 2019), we set the number of character queries N to 25 which exceeds the lengths of most English words. Since scene text appeared in natural scenes always with distortions or irregular layouts, we employ a spatial transformer network Jaderberg et al. (2015) which is adopted in Shi et al. (2019) to rectify the input image and train it with our recognizer jointly.
Text recognition in the carly days faced the challenge of limited annotated data, which led to the common practice of training on synthetic data and testing on real data. However, as data has accumulated over the past decade and unsupervised techniques have advanced, pre pre-training on synthetic data and testing
12

---

Published in Transactions on Machine Learning Research (05/2024)
Table 10: Text recognition results on the BCTR dataset. VP and LP mean Visual Pre-training and Language Pre-training respectively.
Methods
VP
LP
See
Web
Doc
Hand
Avg
# PParams
CRNN Shi al.
(201aa)
X
53.4
54.5
97.5
46.4
67.0
ASTER Shi et al.
[2099
X
54.5
52.3
93.1
38.9
64.7
SAR Li et al. 2019
X
62.5
54.3
93.8
31.4
67.3
SRN
Yu et al. 2020
X
60.1
52.3
96.7
18.0
65.0
Trans
CCR
Chen et al.
2022a
X
X
63.3
62.3
96.9
53.4
72.8
84M
Ours
(VIT-S)
X
X
68.6
70.3
98.5
49.1
75.6
36M
Ours
(ViT-B)
X
X
68.8
70.7
98.6
49.4
75.8
100M
Ours (ViT-S)
71.4
72.5
98.8
55.6
78.1
36M
Ours (ViT-BB
V
73.9
74.8
99.3
63.7
80.8
100M
Table 11: Text recognition results on six English scene text datasets. training, Language Pre-training and Annotated Data respectively.
VP, LP and AD mean Visual Pre-
Methods
VP
LP
AD
IC13
SVT
IITTSK
IC15
SVTP
CUTE
Avg
#Params
ASTER Shi et al.
2019
X
X
syuth
91.8
89.5
93.4
76.1
78.5
79.3
86.7
TextScanner Wan et al
2020
X
syuth 92.9
90.1
93.9
79.4
84.3
83.3
84.4
PIMNet Qiao et 8 202
X
x
synth
95.2
91.2
95.2
83.5
84.3
84.4
90.5
SRN Yu ct al. 2020
X
syuth
95.5
91.5
94.8
82.7
85.1
87.8
90.4
55M
Vision Wang
2021c
syuth
95.7
91.7
95.8
83.7
86.0
88.5
91.2
33M
ABINet Pang ct al 2021
syuth
94.9
90.4
94.6
81.7
84.2
86.5
89.8
24M
12C2W
Xue et al
2021
syuth
95.0
91.7
94.3
82.8
83.1
93.1
90.2
Ours
VIT-S) (VIT-B)
synth
97.7
93.7
95.4
86.6
89.0
87.5
92.5
31M
Ours
synth
96.8
94.7
95.3
87.1
89.3
90.6
92.7
97M
SEED Qiao et al. 2020
x
synth
92.8
89.6
93.8
80.0
81.4
83.6
88.3
ABINeT Fang et al 2021
synth 97.4
93.5
96.2
86.0
89.3
89.2
92.7
37M
ConCLi Zhang
2022
syuth 97.7
94.3
96.5
85.4
89.3
91.3
92.8
37M
PerSec Liu et all (2022
X
syuth
97.2
94.6
96.3
84.4
89.5
90.2
92.4
DiG (VTT-SJ Yang et al
2022
syuth
97.1
93.4
96.7
87.1
90.1
88.5
93.1
36M
DiG (VTT-B) Yang
2022
synth
96.9
94.6
96.7
87.1
91.0
91.3
93.4
52M
TrOCR Li et al. 20216
syuth
98.3
93.2
91.0
84.0
91.0
89.6
90.3
558M
Ours (VIT=S)
syuth
97.7
94.0
95.8
87.5
90.2
89.2
93.0
31M
Ours (VIT-B)
synth
98.1
94.9
95.8
87.5
89.8
93.1
97M
DIG (ViI-S) Yang et al.
2022
VYV
VVR
real real
97.3
96.1
97.7
88.6
91.6
96.2
94.6
36M
DIG (VTT-B) Yang
2022
97.6
96.5
97.6
88.9
92.9
96.5
94.9
62M
MAERec-S Jiang al. 2023
real real 97.8
94.6
21M
Ours (ViT-S)
96.9
98.0
90.2
94.9
96.2
95.6
31M
Ours (ViT-B)
real
98.2
96.9
98.0
90.1
94.6
95.8
95.6
97M
real data has become increasingly impractical. Firstly, there is now an abundance of millions of real data samples available, providing sufficient resources for training recognition models. Secondly, domain differences between synthetic and real data can cause differences in the effectiveness of training. Therefore, the performance demonstrated in synthetic data training may not be representative of real data training. Thirdly, current unsupervised methods usually pre-train on real data learn good feature ropresentations, and fine-tuning on synthetic data may impair the effectiveness of pre-training. Consequently, pre-training unlabeled real data and fine-tuning on labeled real data for evaluating model performance. As most methods are trained only on synthetic data, we also provide comparison with this approach for reference. We report the results of our method and existing methods in Table and compare them in detail.
advocate
When trained on "synth" (MJSynth + Sexth without pro-training, our method significantly surpasses the previous methods and achieves the best performance. When trained on synthetic data with pre-training, method achieves comparable results with previous best model DiG Yang et al. (2022). Besides, When
13

---

Published in Transactions on Machine Learning Research (05/2024)
Comparison of run time. FPS is an abbreviation for Frames per Second, which denotes the number of images processed per second.
Table 12:
Methods
Avg
#Params
Flops
FPS
ABINet Fang et al. 2021
92.7
37M
2.97G
545
DiG Yang 2022
94.6
36M
15.74G
188
Ours(ViT-S)
95.6
31M
1.17G
1160
finetuned on real datasets (TextOCR + Open Images Dataset v5) as (2023), our method achieves the best performance (95.6% vs. 94.9% vs. 94.6%). The extensive comparisons demonstrate the excellence of our proposed method.
Yang et al. 2022 Jiang et al.
Note that, some methods may also achieve good performance but not relevant to pre-training are not listed. These methods, perform semi-supervised learning on real data Fang et al. (2021); Zheng al. (2022). refine the results with iterative mechanism Fang et al. [2021) ; Bautista & Atienza (2022), or predict text in multi-
granularity Wang et al. 2022 are complementary to our approach and may benefit our approach.
We also observe that the improvement brought by pre-training is more pronounced for Chinese benchmarks than for English benchmarks. This could be attributed to several reasons. Firstly, the current English recognition datasets contaín fewer characters and shorter text length, making it relatively easier than Chi- nese rocognition. Sccondly, the accuracy of current models on English benchmarks is already high and near saluration, with only marginal gains from pre-training as shown by the small improvement margin (0.4%) between DiG (ACM MM2022) and ABINet (CVPR 2021). Lastly, pre-training is more beneficial for chal- lenging scenarios where the original model performs poorly. In many real-world scenarios that are more challenging and lack sufficient labeled data, the performance gain from pre-training can be substantial, such as the handwriting set of BCTR.
Runtime Comparison. We compare the runtime of our method with two competitive other recognition methods that with similar parameters ours. The runtime is evaluated size of 64, the results are shown in Table 12] Compared to ABINet and DiG, our method has the smallest number of parameters, the highest accuracy, and the fastest running speed. We measured the computational complexity of the three networks to analyze the difference in inference speed. ABINet has 2.97G Flops, DiG has 15.74G Flops, and MaskOCR has 1.17G Flops, showing a positive correlation with speed. We further investigated why DiG's computational complexity far exceeds that of ABINet and MaskOCR under similar parameters. We found that DiG uses relatively small patch size (4 * 4), while our approach employs a patch size of 32 * 4. This results in longer feature sequence input to ViT in DiG, leading to an exponential increase in computational complexity.
A100 GPU with batch
5 Conclusion
The core of the proposed approach for text recognition lies in that pre-train the recognition model, including both the encoder and the decoder to learn vision and language representation. The visual pre- training is able to benefit from large-scale real text images that are easily available without the need for text annotation. The language pre-training is able to benefit from the synthetic text images that are also easily available with the character-level annotation easily obtained. Experiments verify the effectiveness of our proposed vision- languag pre-training.
References
Hangbo Bao, Li Dong, Songhao Piao, and Furn Wei. Beit: BERT pre-training of image transformers. In ICLR, 2022.
Darwin Bautista and Rowel Atienza. Scene text recognition with permuted autoregressive sequence models. Tn FCCV, volume 13688, pp. 178 196, 2022.
14

---

Published in Transactions on Machine Learning Research (05/2024)
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, pp. 213 22-2 2020.
Jingve Chen, Bin Li, and Xiangyang Xue. Scene text telescope: Text-focused scene image super-resolution CVPR, pp. 12026-12035, 2021a.
Jingye Chen, Haiyang Yu, Jianqi Ma, Mengnan Guan, Xixi Xu, Xiaocong Wang, Shaobo Qu, Bin Li, and Xiangyang Xue. Benchmarking chinese text recognition: Datasets, baselines, and an empirical study, CoRR, Abs/2112.15001 2021b.
Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervised representation learning. CoRR, abs/2202.03006 2022.
Xinlei Chen, Haoqi Fan, Ross B. Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. CoRR, abs/2003.04297, 2020.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In ICCV, pp. 9620 9629, 2021c.
Chee Kheng Chng, Errui Ding, Jingtuo Liu, Dimosthenis Karatzas, Chee Seng Chan, Lianwen Jin, Yuliang Läu, Yipeng Sun, Chun Chet Ng, Canjie Luo, Zihan Ni, ChuanMing Fang, Shuaitao Zhang, and Junyu Han. ICDAR2019 robust reading challenge on arbitrary-shaped text rre-art. In ICDAR, pp. 1571 1576, 2019.
Jacob Devlin, Ming-Way Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirec- tional transformers for language understanding. In NAACL-HLT, pp. 4171 1711-4186,1 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Shancheng Fang, Honglao Xie, Yuxin Wang, Zhendong Mao, and Yongdong Zhang. Read like humans: Autonomous, bidirectional and iterative language modeling for scene text recognition. In CVPR, pp. 7098 7107, 2021.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large miníbatch SGD: training imagenet in hour. CoRR, abs/ has/1706.02677. 2017.
Alex Graves, Santiago Fernández, Faustino J. Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In ICML, pp. 369 376, 2006.
Ankush Gupta, Andrea Vedaldi, and Andrew Zisserman. Synthetic data for text localisation in natural images. In CVPR, pp. 2315-2324, 2016.
Kaiming Hc, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, pp. 9726-9735. 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autoencoders are scalable vision learners. In CVPR, pp. 15979-15999 2022.
Mengchao He, Yuliang Liu, Zhibo Yang, Sheng Zhang, Canjic Luo, Feiyu Gao, Qi Zheng, Yongpan Wang. Xin Zhang, and Lianwen Jin. ICPR2018 contest on robust reading for multi-type web images. ICPR, pp. 7-1 2018.
Tla Loshchilov and Frank Hutter. SGDR: Stochastic Gradient Descent with Warm Restarts. In ICLR, 2017.
15

---

Published in Transactions on Machine Learning Research (05/2024)
Max Jaderberg, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Synthetic data and artificial neural networks for natural scene text recognition. CoRR, abs/1406 22221,06. 2014a.
Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Deep features for text spotting. In ECCV, pp. 512-528, 2014b.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukeuoglu. Spatial transformer networks. In NeurIPS, pp. 2017-2025, 2015.
Chao Jía, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In Marina Mcila and Tong Zhang (eds.), ICML, volume 139, pp. 4904-4916, 2021.
Qing Jiang, Jiapeng Wang, Dezhi Peng, Chongyu Liu, and Lianwen Jin. Revisiting scene text recognition: A data perspective. In ICCV, pp. 15543-20554, October 2023.
Dimosthenis Karatzas, Faisal Shafait, Seiichi Uchida, Masakazu Twamura, Lluis Gomez Bigorda, Sergi Rob- les Mestre, Joan Mas, David Fernández Mota, Jon Almazán, and Lluis-Pere de las Heras. ICDAR 2013 robust reading competition. In ICDAR, pp. 1484 1493, 2013.
Dimosthenis Karatzas, Lluis Gomez-Bigorda, Anguelos Nicolaou, Suman K. Ghosh, Andrew D. Bagdanov, Masakazu Twamura, Jiri Matas, Lukas Neumann, Vijay Ramaseshan Chandrasekhar, Shijian Lu, Faisal Shafait, Seiichi Uchida, and Ernest Valveny. ICDAR 2015 competition on robust reading. In ICDAR, pp. 1156-1160, 2015.
Hui Li, Peng Wang, Chunhua Shen, and Guyu Zhang. Show, atlend and read: A simple and strong baseline for irregular text recognition. In AAAI, pp. 8610 8610-8617, 2019.
Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Gotmare, Shafiq R. Joty, Caiming Xiong, and Steven Chu- Hong Hoi. Aligu before fuse: Vision and language representation learning with momentum distillation. In NeurIPS, pp. 9694 0694-9705, 2021a.
Jumnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: bootstrapping language-image pre- training for unified vision-language understanding and generation. In ICML, volume 162, pp. 12888-1290M 2022.
Minghao Li, Tengchao Lv, Lei Cui, Yijuan Lu, Dinei A. F. Florêncio, Cha Zhang, Zhoujun Li, and Furu Wei. Trocr: Transformer-baseel optical character recognition with pre-trained models. CoRR, abs/2109.10283 2021b.
Minghui Liao, Jian Zhang, Zhaoyi Wan, Fengming Xie, Jiajun Liang, Pengyuan Lyu, Cong Yao, and Xiang Bai. Scene text recognition from two-dimensional perspective. In AAAI, pp. 8714 8721, 2019.
Hao Liu, Bin Wang, Zhimin Bao, Mobai Xue, Sheng Kang, Deqiang Jiang, Yinsong Liu, and Bo Ren. Perceiving stroke-semantic context: Hierarchical contrastive learning for robust scene text recognition. Tn AAAI, 2022.
Nya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Pengyuan Lyu, Minghui Liao, Cong Yao, Wenhao Wu, and Xiang Bai. Mask textspotter: An end-to-end trainable neural network for spotting text with arbitrary shapes. In ECCV, volume 11218, pp. 71-88, 2018.
Pengyuan Lyu, Zhicheng Yang, Xinhang Leng, Xiaojun Wu, Ruiyu Li, and Xiaoyong Shen. 2d attentional irregular scene text recognizer. CoRR, abs/1906.05708, 2019.
Max Jaderberg and Karen Simonyan and Andrea Vedaldi and Andrew Zisserman. Reading text in the wild with convolutional neural networks. TJCV, pp. 1-20, 2016.
16

---

Published in Transactions on Machine Learning Research (05/2024)
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In ICLR, 2017.
Anand Mishra, Karteek Alahari, and C. V. Jawahar. Scene lext recognition using higher order language priors. In Richard Bowden, John P. Collomosse, and Krystian Mikolajezyk (eds.), BMVC, pp. 1-11, 2012.
Trung Quy Phan, Palaiahnakote Shivakumara, Shangxuan Tian, and Chew Lim Tan. Recognizing text with perspective distortion in natural scenes. In ICCV, pp. 569 576,576,44 2013.
Zhi Qiao, Yu Zhou, Dongbao Yang, Yucan Zhou, and Weiping Wang. SEED: semantics enhanced encoder- decoder framework for scene text recognition. In CVPR, pp. 13525-13534, 2020.
Zhi Qiao, Yu Zhou, Jin Wei, Wei Wang, Yuan Zhang, Ning Jiang, Hongbin Wang, and Weiping Wang. Pimnet: A parallel, iterative and mimicking network for scene text recognition. In ACM MM, pp. 2046 2055, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Πla Sutskever. Learning transferable visual models from natural language supervision. ICML,, volume 139, pp. 8748 8763, 2021.
Anhar Risnumawan, Palaiahnakote Shivakumara, Chee Seng Chan, and Chew Lim Tan. A robust arbitrary text detection system for natural scene images. Expert Syst. Appl, pp. 8027-8044 2014.
Phillip Rust, Jonas F. Lotz, Emanuele Bugliarello, Elizabeth Salesky, Miryam de Lhoneux, and Desmond Elliott. Language modelling with pixels. In ICLR, 2023.
Baoguang Shi, Xiang Bai, and Cong Yao. An end-to-end trainable neural network for mmage-baserr sequence recognition and its application to scene text recognition. TPAMI, pp. 2298-2304 2017a.
Baoguang Shi, Cong Yao, Minghui Liao, Mingkun Yang, Pei Xu, Linyan Cui, Serge J. Belongie, Shijian Lu, and Xiang Bai. ICDAR2017 competition on reading chinese text in the wild (RCTW-17). In ICDAR, pp. 1429 1434, 2017b.
Baoguang Shi, Mingkun Yang, Xinggang Wang, Pengyuan Lyu, Cong Yao, and Xiang Bai. ASTER: an attentional scene text recognizer with flexible rectification. TPAMI, pp. 2035-2048, 2019.
Amanpreet Singh, Guan Pang, Mandy Toh, Jing Huang, Wojciech Galuba, and Tal Hassner. Textocr:
Towards large-scale end-to-end reasoning for arbitrary-shaper scene text. In CVPR, pp. 2012-8812, 2021.
Sibo Song, Jianqiang Wan, Zhibo Yang, Jun Tang, Wenqing Cheng, Xiang Bai, and Cong Yao. Vision- language pre-training for boosting scene text detectors. In CVPR, pp. 15660-15670. TEEE, 2022.
Yipeng Sun, Dimosthenis Karatzas, Chee Seng Chan, Lianwen Jin, Zihan Ni, Chee Kheng Chng, Yuliang Tiu, Canjie Luo, Chun Chet Ng, Junyu Han, Errui Ding, and Jingtuo Liu. ICDAR 2019 competition on large-scale street view text with partial labeling RRC-LSVT. In ICDAR, pp. 1557 1562, 2019.
Zhaoyi Wan, Minghang He, Haoran Chen, Xiang Bai, and Cong Yao. Textscanner: Reading characters in order for robust scone text recognition. In AAAI, pp. 12120-12127, 2020.
Jianfeng Wang, Xiaowei Hu, Zhe Gan, Zhengyuan Yang, Xiyang Dai, Zicheng Liu, Yumao Lu, and Lijuan Wang. UFO: A unified transformer for vision-language representation lcarning. CoRR, abs/2111.10023,, 2021a.
Kai Wang, Boris Babenko, and Serge J. Belongie. End-to-end scene text recognition. In ICCV, pp. 1457 1464, 2011.
Peng Wang, Cheng Da, and Cong Yao. Multi-gramularity prediction for scene text recognition. In ECCV, volume 13688, pp. 355, 2022.
Wenhui Wang, Hangbo Bao, Li Dong, and Furu Wei. Vlmo: Unified vision-language pre-training with mixture-o--modalltyyexpepe CoRR, 0tx/2111.0235, 2021b.
17

---

Published in Transactions on Machine Learning Research (05/2024)
Yuxin Wang, Hongtao Xie, Shancheng Fang, Jing Wang, Shenggao Zhu, and Yongdong Zhang. From two to one: A new scene text recognizer with visual language modeling network. In ICCV, pp. 14174-14183, 2021c.
Yiheng Xu, Minghao Tá, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. Layoutlm: Pre-training of text and layout for document image understanding. In Rajesh Gupta, Yan Liu, Jiliang Tang, and B. Aditya Prakash (eds.), KDD, pp. 1192 1200, 2020.
Chuhui Xue, Shijian Lu, Song Bai, Wenqing Zhang, and Changhu Wang. I2C2W: image-tochharaatte---oowtrr transformers for accurate scene text recognition. CoRR, abs/2105.08383, 2021.
Mingkun Yang, Minghui Liao, Pu Lu, Jing Wang, Shenggao Zhu, Hualin Luo, Qi Tian, and Xiang Bai. Reading and writing: Discriminative and generative modeling for self-supervised text recognition. ACMMM, pp. 4214-4223, 2022.
Zhengyuan Yang, Yijuan Lu, Jianfeng Wang, Xi Yin, Dinei Florêncio, Lijuan Wang, Cha Zhang, Lei Zhang, and Jiebo Luo. TAP: text-aware pre-training for text-vqa and text-caption. In CVPR, pp. 8751 8761, 2021.
Deli Yu, Xuan Li, Chengquan Zhang, Tao Liu, Junyu Han, Jingluo Liu, and Errui Ding. Towards accurate scene text recognition with semantic reasoning networks. In CVPR, pp. 1110011219, 2020.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. CoRR, hbs/2205.00911, 2022.
Tai-Ling Yuan, Zhe Zhu, Kun Xu, Cheng-Jun Li, Tai-Jiang Mu, and Shi-Min Hu. A large chinese text dataset in the wild. JOST, pp. 509 521, 2019.
Hesuo Zhang, Lingyu Liang, and Lianwen Jin. Scut-hcodoc: A new benchmark dataset of handwritten chinese text in unconstrained camera-captured documents. PR, pp. 107559, 2020.
Rui Zhang, Mingkun Yang, Xiang Bai, Baoguang Shi, Dimosthenis Karatzas, Shijian Lu, C. V. Jawahar, Yongsheng Zhou, Qianyi Jiang, Qi Song, Nan Li, Kai Zhou, Lei Wang, Dong Wang, and Minghui Liao. TCDAR 2019 robust reading challenge on reading chinese text on signboard. In ICDAR, pp. 1587-1581, 2019.
Xinyun Zhang, Binwu Zhu, Xufeng Yao, Qi Sun, Ruiyu Li, and Bei Yu. Context-based contrastive learning for scene text recognition. In AAAI, 2022.
Caiyuan Zheng, Hui Li, Seon-Min Rhee, Seungju Han, Jae-Joon Han, and Peng Wang. Pushing the perfor- mance limit of scene text recognizer without human annotation. In CVPR, pp. 14096-14105, 2022.
Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason J. Corso, and Jianfeng Gao. Unified vision- language pre-training for image captioning and VQA. In AAAI, pp. 10041,1049, 2020.
18


Reviews:
[{"review_id": "EnHYJrno5r", "reviewer": "Reviewer_PBQ5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "In general I like the simplicity of the approach, its strong results, and high inference throughput.\n\n\n## Strengths:\n- The proposed approach is simple and intuitive.\n- The model seems to scale well, with results improve substantially from 36M to 100M parameters.\n- The ablations are well conducted and informative, and showcase the benefits of the pretraining schemes and architectural design choices.\n\n\n## Weaknesses:\n- What is the reason for the proposed model being so much faster than the other baselines, despite parameter counts being comparable (Table 10)? Could you be more specific in identifying reasons for the speedup?\n- The proposed pretraining scheme appears to be the same as the one proposed in [1], but there is no discussion of that paper. Can you provide more explicit comparisons against this work, and highlight the differences?\n- Minor typos: “Encode-decoder” -> “Encoder-decoder” (Figure 2 caption), Sec 4.4 (72.8% v.s. 75.6% has incorrect formatting)\n\n\n**References**\n\n[1] Rust, Phillip, et al. \"Language modelling with pixels.\" ICLR 2023.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a masked image-language modeling pretraining scheme to learn a strong encoder-decoder model for the task of text recognition. The proposed pretraining scheme applies masked language modeling to the image encoder over a set of unlabeled real text images, and pretrains the language decoder by transforming a text corpus into images. The method performs well on several Chinese and English benchmarks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Would strengthen the work:\n- Discussion about model throughput, and why exactly it’s faster than prior approaches\n\nCritical:\n- Comparisons against Rust et al. (2023)", "broader_impact_concerns": "There are no major issues with the ethical implications of the work. It studies text recognition and is relatively low risk.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "m5WBgGUEUb", "reviewer": "Reviewer_gMeq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "First, this paper, entitled \"Text Recognition with Masked Vision-Language Pre-training\", focuses primarily on the area of scene text recognition, a detail that could be more explicitly reflected in the title for clarity. While the paper presents a method that integrates vision and language pre-training within an encoder-decoder architecture, it notably omits any discussion of various models developed for document recognition, which are closely related to the field of scene text recognition. Furthermore, the related work section would benefit greatly from a more explicit exploration of the links between these two fields. \n\nSecond, The approach presented in the paper, which integrates vision and language pre-training within an encoder-decoder architecture for text recognition, must be reconsidered in the context of the paper introducing the TrOCR model, particularly its 2023 version as presented at the Thirty-Seventh AAAI Conference on Artificial Intelligence (AAAI-23). It's noteworthy that while the paper cites the 2021 version of TrOCR, it overlooks the significantly updated and improved 2023 iteration. This oversight raises concerns regarding the originality and contemporary relevance of the proposed approach. \nIndeed, the core of the paper is centered on the proposed pretraining methodology, which 1) combines text-to-image conversion, 2) masked image-language modeling, and 3) a strategic use of a frozen encoder to effectively bridge the gap between visual and textual data interpretation.\nThe first and second proposition are already used in the 2021 TrOCR paper. The third proposition seems to be new compared to trOCR and to improve the results as show on table 2. However, the main contribution of this article comes down to this aspect, which is very limited.\n\nThird, in this text recognition model, there is a necessary dependency on a separate text recognition model to provide the positions of the text, a dependency that should be explicitly stated. It's important to anticipate a potential drop in performance when these two models are combined, as they are not optimized together. This raises a critical question: given the interdependencies and potential performance trade-offs, is it still worth optimizing the text recognition module in isolation? This broader perspective should be considered when assessing the overall effectiveness and progress of approaches in this area.\n\nIt's worth noting that there is an emerging trend within the document recognition community towards the development of fully integrated models that combine both text detection and recognition(End-to-End Document Recognition and Understanding with Dessurt in ECCV2022, DAN : a Segmentation-free Document Attention Network for Handwritten Document Recognition in PAMI 2023). The field's shift towards integrated models highlights the importance of addressing the issues of interdependence and potential performance trade-offs when text detection and recognition processes are not optimized together.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a novel approach to text recognition that combines vision and language pre-training in a classical encoder-decoder framework. The authors propose masked image modelling for the encoder and masked image-language modelling for the decoder. The approach involves pre-training the encoder on unlabeled real text images and the decoder on synthesized text images, thereby enhancing both visual and linguistic representation learning. The paper presents extensive experiments on Chinese and English text images, demonstrating superior performance over state-of-the-art methods.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "- In section 3.1, it is written \"We partition the image vertically into a set of M vertical patches,[p1, p2, . . . , pM].\" This supposes that the text orientation on the image is given.\n- The loss (Eq. 1) is purely sequential, which implies that a single insertion makes the full sequence considered incorrect. It is surprising that the model can be trained with this loss whereas loss based on alignment such as CTC have been generally used in the document recognition community. \n- Section 4.2 Evaluation: the metrics used for BCTR is the sentence level accuracy, which seem very strict (one character error discard the full sentence). Could you confirm? The metrics is different for the English datasets.\n- Table 3: the model is not very sensitive to the masking ration, which is good. What are the results with a masking ratio at 0.75 which is reported as the best for MAE?\n- Table 7: what are the 1% 10% 100% for?\n- Table 10: what is the time unit?", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "D94Vg2nbX6", "reviewer": "Reviewer_jHaA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The key contributions of the paper are 1) the use of real images for pretraining of encoder, 2) the use of synthetic images for pretraining of decoder while freezing encoder, 3) both encoder and decoder are pretrained. None of which is probably very novel by itself, but the combination and providing the strategy supported by the thorough experiments have a good value. Especially that showing a practical strategy to pretrain decoder with synthetic images seems important. It gives an insight how to use synthetic images for training OCR systems. The experiments are extensive and it is great that the methods are evaluated on two languages.\n\nThe presentation could be improved in terms of writing and how to summarize results, but I do not find major weakness in the paper. A possible weakness is that the comparisons with prior studies may not be very fair or it may not contrast the differences in terms of algorithm. It is a shared common problem in the field that you just put the numbers from other papers. As discussed in the paper, we often observe significantly better results by just retraining a simple weak model like Resnet. It is great to have results of GiG from this perspective.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper discusses a pretraining strategy for encoder-decoder-based OCR systems and proposes a training procedure consisting of three steps: Pretraining of encoder with real images, pretraining of decoder with synthetic images while fixing the encoder part, and finetuning of the whole system. The pretraining criteria are basically the prediction of masked regions of pixels and texts for encoder and decoder, respectively. The key design principle is that the pretraining of encoder is done by real images and the pretraining of decoder is done by synthetic images while freezing the encoder. It allows to pretrain all the components effectively. The extensive experiments show the effectiveness of the approach and it reports strong numbers on OCR benchmarks in English and Chinese.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "It seems the conclusion from the results in Table 1, 2, 5 is that we should never train the encoder part with synthetic data only. Does that sound reasonable? What if we combine real images with synthetic data when finetuning, etc. It would be great to have discussions on this.\n\nThere are many tables with the same result at the first row (i.e. Scratch). I'm wondering if it is a better idea to create a consolidated table listing all results. There are several numbers that are just mentioned in text. Those could be put in the table, too. If it makes the readability worse, you can ignore this. \n\nThere seem to be many unnatural expressions in the text although I was able to understand the meaning. I would recommend a proof-reading by a third-party person.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Interesting and novel combination of methods. OCR researchers would likely find it valuable to see the experiments that show how to pretrain the decoder with synthetic images. Lots of interesting ablations.
Decision: Accept (poster)