VL-Reader: Vision and Language Reconstructor is an Effective Scene Text Recognizer

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 10:58:56

Abstract: Text recognition is an inherent integration of vision and language,
encompassing the visual texture in stroke patterns and the semantic
context among the character sequences. Towards advanced text
recognition, there are three key challenges: (1) an encoder capable
of representing the visual and semantic distribution; (2) a decoder
supervises the alignment between vision and semantics; and (3)
consistency in the framework during pre-training and fine-tuning.
Inspired by masked autoencoding, a successful pre-training strategy
in both vision and language, we propose an innovative scene text
recognition approach, named VL-Reader. The novelty of VL-Reader
lies in that the interplay between vision and language is pervasive
throughout the entire process, not only in the encoding stage but
also the decoding stage, which has been previously overlooked.
Concretely, we first introduce a Masked Visual-Linguistic Recon-
struction (MVLR) objective, which aims at simultaneously modeling
visual and linguistic information. Then, we design a Masked Visual-
Linguistic Decoder (MVLD) to further leverage bi-modal feature
interaction. The architecture of VL-Reader maintains consistency
from training to inference. In the pre-training stage, VL-Reader
reconstructs both masked visual and text tokens, while in the fine-
tuning stage, the network degrades to reconstruct all characters
from an image without any masked regions. VL reader achieves an
average accuracy of 97.1% on six typical datasets, surpassing the
SOTA by 1.1%. The improvement was even more significant on chal-
lenging datasets. The results demonstrate that vision and language
reconstructor can serve as an effective scene text recognizer.

Main paper: 
 related work 2.1 vision-dominated methods: 
 Vision-dominated methods treat characters as distinct visual symbols and directly classify them into different categories solely based on visual features. CTC-based methods [12, 13, 34] utilize a CTC decoder for converting a sequence of features into a sequence of characters. Segmentation-based methods [24, 25, 42] employ a semantic segmentation pipeline to address the scene text recognition task. Recent methods [16, 47] integrate masked image modeling (MIM) as an additional pre-training step to enhance visual representation. DiG [47] combines masked image modeling and contrastive learning to improve discriminative and generative representation. MAE-Rec [16] adopts the pipeline of MAE and utilizes large-scale unlabeled data to develop a strong visual encoder. Although these approaches have achieved promising progress in standard benchmarks, their lack of integration of linguistic knowledge may hinder performance when handling low-quality images.Recent approaches [4, 9, 44, 46, 49] have recognized the importance of linguistic knowledge and have started to integrate it into their systems. SRN [49] adopts a semantic reasoning module to model contextual information and achieves promising results. ABINet [9] introduces an iterative refinement stage, where linguistic knowledge is used to progressively correct text recognition results with a standalone language model. VisionLAN [46] successfully integrates visual and linguistic information into a single model. MGP-STR [44] recognizes characters in a multi-granularity approach by additionally predicting subwords. PARSeq [4] utilizes linguistic knowledge in an implicit way by using a permuted auto-regressive sequence model. Despite integrating linguistic knowledge into their models, these methods typically consider it as merely a supplement to visual knowledge. Thus we need a comprehensive exploration of jointly modeling visual and linguistic knowledge. In the second training phase, the attention mask for masking out characters will be a matrix completely filled with "1"s. 
methodology 3.1 overall architecture: 
 The overview of the proposed VL-Reader is depicted in Fig. 3 (a) which consists of a visual encoder, a linguistic embedding layer, a masked visual-linguistic decoder, and two reconstruction heads. We will first introduce the proposed training objective Masked Visual-Linguistic Reconstruction (MVLR). After that, the details of all components are presented in the subsequent sections.In recent years, drawing inspiration from MAE [11] , several approaches [16, 47] Visual-Linguistic Reconstruction Objective. The reconstruction objective can be further divided into two parts, the visual reconstruction and the linguistic reconstruction. In the case of visual reconstruction, the model is trained to reconstruct the masked visual patches utilizing both the unmasked patches and unmasked text tokens. Similarly, for linguistic reconstruction, the model is trained to reconstruct the masked text tokens by leveraging both the unmasked tokens and unmasked visual patches. By effectively integrating visual and linguistic information, VL-Reader is capable of learning strong cross-modal feature representation.Visual-Linguistic Masking. For an image 𝐼 ∈ 𝑅 𝐻 ×𝑊 ×𝐶 and its corresponding text sequence 𝑇 , we first cut the image with patch size (𝑝 ℎ , 𝑝 𝑤 ) and then randomly mask a subset of patches with ratio 𝑟 𝑣 . The remaining patches 𝐼 * are visible patches and will be encoded by the ViT encoder. For text sequence 𝑇 , we randomly All the text tokens will be encoded by a text embedding vector, which follows PARSeq [4] . The effects of 𝑟 𝑣 and 𝑟 𝑙 will be discussed in the ablation studies. Encoder. Our encoder consists of a visual encoder to encode visual information and a linguistic encoder to convert character tokens into embedding. Following prior works [4, 16, 44] , the visual encoder employs the architecture of ViT [8] but is applied only on visible patches 𝐼 * . The visual encoding process can be formulated as follows:EQUATIONwhere 𝑖 is the image patch. The first part denotes the visible patches encoded by 𝑉 𝑖𝑇 , and the second part denotes the masked patches with random vectors 𝑉 𝑒𝑐. 𝐹 𝑣 is the final visual embedding.As for encoding context information, we adopt a text embedding layer to convert character tokens into context embedding. The contextual encoding process can be formulated as follows:EQUATION)where 𝑡 is the character token. Different from visual encoding, the visible characters and masked characters are encoded in the same way. 𝐹 𝑙 represents the final contextual embedding. Note that, both visual and text mask token [MASK v ] and [MASK l ] are shared and learnable vectors, indicating a visual patch or a text token to be reconstructed. Besides, we omit position embedding in the formulas for brevity, both visual embedding and contextual embedding contain position embedding.Masked Visual-Linguistic Decoder. The Masked Visual-Linguistic Decoder (MVLD) has 𝑁 𝑑 layers. The detailed structure of each layer is illustrated in Fig. 3 (c ). During decoding, we use a sequence of query tokens 𝐹 𝑞 ∈ 𝑅 𝐿 𝑞 ×𝐶 as the bridge to gather visual-linguistic information and carry out cross-modal feature interaction. Concretely, we adopt a visual self-attention to model visual context, a query-language cross-attention to model linguistic context, and a visual-linguistic cross-attention to model the interaction between visual and linguistic modality. All attention layers are implemented with standard Multi-Head Attention MHA(q, k, v, m), where q, k, v and m indicates query, key, value and an optional attention mask. The decoding process in 𝑛 𝑡ℎ layer can be formulated as follows:EQUATIONEQUATIONEQUATIONEQUATIONwhere, 𝑛 ∈ [0, 1, . . . , 𝑁 𝑑 ] indicates the 𝑛 𝑡ℎ layer of MVLD, 𝐻 𝑣 and 𝐻 𝑞 represent hidden layers for visual self-attention and linguistic cross attention respectively. 𝐹 𝑛-1 𝑣 is the output visual features of the (𝑛 -1) 𝑡ℎ layer and is also the input of the 𝑛 𝑡ℎ layer. 𝐹 𝑙 is the encoded linguistic feature introduced in Equation 2, and remains the same across all 𝑁 𝑑 layers. 𝐹 𝑛-1 𝑞 is the input query tokens of the 𝑛 𝑡ℎ layer and 𝐹 0 𝑞 is randomly initialized before the first layer. We employ attention mask 𝑚 𝑞,𝑙 in the query-text cross-attention to avoid information leakage. The positions with masked characters are set as [-inf] in 𝑚 𝑞,𝑙 , as exemplified in Fig. 4 . Concretely, we adopt the permuted attention mask for robust context modeling and the masked attention mask to avoid information leakage. Then we merge them with an AND operation to form our query-text attention mask 𝑚 𝑞,𝑙 .For the reconstruction of each layer, we also send the decoded visual and linguistic features 𝐹 𝑘 𝑣 and 𝐹 𝑘 𝑞 to the visual and linguistic reconstruction head respectively:EQUATIONEQUATIONwhere 𝑣 𝑛 and 𝑙 𝑛 indicate the reconstructed visual patches and linguistic tokens of the 𝑛 𝑡ℎ layer. 𝐻𝑒𝑎𝑑 𝑣 and 𝐻𝑒𝑎𝑑 𝑙 both consist of several linear layers. Mean Square Error (MSE) and CrossEntropy loss functions are adopted to supervise the visual and linguistic reconstruction objectives respectively.Optimization. The model is trained end-to-end using the following objective:EQUATIONwhere 𝐿 𝑣 and 𝐿 𝑙 indicate the loss of visual and linguistic reconstruction respectively. We use Mean Square Error (MSE) for 𝐿 𝑣 :EQUATIONwhere 𝑀 𝑣 represents the masked visual patches, 𝑣 𝑛 𝑖 and 𝑦 𝑖 represents the 𝑖 𝑡ℎ reconstructed and ground-truth pixel of the 𝑛 𝑡ℎ layer.We use Cross-Entropy loss for 𝐿 𝑙 :EQUATIONwhere 𝑀 𝑙 indicates the masked language tokens, 𝑙 𝑛 𝑖 and 𝑡 𝑖 indicates the 𝑖 𝑡ℎ reconstructed and ground-truth language token of the 𝑛 𝑡ℎ layer.Training. The training process of VL-Reader has two phases, as illustrated in Fig. 3 . In the first phase, we adopt MVLR as the training objective, in which both image and text reconstruction are implemented. Under the guidance of MVLR, our model learns a cross-modal representation. Compared to the methods that use a single visual reconstruction, our method takes into consideration the accuracy of text during the visual reconstruction process. On the contrary, compared to the method of simply using a language model for text correction, our approach also takes into account the visual context during text reconstruction.The second training phase, also known as the fine-tuning phase, involves performing text recognition tasks based on the architecture from the first phase. In this stage, we deactivate the visual reconstruction and focus solely on the linguistic reconstruction.Specifically, we set 𝜆 𝑣 and 𝜆 𝑙 as 1.0, 𝑟 𝑣 as 0.75, and 𝑟 𝑙 as 0.2 during the first training phase to activate both visual and linguistic reconstruction. We set 𝜆 𝑣 as 0 during the second training phase to deactivate visual reconstruction and focus on linguistic reconstruction. To further enhance the language capabilities during the recognition stage, we utilize the permutation language model strategy from PARSeq [4] . This only requires substituting the attention mask in the MVLD module (see Fig. 4 ). In summary, the two training phases have the same model architecture. The only variation lies in the presence of the mask in the input and the attention mask in the decoder.Inference. We adopt an auto-regressive pattern to decode characters in the reading order during inference. For the first iteration, we use only the first query token q 1 of the initial query sequence 𝐹 0 𝑞 . And for the succeeding iteration 𝑡, we use query tokens [q 1 , q 2 , . . . , q 𝑡 ]. We set the visual masking ratio 𝑟 𝑣 as 0 to allow full visual perception. Inspired by prior works [4, 9] , a refinement stage is employed to further adjust predicted results. We use the cloze mask as the attention mask 𝑚 𝑞,𝑙 for query-text cross-attention in the refinement stage. 
experiments 4.1 datasets and implementation details: 
 Datasets. Following prior works [4] [16], we use both synthetic and real datasets for training. The synthetic datasets include MJSynth (MJ) [14, 15] and SynthText (ST) [10] . The real datasets are introduced in PARSeq [4] , including ArT [5] , COCO-Text [41] , LSVT [39] , MLT-19 [29] , RCTW17 [36] , ReCTS [52] , Uber [53] , TextOCR [37] and OpenVINO [22] . To evaluate our method as fair as possible, we train our model on three groups of datasets (i.e., synthetic datasets only, real datasets only and a mixture of synthetic and real datasets).To conduct a fair comparison with previous methods, we follow the evaluation protocol of PARSeq [4] . Concretely, (1) six standard benchmark datasets including IIIT5K (IIIT) [27] , ICDAR2013 (IC13) [18] , ICDAR2015 (IC15) [17] , Street View Text (SVT) [43] , Street View Text-Perspective (SVTP) [31] and CUTE80 (CUTE) [33] are used for evaluation. (2) In addition, we also evaluate our model on seven more challenging datasets, including two occluded datasets WOST and HOST [46] , two handwritten datasets IAM [26] and CVL [21] and three large-scale datasets COCO-Text [41] (lowresolution, occluded), ArT [5] (curved, rotated) and Uber [53] (vertical, rotated) to validate the robustness of our methods in more challenging scenarios.Implementation Details. The training process has two phases. In the first phase, we employ MVLR as our training objective to simultaneously reconstruct visual and linguistic information. In the second phase, we set 𝜆 𝑣 , 𝑟 𝑣 , 𝑟 𝑙 as 0 to disable visual reconstruction and focus on the text recognition task only. We train our model for 20 epochs for real datasets or 10 epochs for synthetic datasets with an initial learning rate of 7𝑒 -4 in the first phase. We fine-tune our model for another 10 epochs with an initial learning rate of 1𝑒 -4 in the second phase. All models are trained with a total batch size of 768 on 4 GPUs (192 images per GPU).Following previous state-of-the-arts [2, 44, 47] , we use ViT-Base [8] as visual encoder with a patch size of 4×8. Unless specified, the decoder depth 𝑁 𝑑 is set to 4, and the visual masking ratio 𝑟 𝑣 and linguistic masking ratio 𝑟 𝑙 are set as 0.75 and 0.2 respectively. We employ the Adam optimizer [20] together with the 1cycle [38] learning rate scheduler. During the second training phase, the Permutation Language Modeling (PLM) [48] introduced in [4] is also adopted for better context modeling and we set the number of permutations as 6. Following prior works [4, 35] , the maximum label length is set to 25. Images with label lengths larger than 25 will be neglected during training. During evaluation, we set the charset size as 36, including lower-case alphanumeric characters.For image pre-processing, RandAugment [6] with 3 layers and a magnitude of 5 excluding Sharpness is employed as our data augmentation strategy. Following PARSeq [4] , we also add Invert, GaussianBlur and PoissonNoise due to their effectiveness in STR task. After augmentation, all images will be resized to a fixed size of (32, 128) and will be normalized to [-1, 1].To make a fair comparison with prior arts, We follow the evaluation protocol of PARSeq [4] and choose word accuracy as our evaluation metric. We evaluate our model on thirteen benchmark datasets, including six standard benchmarks and seven more challenging benchmarks.We evaluate VL-Reader on six standard benchmark datasets, including three regular datasets (IIIT5K, SVT, and IC13) and three irregular datasets (IC15, SVTP, and CUTE80). Results are presented in Table 1 . When trained on synthetic datasets, VL-Reader achieves state-of-the-art performance. Specifically, in comparison to previous SOTA methods, VL-Reader exhibits superior results on IIIT5K (97.1%), SVTP (92.9%), and CUTE80 (92.7%), while maintaining competitive results on the remaining datasets. The utilization of real-world data further amplifies the performance enhancement for the VL-Reader model, enabling it to attain state-ofthe-art performance consistently across six standard benchmarks with an average accuracy of 96.9%. Furthermore, leveraging a hybrid dataset composed of both synthetic and real images, the VL-Reader model successfully maintains consistent performance improvements, ultimately attaining an average accuracy of 97.1% and setting new state-of-the-art benchmarks.The model is also evaluated on seven more challenging benchmarks, including two occluded datasets (WOST and HOST), two handwritten datasets (IAM and CVL), and three large-scale datasets (COCO, ArT, and Uber). As indicated in Table 2 , VL-Reader significantly outperforms previous works on the occluded datasets WOST(+4.4%) and HOST (+7.8%).One key factor may be that VL-Reader develops a robust crossmodal representation, enabling it to effectively handle visually challenging images. Furthermore, VL-Reader achieves a consistent performance boost across two handwritten datasets (IAM (+2.7%) CVL (+1.2%)) and three large-scale datasets (COCO (+2.4%), ArT (+0.7%) and Uber (+1.9%)). These results demonstrate the enhanced robustness of VL-Reader on large real world benchmarks.By utilizing the hybrid dataset of synthetic and real images, VL-Reader consistently attains improved performance. However, VL-Reader does not demonstrate improved results on Uber. This might be attributed to the fact that synthetic images are mainly horizontal, which could hamper the performance on Uber as it primarily contains vertical and rotated images.The proposed MVLR plays a key role in the training process of VL-Reader. We conduct several experiments to validate the effectiveness and explore the impact of MVLR.Table 1 : Word accuracy on six standard benchmark datasets. "S" represents synthetic datasets (MJ and ST), and "R" represents real datasets introduced by [4] . Superscript "-" and "+" represent using a subset of data and using external data respectively. The bold and underline results represent the best and the second-best respectively. † indicates results are from [4] .Train The results can be seen in Table 3 . By integrating linguistic knowledge, VL-Reader boosts recognition performance by +1.4%. With the addition of the proposed MVLR, there is an additional improvement of 0.8%, resulting in an overall improvement of 2.2%. Compared to experiment #2 which solely integrates linguistic information but is not trained under the MVLR objective, integrating MVLR could obtain a +0.8% performance gain, demonstrating the effectiveness of MVLR.𝑟 𝑣 . To examine the impact of varying visual masking ratios (𝑟 𝑣 ) on ultimate performance, we manipulated 𝑟 𝑣 while maintaining a constant 𝑟 𝑙 of 0.2 during the initial training phase. We fix the left hyper-parameters in various masking-ratio settings. The results are presented in Fig. 5 (a) . The performance reaches the peak with a masking ratio around 0.7-0.75 and gradually decreases when enlarging or decreasing the visual masking ratio 𝑟 𝑣 . When 𝑟 𝑣 is set to less than 0.65, the VL-Reader encounters a significant performance drop. The results demonstrate that the visual reconstruction task with an appropriate masking ratio is capable of boosting text recognition performance. As a result, we set the visual masking ratio 𝑟 𝑣 as 0.75 in our work.Analysis of linguistic masking ratio 𝑟 𝑙 . Similarly, we also examine the impact of varying linguistic mask ratios 𝑟 𝑙 . We set different 𝑟 𝑙 while maintaining a fixed 𝑟 𝑣 of 0.75. The results are shown in Fig. 5 (b ). The performance of VL-Reader reaches the peak when 𝑟 𝑙 is set around 0.15-0.2. However, further increasing the linguistic masking ratio results in a subsequent decrease in performance. This decline is attributed to the escalating training difficulty associated with enlarging 𝑟 𝑙 , as reconstructing 50% of characters based on the remaining 50% becomes increasingly challenging. We conduct an analysis of model size by implementing our model with different ViTs, specifically, tiny, small, and large. As can be seen from Table 4 , when employing ViT-Tiny, VL-Reader can outperform ABINet by +0.15% on standard benchmarks and by +1.2% on occluded benchmarks with much smaller model size (9M vs. 36M). When utilizing ViT-Small, our VL-Reader consistently outperforms ABINet by +1.18% and +6.75% on standard and occluded benchmarks with similar model size. Furthermore, VL-Reader with ViT-Base achieves 96.9% and 86.28% on such benchmarks, surpassing all previous methods. 4.3.5 Qualitative Results of Recognition. We perform qualitative comparisons of VL-Reader and previous state-of-the-art models on typical images from standard and more challenging benchmarks. As illustrated in Fig. 6 , we present some representative images to study the reason that VL-Reader succeeds but prior works fail. The results demonstrate that VL-Reader can obtain correct results on various challenging scenarios including occluded, artistic, blurred, and rotated images. Moreover, the VL-Reader is more robust to low-quality images and disturbances due to its comprehensive utilization of visual-linguistic context. For the occluded image "rock" (the second sample of "Occluded Text"), VL-Reader produces the correct answer while previous methods predict it as "book". In the heavily blurred scenario (refer to the "mandarin" sample, the fourth instance in the "Blurred Text"), other methods incorrectly identify "n" as "ri", while VL-Reader recognizes it accurately. For the rotated scenario (see "symphony", the second sample of "Rotated & Curved Text"), the character "N" is hard to be distinguished from character "R" from the visual perspective due to adhered strokes under a perspective view. Thus Previous methods all incorrectly predict the character "N" as "R", while VL-Reader correctly identifies the word "symphony" with a valid linguistic meaning. These results demonstrate the robustness of VL-Reader on various challenging scenarios. As depicted in Fig. 7 , VL-Reader effectively reconstructs most of the visual information even when the source image is heavily blurred (column#1). Additionally, the VL-Reader is able to reconstruct a character that is completely masked (the last character "g" in column#6). These reconstruction results confirm the successful acquisition of visuallinguistic representation by VL-Reader. Moreover, we also present a further discussion regarding the vision-language reconstruction results in our supplementary materials. 
conclusion and future work: 
 In this paper, we have presented a novel scene text recognition approach by tuning a vision and language reconstructor to a text recognizer. Our approach, based on mask and reconstruction, not only learns rich visual and semantic representation but also ensures consistency in pre-training and fine-tuning stages. Benefiting from the architecture and innovative modules, our model achieves state-of-the-art performance on standard STR, particularly demonstrating significant improvement in challenging scenarios. Moving forward, there are two potential directions for future expansion: developing a multilingual variant of the proposed method and extending it to line recognition and whole-image recognition. 
: 
 Retrospective studies show a steady stream of methods propelling the advancement of STR, including vision-dominated methods [16, 34, 35] , and language-aware methods [4, 9] . A comprehensive consideration of both vision and language is essential in designing advanced STR. Since text images possess both the textual texture in stroke patterns and the semantics in words or lines. The significance of the interplay between vision and semantics becomes evident when dealing with occluded characters, blurred backgrounds, or messy handwriting. Previous visiondominated methods [16, 34, 47] treat characters simply as distinct visual symbols and directly classify them into different categories mainly based on visual features, overlooking the underlying semantics. Recent language-aware methods [44, 46, 49] incorporate a language-aware module into the decoding stage to rectify recognition results, but fail to adequately account for the collaborative influence of vision and semantics. Both of these approaches fail to simultaneously consider bi-modal information in both the encoding and decoding stages. Vision-dominated decoder tends to prioritize visual facts, which is occasionally ambiguous, as exemplified by the blurred characters "m" and "n" in the Fig. 1 (a) representing "chimney". While language-dominated decoder is more likely to select a word with a higher frequency in the dictionary, such as "book" instead of "rock" as seen in Fig. 1 To enhance robustness against various visual conditions and improve the model's understanding of context and syntax in text, we advocate that an optimal text recognition model should possess the following three key properties: 1) representativeness: an encoder capable of representing the visual and semantic distribution, 2) multi-modal decoding ability: a decoder that ensures the alignment between vision and semantics, and 3) structural consistency: consistency in the framework during the pre-training and fine-tuning. In terms of representativeness, masked autoencoding has been shown to be effective in learning either vision ( MAE [11] ) or language ( BERT [19] ) representation. In this work, we investigate that text recognition itself can be viewed as the reconstruction of masked characters. Therefore, we can seamlessly transfer a vision-language reconstruction model into a text recognition model, requiring no extra layers. This results in a simple yet effective text recognizer, which we have named VL-Reader. Concretely, we first introduce a Masked Visual-Linguistic Reconstruction (MVLR) objective, which learns visual and semantic representations by means of self-supervised masked autoencoders. Then, we devise a Masked Visual-Linguistic Decoder (MVLD) to further leverage masked vision-language context and achieve bi-modal feature interaction. The architecture of VL-Reader maintains consistency from pre-training to fine-tuning. In the first training stage, the VL-Reader reconstructs both masked visual and text tokens, while in the second stage, the network degrades to reconstruct all characters from an image without any masked regions.Different from previous methods that reconstruct signals from either visual or linguistic modality, our work emphasizes the importance of jointly reconstructing both visual and linguistic signals (see Fig. 2 ). The proposed training objective MVLR forces effective cooperation between the visual and linguistic modalities, thereby aiding in building a strong cross-modal feature representation. Thanks to the generality of Transformer [40] , we are able to model visual and textual information, which have different levels of densities, in the same dimension. Furthermore, the entire architecture can be seamlessly transitioned from pre-training to fine-tuning, simply with a change of masking matrix, which will be detailed in Sec. 3.2. Experiments are conducted on six standard benchmarks as well as seven more challenging benchmarks. VL-Reader achieves stateof-the-art performance on all thirteen benchmarks (Table 1 and Table 2 ). The magnitude of improvement achieved is particularly significant when compared to the already high baseline accuracy in the field. Moreover, our method demonstrates an even more significant performance boost on seven more challenging datasets, additionally validating the effectiveness and robustness of our proposed VL-Reader.The contributions of our work are summarized as follows:• We propose VL-Reader, a novel STR approach that leverages masked vision and language for auto-encoding and reconstruction for text decoding. This method demonstrates a concise yet highly effective architecture.• We introduce masked visual-linguistic reconstruction for STR, which jointly learns representations of the vision and semantics of text images. • We design a cross-modal masked visual-linguistic decoder, which serves the dual purpose of supervising the reconstruction task and acting as the output for recognition results. • Experiments on extensive benchmarks show that the proposed VL-Reader outperforms the existing methods by a significant margin, demonstrating that vision and language reconstructor can serve as an effective scene text recognizer. 
 
 Appendix:  


Decision: The paper proposes a scene text recognition approach called VL-Reader based on vision and language reconstruction. Firstly, a Masked Visual-Linguistic Reconstruction (MVLR) objective to simultaneously model visual and linguistic information. Then, a Masked Visual-Linguistic Decoder (MVLD) is used to further leverage masked vision-language context and achieve bi-modal feature interaction. The architecture of VL-Reader maintains consistency from pre-training to fine-tuning. In the pretraining stage, VL-Reader reconstructs both masked visual and text tokens, while in the fine-tuning stage, the network degrades to reconstruct all characters from an image without any masked regions. 

Strengths:
- The paper is very well-written and clearly explains the motivation and effectiveness of each component. A detailed analysis and evaluation has been conducted and thoroughly described to explain how and why this is approach is superior to the prior ones.
- Every claim made in the paper has been well substantiated by theory as well as results.

Any of the concerns mentioned in the weakness by the reviewers have been well addressed in the rebuttal.

Reviews: review: paper_summary: paper_summary: In this study, a vision and language reconstructor model VL-Reader is proposed to recognize scene texts, where two-step (masked and non-masked) training strategy is designed to ecnode and decode paris of image and text, and cross-modal masked VL decoder is introduced to enhance cross-modal representation learning. Experimental result on 6 standard benchmarks and 7 challenging benchmarks have showed the superiority of the proposed model VL-Reader. The proposed model has achieved the best accuracy compared with the other methods. But the model's size seems to much be larger than other models like ABINet. This makes the model be hard to be used on edge devices.
strength_weakness: strength_weakness: The manuscript is well writen.The proposed VL-Reader has achieved the best performance among more than 20 baselines.
limitations: limitations: As for model size analysis, only one model ABINet is compared. More models and their inference speeds (FLOPs) had better be compared. Although the propsed method has achieved good performance, its model parameter seems large.
paper_summary: paper_summary: This paper proposes a VL-Reader method for scene text recognition. The core of VL-Reader is a Masked Visual-Linguistic Reconstruction (MVLR) loss, which simultaneously reconstructs visual and linguistic information. Then, a Masked Visual-Linguistic Decoder (MVLD) is introduced to further leverage masked vision-language context and achieve bi-modal feature interaction. The training strategy and architecture maintain consistency from pre-training to fine-tuning. Comprehensive experiments are conducted on 6 datasets.
strength_weakness: strength_weakness: 1. This paper shows comprehensive experiments on the effectiveness of the proposed MVLR, the visual masking ratio and linguistic masking ratio, model size, the qualitative results, and the reconstruction results.
2. The reconstruction is conducted for vision as well as the text, which is good for vision language information integration. MVLR can inherit the entire architecture across different training stages, rather than only use a vision encoder. 
3. The writing and representation are nice.
limitations: limitations: 1. This paper lacks comparison with more newly-published work, such as:
(1) “Image as a Language: Revisiting Scene Text Recognition via Balanced, Unified and Synchronized Vision-Language Reasoning Network” in AAAI2024; (2) “Self-supervised Character-to-Character Distillation for Text Recognition” in ICCV2023; (3) “Relational Contrastive Learning for Scene Text Recognition” in ACM MM2023, etc.
2. In equation (1), why the visible patch and masked pathed is processed in different ways? Is there any referenced work or experiments?
3. The vision reconstruction is conducted after MVLD. In the ablation study, the comparison between this strategy and those with reconstruction only after the vision encoder should be presented. 
4. In line 512, GPU card should be specified. 
5. In Table 1, with synthetic training data, [14][28][47] outperforms the proposed method on specific datasets. Explanations should be provided. 
6. Comparison or discussion with LLM-based methods is suggested.
paper_summary: paper_summary: This work propose a VLM-based text Recognizer where a novel masked visual-linguistic reconstruction is used to jointly learns representations of the vision and semantics of text images.
Experiments on extensive benchmarks show that the proposed VL-Reader outperforms the existing methods by a significant margin, and the analysis reveal how this method advance.
strength_weakness: strength_weakness: 1. The proposed method masked visual-linguistic reconstruction is very concise and highly effective.

2. This idea may serves similar V-L tasks intuitively,  providing good insights for multimodal researching.

3. Good writing and clear presentation.
limitations: limitations: I do not find major weakness for this work, except a minor concern: "The average performance of this task has been pushed to a very high level (near 100%) and the shown increase is relatively small." I think adding the significance analysis will help.

I will give a border score first and may rise it after considering other comments in the following discussing period.