Trust Prophet or Not? Taking a Further Verification Step toward Accurate Scene Text Recognition

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 14:40:44

Abstract: Inducing linguistic knowledge for scene text recognition (STR) is a new trend that could provide semantics for performance boost. However, most auto-regressive STR models optimize one-step ahead prediction (i.e., 1-gram prediction) for character sequence, which only utilizes the previous semantic context. Most non-auto-regressive models only apply linguistic knowledge individually on the output sequence to refine the results in parallel, which do not fully utilize the visual clues concurrently. In this paper, we propose a novel language-based STR model, called ProphetSTR. It adopts an n-stream self-attention mechanism in the decoder to predict the next 
 characters simultaneously based on the previous predictions at each time step. It could utilize the previous semantic information and the near future clues, encouraging the model to predict more accurate results. If the prediction results for the same character at successive time steps are inconsistent, we should not trust any of them. Otherwise, they are reliable predictions. Therefore, we propose a multi-modality verification module, masking the unreliable semantic features and inputting with visual and trusted semantic ones simultaneously for masked prediction recovery in parallel. It learns to align different modalities implicitly and considers both visual context and linguistic knowledge, which could generate more reliable results. Furthermore, we propose a multi-scale weight-sharing encoder for multi-granularity image representation. Extensive experiments demonstrate that ProphetSTR achieves state-of-the-art performances on many benchmarks. Further ablative studies prove the effectiveness of our proposed components.

Main paper: 
 related works 2.1 language-free str methods: 
 Language-free methods usually focus on leveraging visual features without considering the relationship between characters, such as [7, 8, 29, 38] . CRNN [29] employed CNN and RNN to model sequential visual features, and then directly fed them into a CTC decoder for prediction. DAN [38] built a convolutional alignment module, which performed alignment operations from a visual perspective and avoided using historical decoding information, thereby eliminating misalignments caused by decoding errors. ViTSTR [8] mainly focused on ViT-alike visual model construction and adopted a ViT training scheme for STR. SVTR [7] was designed to perceive inter-character and intra-character patterns using a mixture of global and local blocks, resulting in multi-grained character component perception within a single visual model. Some methods were implemented through segmentation. The segmentation-based STR methods utilized Fully Convolutional Networks (FCN) to perform pixel-level character segmentation. Liao et al. [17] employed a pixel grouping approach to recognize characters by forming text regions from the segmented pixels. Textscanner [34] introduced an additional segmentation map that accurately transcribes characters in the correct order. Language-free methods struggle to effectively address recognition challenges in low-quality images due to their limited access to linguistic information.To incorporate linguistic information, many language-involved methods have been proposed. Most of those methods [18, 28] followed the one-way semantic transmission manner that guided the encoded visual features to attend the corresponding region with the help of semantic information of previous prediction. Some STR methods [8, 24, 44] used pre-trained LM or visually independent LM to correct the inaccurate recognition results with the text context in the image. SRN [44] proposed a global semantic reasoning module to consider global semantic context information and effectively combine it with visual context information to enhance prediction accuracy. ABINet [8] proposed autonomous, bidirectional, and iterative principles to guide the design of LM in STR.Recently, many methods combined both visual and semantic modalities [1, 20, 39, 46] for STR. VisionLAN [39] proposed a languageaware mask to enhance the semantic features of visual information and proposed weakly-supervised complementary learning to generate accurate character-wise mask maps in MLM with only word-level annotations. JVSR [1] introduced a multi-stage character decoding paradigm with incremental refinement, where each stage utilized visual features for initial predictions, followed by a refinement step using combined visual-semantic information. MATRN [20] Visual token vℎ(1) 1 𝑜 + + + … … … …Untrusted semantic masking ( )Scalable Patch embedding (SPE)EQUATION𝑦 𝑇-1(1)𝑦 1(1) 𝑦 2(1)EQUATION𝑦 3(2) bi-directional fusions. LPV [46] introduced a global linguistic reconstruction method to perceive the linguistic information in the visual space, which converted visual features into semantically rich ones gradually during the cascade process. Xiao et al. [41] related the language model with text length and proposed a dual character counting-aware visual and semantic modeling network for STR.𝑦 4 𝑦(1) 𝑛 ≠ 𝑦(2) 𝑛 𝑦 0 𝑦 1 𝑦 𝑖 … 𝑦 𝑇 𝑦 𝑗 … 𝑣 1 𝑣 2 𝑣 3 𝑣 𝐿 … 𝑣 1 𝑣 2 𝑣 3 𝑣 𝐿 Muti-Head Self-Attention … 𝑦 𝑖 𝑦 𝑗 ×O 
proposed method 3.1 overall architecture: 
 The overall structure of the proposed method is present in ′ , the multi-scale encoder can obtain the coarse-grained image features. An attention operation is followed with a position query to remove redundant information and achieve the final position-aware visual features 𝑣 𝑙 .Afterward, PD is used to predict the next 𝑛 character in parallel with the n-stream self-attention mechanism, which takes the embedding of the previously predicted character 𝑦 𝑡 -1 as query and the visual features 𝑣 𝑙 as keys and values. 𝑛 special tokens are added with the input character embedding to indicate the steps of the next prediction.The prediction manner of the PD may generate inconsistent results. They should not be trusted and required to be verified again. So, we mask those untrusted character embeddings to simulate the case of missing character-wise linguistic semantics. Finally, MVM takes both the masked prediction character embeddings and the position-aware visual features as input to predict the final character sequence in parallel. To better capture the character information with various text lengths in the image, we propose to model the input image with a multiscale representation. Changing the patch resolution requires training new encoders, thus resulting in parameters increasing. To overcome this limitation, we use the SPE strategy for normalizing the input patches of different sizes without losing information.The SPE process is displayed in Alg. 1. The patch embedding for the patch 𝑃 𝑖 at scale 𝑖 is denoted as 𝑥 𝑖 ∈ R 𝑐 . The superscript label 𝑗 of 𝑥 𝑖 is ignored for a general patch embedding representation. We expect to find a "resized" patch embedding weights ŵ, so the patch representations before and after resizing remain the same, i.e., ⟨𝑥 𝑖 , 𝑤⟩ = ⟨𝑥 𝑖 * , ŵ⟩. After optimization, we find that the matrix to transform the patch embedding weights 𝑤 for a certain scale 𝑖 corresponds to the inverse of the bilinear resize operation 𝐵. The normalized patch embedding at any scale 𝑖 * could be acquired by step 5 EQUATIONDifferent scene text images might have their corresponding best resolutions for patch representation. Utilizing all the aggregated information of 𝑥 𝑖 is redundant, which hinders model learning and generalization. Therefore, we use length-fixed position queries [45] to extract positional-aware visual features through 𝑙 2 cross-attention layers:EQUATION𝑝 0 𝑙 ∈R 𝑑 (𝑙 ∈{1,. . . ,𝐿}) denote positional vectors, projecting from one-hot vectors at each location ranging from 0 to 𝐿. They are passed into the cross-attention module as the position query Q. The features of 𝑥 1:𝑠 are projected to the key K and value V. MHA(Q, K, V) denotes the multi-head attention operation [6] . 𝑝 𝑚+1 1:𝐿 is the output query after 𝑚 𝑡ℎ cross-attention layer. Finally, we get the positional aware visual tokens 𝑣 1:𝐿 (namely 𝑝 𝑙 2 𝑙 ).Algorithm For many Transformer-based decoders used in STR approaches [18, 32] , their target is to estimate the probability of the next character given previous predictions and visual context 𝑣 1:𝐿 , namely 𝑃 (𝑦 𝑡 |𝑦 <𝑡 , 𝑣 1:𝐿 ). However, we expect the decoder to learn to predict future n-grams simultaneously, i.e., 𝑃 (𝑦 𝑡 :𝑡 +𝑛-1 |𝑦 <𝑡 , 𝑣 1:𝐿 ). In this way, the decoder can utilize previous contexts and also consider future clues for predicting the current results. Inspired by the ProphetNet [43] for word prediction, we propose a character PD adopting n-stream weight sharing self-attention block to achieve this goal.As shown in Fig. 2 , PD changes the normal masked multi-head self-attention block to a masked multi-head N-stream attention block. This block is based on the generally masked self-attention but is divided into 𝑛 parallel masked self-attention modules to predict the next 𝑛 consecutive future tokens respectively at each time step. Specifically, the 𝑖 𝑡ℎ prediction stream is responsible for modeling the probability 𝑝 (𝑦 𝑡 +𝑖 -1 |𝑦 <𝑡 , 𝑣 1:𝐿 ). The input to PD is the embeddings of previously predicted character 𝑦 𝑡 -1 and 𝑛 learnable tokens 𝑅 𝑖 (𝑖 ∈{1,. . . ,𝑛})to indicate the order of prediction streams. Note that the N-stream attention block of ProphetNet is different from our PD. We find the mainstream attention of ProphetNet, which is the same as the masked multi-head self-attention in the traditional Transformer decoder for calculating the hidden states, is redundant and increases additional computational cost. Therefore, we remove it and use 𝑛 weight-sharing masked multi-head attention blocks as follows:EQUATIONwhere ℎ(𝑖) 𝑜 = (ℎ(𝑖) 𝑜 0 , • • • , ℎ(𝑖) 𝑜 𝑇 ), denoting the input sequence of the 𝑖 𝑡ℎ predicting stream in 𝑜-th layer of PD. 𝑇 is set as the maximum character sequence length. As the input of the first layer in Multi-Head N-stream Self-Attention, ℎ(𝑖) 0 𝑡 is the concatenation of learnable stream indication token 𝑅 𝑖 and the corresponding previous input character embeddings 𝑦 𝑡 -1 at time step 𝑡. During training, a lower triangular matrix is set to control each position to only focus on its previous tokens [8] for each prediction stream.𝑔(𝑖) 𝑜 is the output of the layer 𝑜 in 𝑖 𝑡ℎ masked self-attentionbased block. It interacts with the visual tokens 𝑣 1:𝐿 by cross-attention operation and the Feed Forward Network (FFN) to get the input ℎ(𝑖) 𝑜+1 of next (𝑜 + 1) 𝑡ℎ layer of PD as:EQUATION)The output of the last layer 𝑂 𝑡ℎ , i.e., ℎ(𝑖) 𝑂 , is linear projected to predict the next (𝑖 -1) 𝑡ℎ output character 𝑦 𝑡 +𝑖 -1 . Although the calculations of each stream are very similar, they are distinguished by different prediction steam tokens. Since each stream has the same structure, the weights can be shared among them.Character in the same position will be predicted 𝑛 times by different streams. In each prediction stream, its referring context is different because the 𝑖 𝑡ℎ stream prediction sees previous context and future 𝑛-𝑖 character clues. Therefore, the decoder may generate inconsistent results for predicting the same character in different streams. If so, they are considered unreliable predictions. Otherwise, they can be trusted.Some STR methods [8, 47] refine the predicted character sequences using a combination of context-free vision and context-aware LMs. However, they are prone to erroneous rectification of correct initial predictions since the LM works conditional independently on the image features.Our proposed PD provides a great criterion to determine the confidence of prediction results through the n-stream prediction way, which could avoid rectifying the initial correct predictions. Only the predicted unreliable results are required to be verified. Therefore, a further verification module, i.e., MVM, is proposed to re-predict the unreliable characters in parallel by referring to multimodality vision and semantic information. It considers both the trusted characters with corresponding orders and position-aware visual features 𝑣 1:𝐿 simultaneously, resulting in a more efficient and robust STR refinement process.MVM has 𝑙 3 stacked Transformer layers, each of which consists of a self-attention layer and a feed-forward layer. Inspired by MLM in NLP models, the untrusted character predictions are replaced with special [mask] tokens, while the trusted ones are represented by their corresponding character embeddings. To better align with the position-aware visual features output from the encoders, the masked semantic embeddings 𝑦 𝑀 1:𝑇 are also added with the positional embeddings 𝑝 1:𝑇 to indict the sequence order:EQUATIONThe position-aware visual tokens 𝑣 1:𝐿 as well as the masked position-aware semantic features 𝑦 𝑀 1:𝑇 ′ are input to the MVM. Between them, a special separator token [SEP] is inserted to distinguish the two modalities. The two kinds of features are interacted through the self-attention mechanism as follows:EQUATIONThe superscript 𝑢∈{1,. . . ,𝑙 3 }) denotes the 𝑢 𝑡ℎ self-attention layer. In the first layer, 𝑦 0 1:𝑇 = 𝑦 𝑀 1:𝑇 and 𝑣 0 1:𝐿 = 𝑣 1:𝐿 . In the last Transformer layer, we only use the interacted semantic features 𝑦 𝑙 3 1:𝑇 for character prediction.This step is implemented by a parallel linear transformation. Concretely, a fully connected layer and softmax operation are employed to generate a transcript sequence of size 𝑇 , where ideally, the prediction results of trusted characters remain the same, while each of the masked characters is classified into a certain character category. The output character sequence is determined by incorporating both trusted multi-modality features, thus being more reliable than that of PD. It is noted that the masked semantics are reasoned in a parallel way for a single time in the MVM, making it run efficiently to generate more accurate results.The loss function is composed of two items. Since the prediction step by PD has 𝑛 different streams to predict the future 𝑛 characters, each prediction stream adopts an independent cross-entropy loss for training. This objective function is formalized as:EQUATION)where 𝜃 is the trainable parameters of the model. The first item of the second equation in Eq.( 7) is a traditional LM to learn the next prediction, and the second item is the objective for future predictions. 𝜆 𝑗 is the weight of 𝑗 𝑡ℎ stream to balance the importance of different prediction streams.The cross-entropy loss is also used to supervise the learning of MVM, which is formulated by:EQUATION) 
experiments 4.1 datasets: 
 MJSynth (MJ) [13] and SynthText (ST) [10] are synthetic text image datasets, which contain 9 million and 8 million text images respectively. We use their union as our training data. Six common benchmarks contain three regular text datasets and three irregular text datasets are used for the test. The regular text datasets are IIIT5K-Words (IIIT5K) [19] , Street View Text (SVT) [37] and IC-DAR 2013 (IC13) [16] . The irregular text datasets are ICDAR 2015 (IC15) [15] , Street View Text Perspective (SVTP) [22] , and CUTE 80 (CUTE) [27] . Images in these datasets are most curved and distorted. WordArt dataset (artistic text) [42] is used for testing to evaluate the generalization of scene text recognizer.In addition, we also train and test the model with the real dataset Union14M-L [26] , which contains 3.2 million samples collected from publicly available datasets. The test set of Union-14M-Benchmark contains 0.4M samples in a variety of situations, such as Curve, Multi-Oriented, Artistic, Contextless, Salient, Multi-Words, and General.We first resize the text images to 36 × 144 and use typical image processing ways, e.g., adding Gaussian noise, rotation, perspective distortion, and motion blur, for data augmentation, To train our model, the initial learning rate is set to 4 × 10 -4 . The first 10k iterations are used for warm-up. The whole training iterations are determined as: 𝑙𝑟 = 𝑑 -0.5 𝑚𝑜𝑑𝑒𝑙 • 𝑚𝑖𝑛(𝑛 -0.5 , 𝑛 • 𝑤𝑎𝑟𝑚 -1.5 𝑛 ), where 𝑛 and 𝑤𝑎𝑟𝑚 𝑛 represent the number of normal iterations and warm iterations. 𝑑 𝑚𝑜𝑑𝑒𝑙 is set to 384. The maximum length of the model output sequence 𝑇 is set to 26. Experimentally, we set 𝑙 1 =𝑙 2 =𝑙 3 =4 and 𝑂=4, respectively. The hyper-parameters 𝜆 𝑗 are set equally. To train MVM, we mask 50% predicted characters of PD randomly. During inference, we use a lowercase alphanumeric charset, and the number of character classes is set to 36. The models were trained . 1. The first four rows show the results using different single-scale patch embedding in the encoders. We can see that the smaller the patch resolution, the higher the accuracy rate. The model E 6 achieves the highest average accuracy rate on both regular datasets and irregular datasets than other single-scale models, but its inference time is the slowest. Model E 9 gets comparable performance with model E 6 . But, it takes half as much time for inference. The averaged recognition accuracy of E 12 and E 9 are approximately equal. With the patch resolution becoming larger, the recognition accuracy model drops gradually.The last three rows in Table . 1 presents the results that take multi-scale patch sizes as input. We can see that the model E 9,12 surpasses E 6 on almost all the datasets with less computational time. With more patch resolutions used in the encoders, the recognition accuracy is increasing continually. This demonstrates that integrating multi-resolution patch tokens is important for effectively capturing scene text image information.To explore the effectiveness of predicting n-gram, we compare our model with setting n=1, 2, and 3 in the PD. Two encoders, E 12 and E 9,12,18 , are used to remove the influence brought by the proposed image encoding module. We can see that the performance of PD 2 and PD 3 is better than PD 1 on all of the six datasets whatever the encoder is. It demonstrates that using a n-gram (n>1) character prediction strategy is better than a single-gram one. However, the performance of using higher grams for character prediction in scene text recognition tasks might be decreased. The gap between the 2-gram model and the 3-gram model is not large. We check the incorrectly recognized text and find that the 3-gram model has the problem of recognizing short text. We list some examples in Fig. 2 . In addition, the computational and time cost becomes higher with the increasing prediction streams in PD. The inference time of E 9,12,18 plus three different prediction streams (i.e, PD 1 , PD 2 and PD 3 ) are 20.95ms, 23.40ms and 26.22ms, respectively. Comprehensively, we select PD 2 as our decoder in other experiments. Some visualization of the character's attention in different prediction streams of the decoder is present in Fig. 3 . The 𝑖 𝑡ℎ row displays the attention at time step 𝑖-1. We can see that the model of E 12 +PD 2 could pay attention to the areas of corresponding characters more correctly. The second stream of PD 2 could also pay high attention correctly to the next character during prediction. It could promote the concentration of character attention for the first stream.To evaluate the effect of different modules of our proposed method, we conduct experiments on eight separate models and display the results in Table . 5. If a multi-scale encoder (i.e., E 𝑚𝑠 ) is not selected, it represents we only use a singlescale encoder E 12 as input. Otherwise, we adopt E 9,12,18 . If PD is not selected, it means we only use PD 1 as a decoder which degrades to a normal Transformer-based decoder. If it is selected, that means the model uses the two-stream PD 2 . If MVM is not selected, the The results in the first row are the baseline which is a simple encoder-decoder-based STR model. The following three rows present the performance of individual effects of multi-scale visual encoder, PD, and MVM. We can see that they improve the averaged accuracy by 2.92%, 2.56%, and 2.5% respectively compared with the baseline model, demonstrating their positive effect for boosting the recognition performance separately.The results from 5 𝑡ℎ to 7 𝑡ℎ rows display the pair-wise combination effect of different modules respectively. The recognition rates are increment. Among them, the model adopting a multi-scale strategy and PD gets the highest accuracy. It illustrates our proposed encoder and decoder model could generate relatively reliable results. After combining the three modules, the ensemble model brings about a further performance boost. The results of the ensemble model are better than that of any pair-wise module combined models, which also proves the importance of each module for STR. Additionally, the average inference time of the ensemble model is 32.82ms. Among them, the modules E 6,9,12,18 +PD 2 take about 29.57ms and MVM takes about 3ms.We first compare our proposed full models with the state-of-the-art (SOTA) methods on seven STR benchmark datasets, including the artistic text dataset WordArt. ProphetSTR 9,12,18 represents the full model of E 9,12,18 +PD 2 +MVM, and ProphetSTR 6,9,12,18 represents that of E 6,9,12,18 +PD 2 +MVM. All the comparing methods are trained on MJSynth and SynthText. Results can be found in Table 3 . Our method achieves superior performance compared with the other methods. Notably, our proposed ProphetSTR 6,9,12,18 obtains the highest average scores on all the datasets.When we compare the model ProphetSTR 6,9,12,18 with others trained on Union14M (without pre-training), our method could also achieve SOTA performance. It almost obtains the best performance on all the datasets of common benchmarks, and performs the best, especially for curve, artistic, and general scene text for the Union14M dataset.Compared with language-free STR methods, e.g., CRNN [29] , ASTER [30] , and some language-aware STR methods, e.g., SRN [44] , ABINet [8] , VisonLAN [39] and LPV [46] , our model has shown significant improvements on almost all the datasets. It implies the strong recognition capabilities of our proposed model. MATRN [20] combines visual and semantic features for AR STR. But, instead of masking semantics, it masks regions on the image. Compared with it, our method outperforms it on most datasets when trained with synthesis data and achieves comparable results when trained with real data. It further illustrates the effectiveness of using multimodality information for STR.In this section, we show the recognition accuracy of different streams in PD and MVM on different test datasets by training with the Union14M-L dataset. Note that the second stream of PD starts the prediction from the second character of the text images. Thus, we count the accuracy of this stream by ignoring the first character of ground truth. The results are shown in Table. 6 . We can see that the recognition accuracy of the first stream is much higher than that of the second stream in PD. It may rise from the reason that the output from the second stream refers to the previous semantic information for predicting character at 𝑡+1 time step. In contrast, the first stream refers to both previous predictions and future 𝑡+1 information for predicting the output of time step 𝑡. The first stream could collect more clues than the second stream in PD. After adding the verification module, we can see that MVM can improve the recognition performance on all the datasets, especially for the Curve data of the Union14M benchmark, which boosts the recognition rate by 7.4%. It demonstrates the essential effectiveness of the MVM for recognition verification. Some prediction results of the PD and MVM are positioned in Fig. 4 . We can see the inconsistent predictions of the two streams in the PD are corrected by MVM.Since the input of MVM is based on the inconsistency of the outputs from different streams in PD, we further count the character-level inconsistency recognition rate f. The inconsistency rate is computed by:EQUATIONwhere the numerator is the accumulated numbers of inconsistent character recognition, results in the two streams of PD. Table . 7 shows the results of the models trained on different datasets. We find that this rate is larger on irregular datasets. Compared with models trained on synthetic data, training on the real dataset, i.e., Union14M, can reduce the recognition inconsistency and obtain more accurate results. However, this great inconsistency rate motivates us to propose MVM for boosting the performance and eliminating the unreliable character recognition results. More inconsistent recognition results of PD 2 and the MVM rectified results are displayed in the Appendix file. 
conclusion: 
 In this paper, we proposed a novel scene text recognizer, Prophet-STR, which consisted of a multi-scale encoder, a prophet decoder (PD), and a multi-modality verification module (MVM). The encoders employed a scalable patch embedding strategy to handle inputs with different patch resolutions, allowing multi-scale patch information aggregated image representation. PD predicted the next consecutive steps of characters by utilizing future information. MVM could generate more accurate results based on both visual features and reliable semantic clues. By assembling the three modules, ProphetSTR shows powerful performance on different datasets. 
: 
 and language model (LM). It has been widely used for many applications, e.g., document analysis [36] , cross-modal retrieval [35] , autonomous driving [9] , and text-based attacks [40] .Recently, many deep learning-based STR methods have been proposed. Most of them follow the paradigm of the encoder-decoder pipeline, where the encoder is responsible for visual feature embedding and the decoder for character sequence prediction. As the text owns linguistic characteristics, the linguistic information is generally used in STR models to improve recognition performance, especially for challenging cases such as occlusion, blur, and distortion.Typically, there are two streams to use the linguistic information for STR. One stream is the auto-regressive (AR) STR [4, 23] generally adopting the generative LM [2] . It estimates the probability distribution of the next character based on the previous predictions at each time step. It is widely used for sequence modeling and sequence-to-sequence (Seq2Seq) learning [25, 31] . The second steam is the non-auto-regressive (NAR) STR [8, 24] that applies visually independent LM to refine recognition results from the vision model. It could output the prediction sequences in parallel by considering the bidirectional context.However, the AR STR methods learn limited linguistic knowledge since they only achieve context from one direction. An intuitive way to capture the previous and future information is to construct a bidirectional model [21, 32] , which merges the results from a left-to-right decoder and a right-to-left decoder. However, since the bidirectional decoders model the semantic features in different directions respectively, they could not share the same weights by using a single decoder. The ensemble models mean the twice amount of parameters.To tackle this problem, we propose a Transformer-based model, called ProphetSTR, to effectively model the linguistic rules of scene text, leveraging both previous semantic context and future clues with a single prophet decoder (PD). It predicts future n-gram (n>1) characters simultaneously at each time step. In addition to the traditional LM or Seq2Seq model in STR approaches that optimize one-step-ahead character prediction, the prophet decoder learns n-step-ahead prediction. This future n-gram prediction serves as extra guidance that explicitly encourages the model to plan for future tokens and make decisions based on both previous context and future clues.The architecture of the PD is based on the original Transformerbased decoder [33] , which is composed of masked multi-head selfattention blocks, cross-attention blocks, and feed-forward layers. Instead, we introduce 𝑛 learnable tokens interacting with previous prediction information for future n-gram prediction. The weights are shared for each stream in the decoder. Thus, there is no great increase in the model size.Each character in the scene text image will be predicted 𝑛 times by referring to different context information. The prediction results may be unreliable if they are inconsistent. Otherwise, they can be trusted. Inspired by the training strategy, i.e., Masked Languageaware Module (MLM), in Natural Language Processing (NLP) methods [5, 43] , we propose to use an additional multi-modal-based module to further verify the prediction results based on the trusted outputs and vision features. The unreliable prediction results are masked and input to the verification module with trusted character embeddings and position-aware visual tokens. The objective of the module is to predict the true character category of the masked tokens based on the vision clues and LM simultaneously. Since the trusted characters provide partial order-aware semantics of the text, the position-aware visual features could be aligned with them implicitly. It benefits the module to learn visual-related LM. Unlike most NAR STR methods that implement iterative steps [8, 23] , this verification module is only executed a single time, which is sufficient to correct the prediction errors.Furthermore, we explore more rational visual representations of scene text images and propose a Transformer-based multi-scale encoder to further boost the STR performance. Most visual feature encoders follow the pipeline of Vision Transformer (ViT) [6] by splitting each image into a sequence of pre-defined size fixed patches and then inputting the patches to Transformer layers after the patch embedding process. The experimental results demonstrate the patch size effect on the classification accuracy of different datasets. Therefore, instead of using a fixed patch resolution, we use multi-scale patch representations, enabling the Transformer's self-attention mechanism to capture scene text information on both small and large patches.However, it is not straightforward to apply the same Transformerbased encoder on the input patches with different sizes. Changing the patch resolution requires training a completely new encoder, thus resulting in parameter increases. To overcome this limitation, we propose to use a novel patch embedding strategy for normalizing the input patches of different sizes without losing information. The normalized patch embeddings are input to the encoder with corresponding scale embeddings for multi-scale scene text image representation.The combination of the multi-scale encoders, PD, and MVM creates a powerful scene text recognizer. Extensive experiments illustrate the effectiveness of the proposed ProphetSTR model and also show its superiority when compared with the state-of-the-art (SOTA) method.In summary, the main contributions of this paper are fourfold:• We propose a strong Transformer-based STR model, i.e., ProphetSTR, exploring the improvements on both visual representations and LMs. • Multi-prediction streams are adopted in the prophet decoder, which could provide the credibility of the results by prediction consistency. Additionally, the unreliable predictions are refined by a novel proposed multi-modal verification module to boost the recognition accuracy using both visual and semantic features with a masking strategy.  
 
 Appendix:  


Decision: The reviewers have come up to the following strengths and limitations

STRENGTH
- Prophet Decoder for Future n-Gram Prediction
- Multi-Modal Verification Module
- Transformer-Based Multi-Scale Encoder
- Innovative Approach to STR
- State-of-the-Art Results

LIMITATIONS
- Lack of Clarity on n-Gram Motivation (it is better after the rebuttal)
- Predicting Future Tokens needs more discussion
- Unclear Mask Generation Strategy (it is better after the rebuttal)
- Unclear Verification Step Motivation
- Modest Improvement Over SOTA
- Unclear Description of Prophet Decoder
- Discrepancy in Output Depiction

The rebuttal did well, and all reviewers agreed to raise their rating. Average rating 3.33

Reviews: review: paper_summary: paper_summary: The authors propose ProphetSTR, a novel Scene Text Recognition (STR) model that uses an n-stream attention mechanism to predict multiple characters simultaneously, leveraging both past and near future semantic clues for improved accuracy. They introduce a multi-modality verification module to handle unreliable predictions by combining visual and trusted semantic features. Additionally, a multi-scale weight-sharing encoder is used for detailed image representation. ProphetSTR demonstrates state-of-the-art performance on benchmarks, validated by extensive experiments and ablation studies.
strength_weakness: strength_weakness: 1. Prophet Decoder for Future n-Gram Prediction: ProphetSTR's prophet decoder (PD) predicts future n-gram characters simultaneously. This n-step-ahead prediction guides the model to plan for future tokens while considering both previous context and future clues.
2. Multi-Modal Verification Module: a multi-modal verification module that uses vision features and trusted outputs to verify predictions.
3. Transformer-Based Multi-Scale Encoder: ProphetSTR uses multi-scale patch representations to capture scene text information at various scales by normalizing input patches of varying sizes without losing information and inputs them to the encoder with corresponding scale embeddings.
limitations: limitations: 1. It is not clear what brings the difference between 1-gram and n-gram and the motivation for using n-gram from the paper. Moreover, the idea of "predicting the next 𝑛 consecutive future tokens respectively at each time step" can be seen in the graph transformer problem, where at the current time (Tc), the model can predict the next n time slices after (Tc+1, Tc+2, ... Tc+n). As the core of the proposed method, this point should have more discussion and explanation.
2. Is there any strategy for generating masks in the n-parallel masked multihead attention block?
paper_summary: paper_summary: This paper introduces ProphetSTR, a novel scene text recognition (STR) model that leverages linguistic knowledge and visual context for improved accuracy. Unlike traditional models, ProphetSTR uses an n-stream self-attention mechanism to predict characters based on both past and near-future context. It includes a multi-modality verification module to mask unreliable predictions and a multi-scale weight-sharing encoder for enhanced image representation. Extensive experiments show that ProphetSTR achieves state-of-the-art performance across various benchmarks, validating the effectiveness of its innovative components.
strength_weakness: strength_weakness: The idea of predicting n-gram with a verification step sounds interesting.
limitations: limitations: It would be beneficial to provide a clearer explanation of the motivation behind predicting n-grams with the verification step and how this approach offers advantages over 1-gram prediction.

Additionally, while the improvement compared to other state-of-the-art methods is evident, it appears relatively modest.
paper_summary: paper_summary: The paper introduces a scene text recognition (STR) model named ProphetSTR. The model incorporates a multi-scale encoder, a prophet decoder (PD), and a multi-modality verification module (MVM) to enhance the accuracy of scene text recognition. The multi-scale encoder handles inputs with varying patch resolutions, the PD predicts characters using future information, and the MVM refines predictions based on visual features and reliable semantic clues. The paper reports state-of-the-art performance on multiple benchmarks and an extensive set of experiments to validate the contributions of different components.
strength_weakness: strength_weakness: 1. The paper presents a unique approach to STR by introducing a prophet decoder that utilizes future information for character prediction.

2. The model achieves state-of-the-art results on multiple benchmarks, indicating a significant advancement in the field of scene text recognition.
limitations: limitations: 1. The description of the Prophet Decoder is unclear. It is not well understood how the authors implement the masked multi-head N-stream attention block, and how the PD decodes the output sequence from the n streams during the inference phase. Please provide a detailed explanation.

2. The output of the Prophet Decoder in Figure 1 is depicted as $(y^{2}_2, y^{2}_3, y^{2}_4, \ldots, y^{2}_T)$ and $(y^{1}_1,y^{1}_2,y^{1}_3,\ldots,y^{1}_{T-1}$. According to the paper, there should be two predictions at each position, but at time step 1, only one prediction is shown. Could you please clarify this discrepancy?