Title: Boosting Semi-Supervised Scene Text Recognition via Viewing and Summarizing
Venue: NeurIPS 2024
Accepted: yes
Keywords: Scene text recognition

Semi-supervised learning

Contrastive Learning
Abstract: Existing scene text recognition (STR) methods struggle to recognize challenging texts, especially for artistic and severely distorted characters. The limitation lies in the insufficient exploration of character morphologies, including the monotonousness of widely used synthetic training data and the sensitivity of the model to character morphologies. To address these issues, inspired by the human learning process of viewing and summarizing, we facilitate the contrastive learning-based STR framework in a self-motivated manner by leveraging synthetic and real unlabeled data without any human cost. In the viewing process, to compensate for the simplicity of synthetic data and enrich character morphology diversity, we propose an Online Generation Strategy to generate background-free samples with diverse character styles. By excluding background noise distractions, the model is encouraged to focus on character morphology and generalize the ability to recognize complex samples when trained with only simple synthetic data. To boost the summarizing process, we theoretically demonstrate the derivation error in the previous character contrastive loss, which mistakenly causes the sparsity in the intra-class distribution and exacerbates ambiguity on challenging samples. Therefore, a new Character Unidirectional Alignment Loss is proposed to correct this error and unify the representation of the same characters in all samples by aligning the character features in the student model with the reference features in the teacher model. Extensive experiment results show that our method achieves SOTA performance (94.7\% and 70.9\% average accuracy on common benchmarks and Union14M-Benchmark). Code will be available.

Boosting Semi-Supervised Scene Text Recognition via Viewing and Summarizing

Abstract

Existing scene text recognition (STR) methods struggle to recognize challenging texts, especially for artistic and severely distorted characters. The limitation lies in the insufficient exploration of character morphologies, including the monotonousness of widely used synthetic training data and the sensitivity of the model to character morphologies. To address these issues, inspired by the human learning process of viewing and summarizing, we facilitate the contrastive learning-based STR framework in a self-motivated manner by leveraging synthetic and real unlabeled data without any human cost. In the viewing process, to compensate for the simplicity of synthetic data and enrich character morphology diversity, we propose an Online Generation Strategy to generate background-free samples with diverse character styles. By excluding background noise distractions, the model is encouraged to focus on character morphology and generalize the ability to recognize complex samples when trained with only simple synthetic data. To boost the summarizing process, we theoretically demonstrate the derivation error in the previous character contrastive loss, which mistakenly causes the sparsity in the intra-class distribution and exacerbates ambiguity on challenging samples. Therefore, a new Character Unidirectional Alignment Loss is proposed to correct this error and unify the representation of the same characters in all samples by aligning the character features in the student model with the reference features in the teacher model. Extensive experiment results show that our method achieves SOTA performance (94.7% and 70.9% average accuracy on common benchmarks and Union14M-Benchmark). Code will be available at https://github.com/qqqyd/ViSu.

1 Introduction

Scene text recognition (STR) aims to recognize text in cropped text images. As a fundamental task, STR can provide auxiliary information for understanding natural scenes and has wide applications in financial systems, virtual reality, and autonomous driving. Because of the expensive and timeconsuming annotation process, most STR methods turn to two commonly used synthetic datasets, MJSynth [14] and SynthText [12]. Despite achieving satisfactory recognition results, they still struggle to perform well in challenging real-world scenarios. We attribute the limitations to insufficient exploration of character morphologies. First, as shown in Fig. 1(a, b, c), synthetic data are almost simple samples, whose character morphologies are significantly different from the artistic and distorted real-world characters. The limitations exhibited by training models using only synthetic data are further amplified on the challenging benchmarks WordArt [45] and Union14M-Benchmark (Union-B) [15]. For example, equipped with a language model, ABINet [10] only reaches 67.4% and 46.0% accuracy on WordArt and Union-B. Second, the model is not robust to the morphological

red

merry

isn't

the

sign

girl

(a) Simple synthetic samples

(b) Different representation

of character 'r'

(c) Similar representation

of different characters

(d) ParSeq

Percipience

socrum

YOURSELF

like

was

Day

Sender:

P

C

e

C

merry

Ssn't

sign

gi

tgat- Gifr.g

changes in characters. As shown in Fig. 1(b, c), due to severe distortion and artistic style, a character can visually differ from others of its category, and on the contrary, characters belonging to different categories can have similar appearances. Fig. 1(d) further shows the distribution of challenging character features in ParSeq [2] from t-SNE [38], where the characters cannot be clearly distinguished.

To compensate for the diversity of character morphologies, some previous methods [1, 10] introduce real unlabeled data by first training a model in the supervised way and generating pseudo-labels. However, it relies heavily on the quality of the pretrained model. The wrong pseudo-label will accumulate the errors and harm the training process. TRBA-cr [54] and Yang et al. [47] employ character-level consistency regularization to train the model in a semi-supervised manner and align the sequence output from two views of unlabeled data. Nevertheless, simply assimilating more real unlabeled data and ignoring the diverse character morphologies can lead to improvement on common benchmarks but little superiority on the challenging Union-B. To enhance the robustness to character styles, some methods [44, 20] synthesis the texts with unified styles and map the texts with different writing styles to the invariant features. However, they focus on handwritten mathematical expressions without the diverse orientations and affine transformations common in complicated scene text images.

In this paper, we attribute the bottleneck of the STR model to the monotonousness of training data and sensitivity of the model to character morphologies . Inspired by the human learning process, we divide the scene text recognition into two steps: viewing and summarizing. Viewing means that when encountering a new language, the first thing is to collect and see as many characters with different morphologies as possible. This also conforms to the principle that deep learning models are data-driven. After seeing enough samples, we can summarize their visual feature commonalities and classify them into different characters. With a small amount of label guidance, we can continuously optimize this cognitive process and distinguish some challenging characters, which is referred to as summarizing. Following this paradigm, we propose our semi-supervised scene text recognition framework by Viewing and Summarizing (ViSu).

To enrich the diversity of character glyphs in the viewing process, firstly, real unlabeled data is similar to practical application scenarios and has a variety of character morphologies. Based on contrastive learning, we adopt the Mean Teacher framework to jointly optimize the model using real unlabeled data and simple synthetic data. Secondly, to further enrich the training data and compensate for the simplicity of synthetic data, we propose an Online Generation Strategy (OGS) to generate background-free samples with challenging styles based on the annotations of the corresponding synthetic data. Compared with the vanilla Mean Teacher framework that simply uses labeled data to train the student model, our method equipped with OGS allows the teacher model to generate guidance of various character styles without the interference of background noise. Thereby, the student model can learn practical features robust to character morphological diversity. Without any human annotation, the model is able to generalize the ability to recognize challenging samples while only training with simple synthetic data, which substantially raises the performance upper bound of the semi-supervised learning framework.

After viewing many characters with various shapes, in order to imitate the summarizing process, the model needs to identify the commonalities of each character and cluster the same characters. Firstly, as shown in Fig. 2, the eight inconsistent representation forms of scene text images caused by reading order and character orientation undoubtedly aggravate the convergence burden. We unify

them into two primary forms based on the aspect ratio to enable the model to extract applicable and discriminative character-level features. Secondly, we theoretically prove that the previous character contrastive loss [45] confuses some character classes and incorrectly encourages the sparsity of the intra-class distribution, thus hindering the optimization of the model. Therefore, a novel Character Unidirectional Alignment (CUA) Loss is proposed to eliminate this mistake and focus on character morphological feature alignment to achieve character clustering within the same class. The images from OGS and weak augmented images, referred to as base images, make it easier to obtain noise-free and glyph-diversified character features that help recognize complex samples. CUA Loss forces the character features under strong data augmentation to be aligned with the base image features from the teacher model, allowing the model to obtain unified features for each category of characters.

The main contributions are as follows: 1) We propose the viewing and summarizing paradigm to conduct the training of the semi-supervised model. Our model can adapt to complex scenarios such as multi-orientation and multi-morphic characters without any human annotation. 2) An Online Generation Strategy is proposed to facilitate model generalization from simple synthetic data to recognize challenging samples, which in turn improves the performance upper limit of the semisupervised learning framework. 3) A novel Character Unidirectional Alignment Loss is proposed to theoretically correct the formula error of regarding some positive samples as negative samples and enhances the compactness of the intra-class distribution. 4) Our method achieves state-of-theart performance with an average accuracy of 94.7% on common benchmarks and 70.9% on the challenging Union14M-Benchmark.

2 Related Work

2.1 Scene Text Recognition

STR methods can be roughly divided into language-free and language-based methods, most of which are trained in a fully supervised manner on synthetic datasets. Due to the simplicity of synthetic data and the diversity of text styles in real scenes, existing methods still have difficulty recognizing complex samples. Some language-free methods [34, 22, 23, 46, 51, 55] design different rectification modules to convert distorted irregular text into regular text to improve recognition accuracy. CornerTransformer [45] proposes to utilize the corner points of artistic text to guide the encoder in extracting useful features from characters. The language-based methods [42, 2, 10, 41, 52, 49] do not rely solely on the visual features of text images. They infer the contextual information of complex characters by introducing additional linguistic knowledge, enabling the model to correct the visual recognition results. MGP-STR [41] proposes a multi-granularity (character, subword and word) prediction strategy to implicitly integrate linguistic knowledge with the model.

2.2 Semi-Supervised Learning

As widely acknowledged, deep learning methods require substantial labeled data for effective model fitting. However, acquiring a large amount of high-quality labeled data is typically challenging. Therefore, semi-supervised learning aims to improve model performance by leveraging large amounts of unlabeled data and limited labeled data. Yang et al. [48] comprehensively analyses semi-supervised learning methods. They can mainly be categorized into deep generative methods [36, 7, 26], consistency regularization methods [29, 31, 19, 37], and pseudo-labeling methods [3, 8, 21, 4]. CatGAN [36] considers the mutual information between observed samples and their class distribution by modifying the original GAN loss. Temporal Ensembling [19] requires consistent predictive outputs of data under various regularization and data augmentation conditions. Mean Teacher [37] utilizes the exponential moving average (EMA) to update the parameters of the teacher model during training, which tends to generate a more accurate model. Co-training [3] models posit that each sample in the dataset has two distinct and complementary views, and either view is sufficient to train a good classifier.

2.3 Semi-Supervised Text Recognition

Some researchers propose integrating semi-supervised learning with text recognition to enhance model recognition performance. Baek et al. [1] chooses to pre-train a model only using a small amount of real labeled data, and then further enhances the model performance using Pseudo Label or Mean Teacher with unlabeled data. ABINet [10] proposes an ensemble self-training strategy to

(b)

(a)

Standar d

party

soft

FORNI

pr

pue's

Ndard

S

enhance model recognition performance. However, these two methods require two-stage training, which greatly reduces the training efficiency of the model. Zheng et al. [54] is the first to apply a semi-supervised framework based on consistency regularization to the STR task. It designs a character-level consistency regularization to align the characters of the recognition sequence. Gao et al. [11] employs well-designed edit reward and embedding reward to assess the quality of generated sequence and optimizes the network using reinforcement learning. Seq-UPS [27], considering that generated pseudo labels may not always be correct, proposes a pseudo label generation and an uncertainty-based data selection framework.

3 Methodology

As illustrated in Fig. 3, our framework adopts the Mean-Teacher architecture. The student model and teacher model use the transformer-based encoder and decoder. The parameters of the student model Θ s are updated through the backpropagation algorithm, while the parameters of the teacher model Θ t are an exponential moving average (EMA) of Θ s . It can be formulated as Θ t ← α Θ t +(1 -α )Θ s , where α ∈ [0 , 1] is the smoothing factor. For the viewing process, we introduce not only augmented real unlabeled data but also the Online Generation Strategy (OGS) to obtain samples with challenging glyphs to eliminate the model's sensitivity to character morphology. For the summarizing process, we theoretically analyze the flaw of the existing character contrastive loss and further propose Character Unidirectional Alignment (CUA) Loss to align character features from different views, thus achieving a unified feature representation of multiple-morphic characters. Details will be introduced in the following sections.

3.1 Online Generation Strategy

Adopting the semi-supervised framework is to leverage extra real unlabeled data to enrich the character morphologies during the viewing process, but no modification has been made to the existing supervised branch. To address the problem that the labeled synthetic training data is still simple and monotonous, we propose OGS to unify the training process of labeled and unlabeled data and raise the performance upper bound of the semi-supervised framework.

Firstly, as shown in Fig. 2(a), eight inconsistent representations brought about by the diverse character orientations and reading orders undoubtedly increase the difficulty of network convergence. Besides, scene text images with vertical reading orders often have small aspect ratios. When they are resized for input into the STR model, aspect ratios change significantly, resulting in them containing little practical information. Therefore, an image with height/width > r will be rotated, otherwise it remains unchanged, which is referred to as Unified Representation Forms (URF). As shown in Fig. 2(b), unified four representations can be further simplified into two primary forms with left-right reading order. Through URF, we drastically reduce eight representation forms to two primary ones to lessen the learning difficulty and lay the foundation for the following generation process.

Secondly, to enrich the character morphology in the supervised branch, according to the labels of simple synthetic training data, we propose to generate text samples in two primary forms in Fig. 2(b) with the red border. The generated samples are background-free but with random character styles to encourage the model to concentrate on character morphology. Following the characteristics of real samples, all character orientations in one generated sample are the same. OGS samples serve as base images to guide the alignment of the character features in the student model. Unlike previous

𝐿𝑐𝑢𝑎

Decoder

MLP

Encoder

EMA

Classifier

Classifier

Projector

MLP

Strong

Augmented

Image

Labeled

Unlabeled

Decoder

Encoder

Labeled and unlabeled data flow

Teacher model

"avenue"

𝐿𝑐𝑐𝑟

Base

Image

Only labeled data flow

Student model

F a

F b

P a

P b

semi-supervised methods [54, 11, 47], our design is based on the following reasons. 1) The goal of a STR model is to obtain character visual features that are not affected by background noise. Because the teacher model is gradient-free and needs to instruct the training of the student model, OGS is expected to generate samples that are noise-free and adaptable to various character morphologies. 2) Due to the existence of word-level annotations and the fact that OGS samples only contain various character features, feeding them to the teacher model is more helpful for the model to learn valuable features for recognizing characters in artistic style or severe distortion. In summary, by exploring the introduction of real unlabeled data and the extension of simple synthetic data during the viewing process, the semi-supervised model can significantly improve the recognition of challenging texts without introducing any human annotation, and break through the performance upper limit.

3.2 Character Unidirectional Alignment Loss

After viewing a large number of various characters, the model is supposed to summarize the commonality of each character. As mentioned above, the base images input to the teacher model are more likely to have practical and noise-free character features with diverse glyphs. Because the gradient of the teacher model is stopped, the character features of strong augmented images can be effectively aligned with base images to enhance the robustness of the model to character morphologies and cluster the same characters. Therefore, the CUA Loss is proposed to achieve this goal. A character will calculate the similarity with all the characters in a minibatch. Characters within the same class are supposed to have similar features, while the others should exhibit distinct features. Specifically, the model extracts a sequence of features F a = { f a 1 , f a 2 , · · · , f a T } , F b = { f b 1 , f b 2 , · · · , f b T } for augmented and base images, respectively, where T is the maximum recognition length in the decoder. At position i , f a i and f b i are the character features for augmented and base images. y b i is the ground truth for labeled data and the prediction from the teacher model for unlabeled data. We define the set for all position I ≡ { 1 , 2 , · · · , B × T } , where B is the batch size. The positive set for character at position i is P ( i ) ≡ { p ∈ I : y b p = y b i , S b i > η cua } , where η cua is the threshold for confidence score S b i . For labeled data, S b i is 1. For unlabeled data, S b i is the confidence score for the whole predicted word S b . The positive set excluding character at position i is P ′ ( i ) ≡ P ( i ) -{ p ∈ I : p = i } , and the negative set can be represented as N ( i ) ≡ I -P ( i ) . Our CUA Loss can be formulated as

where

where | P ( i ) | means the number of positive characters with f a i and τ is the temperature factor.

It is worth noting that our CUA Loss differs from Character Contrastive (CC) Loss [45] in both motivation and operation. The formulation of CC Loss can be expressed as

where all the symbols have the same meaning as Eq. 1 and Eq. 2. First, the motivation of CC Loss is to simply cluster the same character in a minibatch, and all images are treated equally. While our CUA Loss works on two types of images: strong augmented images and base images. As stated before, due to the different roles played by the two types of images in our framework and the gradient-free characteristic of the teacher model, all positive characters in base images are targeted for alignment by strong augmented images. Second, CC Loss suffers from a theoretical formula error and incorrectly sparses the distribution of characters belonging to the same category. As shown in Eq. 3, when f i calculates the similarity with f p , other characters belonging to the same class appearing in the denominator are considered as negative samples by mistake. To be specific, the gradient of the two loss functions can be expressed as

The detailed proof is included in Appendix A. As shown in Eq. 5 and Eq. 7, it is clear that the coefficient M 1 , M 2 > 0 and M 3 < 0 , which means the gradient for f a i is towards the positive character features f b p and f b p ′ in base images. f b n ′ is considered a negative sample and needs to be distinguished. As shown in Eq. 6 and Eq. 7, conditions are the same for N 1 > 0 and N 3 < 0 . However, N 2 < 0 indicates that the positive sample f p ′ is mistakenly regarded as a negative sample like f n ′ , which is harmful to the learning of clustering characters in the same class.

3.3 Training objective

Given base images x b , the teacher model obtains a sequence of character features F b and prediction probabilities P b = { p b 1 , p b 2 , · · · , p b T } . Correspondingly, the student model generates F a and P a = { p a 1 , p a 2 , · · · , p a T } for strong augmented images x a . Following [54], P a is obtained with an additional projector employed in the student model to ensure a stable training process. We also adopt Characterlevel Consistency Regularization (CCR) to boost the training of our semi-supervised framework. This process can be formulated as

where I ( · ) is the indicator function. T b is the text length of x b . η ccr is the threshold for confidence score S b , which is 1 for labeled data and the cumulative product of the maximum probability sequence

p b t for unlabeled data. To extract the commonalities of each character and cluster the character features belonging to the same class, we align the character features F b and F a using CUA Loss, which is described in Sec. 3.2. To equip the model with basic recognition ability, both x b and x a for labeled data are input into the student model to obtain the predicted sequence ˆ y b and ˆ y a . We use character-level cross-entropy loss as the recognition loss, which can be expressed as follows,

where ˆ y t means the prediction at position t for ˆ y b and ˆ y a . y gt t is the corresponding label. The overall loss function is

where λ is a hyper-parameter to balance the numeric values and is set to 0.1.

4 Experiment

4.1 Datasets

Without any laborious annotation process, our method only needs Synthetic Labeled (SL) data and Real Unlabeled (RU) data. SL includes two widely used synthetic datasets MJSynth [14] and SynthText [12], which contain 9M and 7M synthetic images. For real data without annotations, we adopt Union14M-U [15] with a total of 10M refined images from Book32 [13], CC [32], and OpenImages [18]. We evaluate STR models on six common benchmarks, including IIIT [24], SVT[40], IC13 [17], IC15 [16], SVTP [28], and CUTE [30]. Following the previous methods [54, 43], we use IC13-857 and IC15-1811 without images containing non-alphanumeric characters. To further evaluate the performance on difficult text images, we introduce several challenging benchmarks, including Union14M-Benchmark (Union-B) [15], WordArt [45], ArT [6], COCO-Text (COCO) [39], and Uber-Text (Uber) [53]. Union-B comprises Curve, Multi-Oriented, Artistic, Contextless, Salient, Multi-Words, and General with 2,426, 1,369, 900, 779, 1,585, 829, and 400,000 images, respectively.

4.2 Evaluation Metrics

The word accuracy is used to evaluate scene text recognition models, which means that all predicted characters in a word must be identical to the label to be considered correct. We further calculate the

weighted average score (WAVG) for six common benchmarks. Because the number of General in Union-B is much larger than the others, we report their average score (A VG).

4.3 Comparison with SOTA

In Table. 1, we compare our method with previous state-of-the-art methods on common benchmarks and Union-B. Table. 2 displays the performance on several challenging datasets. Our method achieves higher accuracy with smaller parameters on all benchmarks than fully-supervised methods. Compared with the semi-supervised methods, because the samples in common benchmarks are relatively easy, our method has an accuracy of 0.2% higher than TRBA-cr [54]. When evaluated on challenging benchmarks, our method shows significant performance superiority, namely 11.7%, 0.3%, 5.5%, 4.6%, and 27.9% accuracy increase on Union-B, WordArt, ArT, COCO, and Uber, respectively. Because the released checkpoint of TRBA-cr [54] uses the different real unlabeled datasets that are not publicly available, for a fair comparison, we reproduce the official code and train with the same real unlabeled datasets as ours. Benefiting from Unified Representation Forms (URF), our method surpasses them by a large margin on the benchmarks containing enormous multi-directional texts (85.8% vs 23.1% on Multi-Oriented and 67.2% vs 40.0% on Uber). For other benchmarks with challenging texts, benefiting from OGS and CUA Loss, our method shows significant improvement in Artistic and WordArt.

We further apply our proposed framework to other models. As shown in the last six lines in Table. 1 and Table. 2, to make the model robust to multi-directional text, the methods without ViSu and Baseline also adopt URF. Therefore, they perform way better than other supervised and semisupervised methods on Multi-Oriented and Uber. Equipped with ViSu, all the listed models gain at least 1.6% on common benchmarks, 3.5% on Union-B, 2.3% on WordArt, 2.9% on ArT, 0.9% on COCO, and 3.5% on Uber, proving the effectiveness of our proposed framework.

4.4 Ablation Study

4.4.1 Online Generation Strategy

We generate 1.6M samples using OGS to replace MJSynth [14] and SynthText [12] datasets. Because MJSynth and SynthText contain approximately 16M images in total, we utilize only 10% of them for fair comparison. As shown in Table. 3, ViSu trained with 10% synthetic data achieves an average accuracy of 58.2%. Upon incorporating OGS and employing the corresponding losses, we observe an improvement of 2.1%, underscoring the effectiveness of our proposed OGS and CUA

Loss. However, the model trained solely on OGS samples struggles to recognize texts. We attribute the poor performance to the diverse styles of characters, domain gap with Union-B, and varied orientations of text images generated by OGS. Models trained from scratch with such challenging data struggle to extract practical character information and are susceptible to background noise, hindering the acquisition of basic text recognition ability. By incorporating simple synthetic data to establish a foundation for recognition abilities and boosting it with our proposed OGS and CUA Loss targeted for challenging texts, the model is able to achieve promising performance. Furthermore, we add OGS samples to the training data of other methods to ensure the consistency of the dataset. The experimental results strongly demonstrate that the superiority does not come from the use of additional data but from the promotion of the semi-supervised learning framework by OGS.

4.4.2 Discussion about loss functions

Weevaluate the performance of the model when using cross entropy and KL-divergence as consistency loss, respectively. As shown in Table. 4, they have similar performance. For the alignment loss, both CC Loss and CUA Loss can bring performance improvement. However, as stated in Sec. 3.2, the original CC Loss only clusters the character features in the minibatch, referred to as CC 1 . Adopting our framework and aligning the character features in strong augmented images to the base images (CC 2 ) achieve 1.0% higher performance. Moreover, because some positive samples are mistakenly treated as negatives, adopting our CUA Loss brings an accuracy gain of 0.8%, demonstrating the effectiveness of our CUA Loss.

4.4.3 Ablation on training setting

As shown in Table. 4, because Curve and Multi-Oriented have many samples with multiple reading orders and character orientations, the model without URF has significantly lower accuracy on these benchmarks. When the model is not trained with real unlabeled data, it only reaches 64.5% accuracy. Compared to the baseline in Table. 1, the proposed OGS, along with the consistency and alignment losses, brings a 1.9% accuracy improvement, which proves that our OGS and CUA Loss improve the

recognition performance of the model on challenging text images when trained using only simple synthetic data. The experiments in the last two lines in Table. 4 also prove the superiority of OGS.

5 Conclusion

In this paper, we attribute the bottleneck of STR models to the insufficient exploration of character morphologies. Therefore, considering the human cognitive process, we develop a semi-supervised framework by viewing and summarizing. In the viewing process, we first adopt the Mean-Teacher framework to introduce real unlabeled data. Secondly, an Online Generation Strategy is proposed to explore the potential of synthetic data and raise the performance upper bound of the semi-supervised framework. In the summarizing process, a novel Character Unidirectional Alignment Loss is proposed to theoretically correct the previous formula error of mistakenly treating some positive samples with negative ones. Extensive experiments show that our method achieves state-of-the-art performance on all benchmarks, with particularly significant superiority on the challenging Union-B.

Acknowledgments

This work is supported by the National Key Research and Development Program of China (2022YFB3104700), the National Nature Science Foundation of China (U23B2028, 62121002, 62102384). We acknowledge the support of GPU cluster built by MCC Lab of Information Science and Technology Institution, USTC. We also thank the USTC supercomputing center for providing computational resources for this project.

References

Jeonghun Baek, Yusuke Matsui, and Kiyoharu Aizawa. What if we only use real datasets for scene text recognition? toward scene text recognition with fewer labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3113-3122, 2021.

Darwin Bautista and Rowel Atienza. Scene text recognition with permuted autoregressive sequence models. In European Conference on Computer Vision , pages 178-196. Springer, 2022.

Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training. In Proceedings of the eleventh annual conference on Computational learning theory , pages 92-100, 1998.

Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi-supervised learners. Advances in neural information processing systems , 33:22243-22255, 2020.

Changxu Cheng, Peng Wang, Cheng Da, Qi Zheng, and Cong Yao. Lister: Neighbor decoding for length-insensitive scene text recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 19541-19551, 2023.

Chee Kheng Chng, Yuliang Liu, Yipeng Sun, Chun Chet Ng, Canjie Luo, Zihan Ni, ChuanMing Fang, Shuaitao Zhang, Junyu Han, Errui Ding, et al. Icdar2019 robust reading challenge on arbitrary-shaped text-rrc-art. In 2019 International Conference on Document Analysis and Recognition (ICDAR) , pages 1571-1576. IEEE, 2019.

Emily Denton, Sam Gross, and Rob Fergus. Semi-supervised learning with context-conditional generative adversarial networks. arXiv preprint arXiv:1611.06430 , 2016.

WDong-DongChen and ZH WeiGao. Tri-net for semi-supervised deep learning. In Proceedings of twenty-seventh international joint conference on artificial intelligence , pages 2014-2020, 2018.

Yongkun Du, Zhineng Chen, Caiyan Jia, Xiaoting Yin, Tianlun Zheng, Chenxia Li, Yuning Du, and Yu-Gang Jiang. Svtr: Scene text recognition with a single visual model. arXiv preprint arXiv:2205.00159 , 2022.

Shancheng Fang, Hongtao Xie, Yuxin Wang, Zhendong Mao, and Yongdong Zhang. Read like humans: Autonomous, bidirectional and iterative language modeling for scene text recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7098-7107, 2021.

Yunze Gao, Yingying Chen, Jinqiao Wang, and Hanqing Lu. Semi-supervised scene text recognition. IEEE Transactions on Image Processing , 30:3005-3016, 2021.

Ankush Gupta, Andrea Vedaldi, and Andrew Zisserman. Synthetic data for text localisation in natural images. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2315-2324, 2016.

Brian Kenji Iwana, Syed Tahseen Raza Rizvi, Sheraz Ahmed, Andreas Dengel, and Seiichi Uchida. Judging a book by its cover. arXiv preprint arXiv:1610.09204 , 2016.

Max Jaderberg, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Synthetic data and artificial neural networks for natural scene text recognition. arXiv preprint arXiv:1406.2227 , 2014.

Qing Jiang, Jiapeng Wang, Dezhi Peng, Chongyu Liu, and Lianwen Jin. Revisiting scene text recognition: A data perspective. In Proceedings of the IEEE/CVF international conference on computer vision , pages 20543-20554, 2023.

Dimosthenis Karatzas, Lluis Gomez-Bigorda, Anguelos Nicolaou, Suman Ghosh, Andrew Bagdanov, Masakazu Iwamura, Jiri Matas, Lukas Neumann, Vijay Ramaseshan Chandrasekhar, Shijian Lu, et al. Icdar 2015 competition on robust reading. In 2015 13th international conference on document analysis and recognition (ICDAR) , pages 1156-1160. IEEE, 2015.

Dimosthenis Karatzas, Faisal Shafait, Seiichi Uchida, Masakazu Iwamura, Lluis Gomez i Bigorda, Sergi Robles Mestre, Joan Mas, David Fernandez Mota, Jon Almazan Almazan, and Lluis Pere De Las Heras. Icdar 2013 robust reading competition. In 2013 12th international conference on document analysis and recognition , pages 1484-1493. IEEE, 2013.

Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. International Journal of Computer Vision , 128(7):1956-1981, 2020.

Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. arXiv preprint arXiv:1610.02242 , 2016.

Anh Duc Le. Recognizing handwritten mathematical expressions via paired dual loss attention network and printed mathematical expressions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops , pages 566-567, 2020.

Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML , volume 3, page 896. Atlanta, 2013.

Wei Liu, Chaofeng Chen, and Kwan-Yee Wong. Char-net: A character-aware neural network for distorted scene text recognition. In Proceedings of the AAAI conference on artificial intelligence , volume 32, 2018.

Canjie Luo, Lianwen Jin, and Zenghui Sun. Moran: A multi-object rectified attention network for scene text recognition. Pattern Recognition , 90:109-118, 2019.

Anand Mishra, Karteek Alahari, and CV Jawahar. Scene text recognition using higher order language priors. In BMVC-British machine vision conference . BMVA, 2012.

Byeonghu Na, Yoonsik Kim, and Sungrae Park. Multi-modal text recognition networks: Interactive enhancements between visual and semantic features. In European Conference on Computer Vision , pages 446-463. Springer, 2022.

Augustus Odena. Semi-supervised learning with generative adversarial networks. arXiv preprint arXiv:1606.01583 , 2016.

Gaurav Patel, Jan P Allebach, and Qiang Qiu. Seq-ups: Sequential uncertainty-aware pseudolabel selection for semi-supervised text recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 6180-6190, 2023.

Trung Quy Phan, Palaiahnakote Shivakumara, Shangxuan Tian, and Chew Lim Tan. Recognizing text with perspective distortion in natural scenes. In Proceedings of the IEEE International Conference on Computer Vision , pages 569-576, 2013.

Antti Rasmus, Mathias Berglund, Mikko Honkala, Harri Valpola, and Tapani Raiko. Semisupervised learning with ladder networks. Advances in neural information processing systems , 28, 2015.

Anhar Risnumawan, Palaiahankote Shivakumara, Chee Seng Chan, and Chew Lim Tan. A robust arbitrary text detection system for natural scene images. Expert Systems with Applications , 41(18):8027-8048, 2014.

Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transformations and perturbations for deep semi-supervised learning. Advances in neural information processing systems , 29, 2016.

Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 2556-2565, 2018.

Baoguang Shi, Xiang Bai, and Cong Yao. An end-to-end trainable neural network for imagebased sequence recognition and its application to scene text recognition. IEEE transactions on pattern analysis and machine intelligence , 39(11):2298-2304, 2016.

Baoguang Shi, Mingkun Yang, Xinggang Wang, Pengyuan Lyu, Cong Yao, and Xiang Bai. Aster: An attentional scene text recognizer with flexible rectification. IEEE transactions on pattern analysis and machine intelligence , 41(9):2035-2048, 2018.

Leslie N Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates. In Artificial intelligence and machine learning for multi-domain operations applications , volume 11006, pages 369-386. SPIE, 2019.

Jost Tobias Springenberg. Unsupervised and semi-supervised learning with categorical generative adversarial networks. arXiv preprint arXiv:1511.06390 , 2015.

Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in neural information processing systems , 30, 2017.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008.

Andreas Veit, Tomas Matera, Lukas Neumann, Jiri Matas, and Serge Belongie. Coco-text: Dataset and benchmark for text detection and recognition in natural images. arXiv preprint arXiv:1601.07140 , 2016.

Kai Wang, Boris Babenko, and Serge Belongie. End-to-end scene text recognition. In 2011 International conference on computer vision , pages 1457-1464. IEEE, 2011.

Peng Wang, Cheng Da, and Cong Yao. Multi-granularity prediction for scene text recognition. In European Conference on Computer Vision , pages 339-355. Springer, 2022.

Yuxin Wang, Hongtao Xie, Shancheng Fang, Jing Wang, Shenggao Zhu, and Yongdong Zhang. From two to one: A new scene text recognizer with visual language modeling network. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1419414203, 2021.

Zixiao Wang, Hongtao Xie, Yuxin Wang, Jianjun Xu, Boqiang Zhang, and Yongdong Zhang. Symmetrical linguistic feature distillation with clip for scene text recognition. In Proceedings of the 31st ACM International Conference on Multimedia , pages 509-518, 2023.

Jin-Wen Wu, Fei Yin, Yan-Ming Zhang, Xu-Yao Zhang, and Cheng-Lin Liu. Handwritten mathematical expression recognition via paired adversarial learning. International Journal of Computer Vision , 128:2386-2401, 2020.

Xudong Xie, Ling Fu, Zhifei Zhang, Zhaowen Wang, and Xiang Bai. Toward understanding wordart: Corner-guided transformer for scene text recognition. In European Conference on Computer Vision , pages 303-321. Springer, 2022.

Mingkun Yang, Yushuo Guan, Minghui Liao, Xin He, Kaigui Bian, Song Bai, Cong Yao, and Xiang Bai. Symmetry-constrained rectification network for scene text recognition. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9147-9156, 2019.

Mingkun Yang, Biao Yang, Minghui Liao, Yingying Zhu, and Xiang Bai. Sequential visual and semantic consistency for semi-supervised text recognition. Pattern Recognition Letters , 178:174-180, 2024.

Xiangli Yang, Zixing Song, Irwin King, and Zenglin Xu. A survey on deep semi-supervised learning. IEEE Transactions on Knowledge and Data Engineering , 2022.

Deli Yu, Xuan Li, Chengquan Zhang, Tao Liu, Junyu Han, Jingtuo Liu, and Errui Ding. Towards accurate scene text recognition with semantic reasoning networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 12113-12122, 2020.

Xiaoyu Yue, Zhanghui Kuang, Chenhao Lin, Hongbin Sun, and Wayne Zhang. Robustscanner: Dynamically enhancing positional clues for robust text recognition. In European Conference on Computer Vision , pages 135-151. Springer, 2020.

Fangneng Zhan and Shijian Lu. Esir: End-to-end scene text recognition via iterative image rectification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2059-2068, 2019.

Boqiang Zhang, Hongtao Xie, Yuxin Wang, Jianjun Xu, and Yongdong Zhang. Linguistic more: Taking a further step toward efficient and accurate scene text recognition. arXiv preprint arXiv:2305.05140 , 2023.

Ying Zhang, Lionel Gueguen, Ilya Zharkov, Peter Zhang, Keith Seifert, and Ben Kadlec. Ubertext: A large-scale dataset for optical character recognition from street-level imagery. In SUNw: Scene Understanding Workshop-CVPR , volume 2017, page 5, 2017.

Caiyuan Zheng, Hui Li, Seon-Min Rhee, Seungju Han, Jae-Joon Han, and Peng Wang. Pushing the performance limit of scene text recognizer without human annotation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14116-14125, 2022.

Tianlun Zheng, Zhineng Chen, Jinfeng Bai, Hongtao Xie, and Yu-Gang Jiang. Tps++: Attentionenhanced thin-plate spline for scene text recognition. arXiv preprint arXiv:2305.05322 , 2023.

A Grandient of Loss Functions

For Character Unidirectional Alignment (CUA) Loss, we have

The derivation of the gradient for CUA loss:

where

And the Character Contrastive (CC) Loss [45] can be expressed as

The derivation of the gradient for CC loss:

where

B Experiment

B.1 Implementation Details

In the model configuration, our model consists of a transformer-based encoder with 12 transformer blocks and 6 attention heads and a transformer-based decoder with 1 transformer block and 12 attention heads. All images are resized to 100 × 32 , and the patch size is 8 × 4 . The maximum length T is set to 25. The character set size is 36, including 10 digits and 26 alphabets.

For training settings, the network is trained in an end-to-end manner without pre-training. We adopt AdamW optimizer and one-cycle [35] learning rate scheduler with a maximum learning rate of 6e-4. The batchsize is 384 for both synthetic data and real unlabeled data. We set the EMA smoothing factor α = 0 . 999 , aspect ratio thresh r = 1 . 3 , confidence threshold η ccr = 0 . 5 , η cua = 0 . 7 , and temperature factor τ = 0 . 1 . Due to the Online Generation Strategy (OGS) and Unified Representation Forms (URF), all images are rotated 180 degrees with a probability of 0.5 during the training stage. Following [2], the student model takes strong augmented images for labeled and unlabeled data. The unlabeled real images fed into the teacher model are weakly augmented, where only color jitter is used, including brightness, contrast, saturation, and hue. ViSu is trained on 4 NVIDIA RTX 4090 GPUs.

B.2 Training with Real Labeled Data

Our method aims to boost the scene text recognition (STR) model without any human costs. However, to demonstrate the effectiveness of our method when training with real labeled data, we replace the synthetic labeled data with Union14M-L [15], which consists of 3.2M refined images from 14 publicly available datasets. For the convenience of comparison and following [15], we employ IC13-1015 and IC15-2077 to evaluate all the STR methods.

As shown in Table. 5 and Table. 6, our method achieves higher accuracy than all fully supervised methods. Compared to the state-of-the-art pretraining method MAERec [15], with approximately 1/5

of the parameters, our method outperforms it by 0.4% on common benchmarks and by 1.7% on UnionB [15]. Furthermore, for a fair comparison, we reproduce TRBA-cr [54] using the same datasets as ours. The experimental results show that ViSu performs better on all benchmarks, especially on challenging benchmarks (86.9% vs 80.5% on Union-B [15], 85.9% vs 82.7% on WordArt [45], 84.9% vs 81.9% on ArT [6], 80.4% vs 77.6% on COCO [39], and 91.6% vs 77.7% on Uber [53]). We attribute this superior performance to the increased diversity in character morphology from the Online Generation Strategy and the clustering of similar character features facilitated by the CUA Loss.

B.3 Efficiency

Table. 7 presents the efficiency metrics. Speed is calculated by averaging the inference time over 4,000 images. All experiments are conducted on an NVIDIA RTX 4090 GPU with a batch size of 1. The reported methods are trained using real data. In comparison to MAERec-B [15], our method achieves higher accuracy (86.9% vs 85.2%) with a significantly lighter model, i.e. , approximately 1 / 5 parameter size and 1 / 17 inference time. Compared to models of similar parameter sizes and inference times, ViSu outperforms them by at least 4.6%, demonstrating the effectiveness of our approach.

B.4 Discuss about configuration settings for OGS

Table. 8 shows the performance with different configuration of OGS. The first 2 to 5 rows indicate that applying independent random font and unified random character orientation to each character in

a sample leads to better performance. This is because random fonts have more diversity in character glyphs, making the model robust to character morphology. All characters in real scene text images tend to have the same orientation, thus adopting a uniform orientation in one sample is a better solution. Moreover, adding a random background and a random color to the text both bring a slight performance degradation. Because OGS samples act as base images in the teacher model to guide the character feature alignment in the student model, they are expected to have noise-free features to enhance the robustness of the model to character morphology. Therefore, introducing extra backgrounds and colors is redundant. In general, changing the configuration settings of OGS brings some slight performance fluctuations, but all are better than not using OGS, which proves the effectiveness of our approach.

B.5 The influence of hyper-parameters in CUA Loss

We conduct several experiments to demonstrate the influence of λ , η cua , and τ . λ is the weight of CUA Loss. When λ is set to 0, indicating that the model is trained without CUA Loss, the accuracy can only reach 68.2%. A small λ implies minimal clustering of characters into the same class and makes the model focus more on the basic recognition loss. The confidence threshold η cua determines the characters involved in the CUA Loss. A low threshold introduces many uncertain characters, where incorrect predictions can harm the alignment. Conversely, a high threshold reduces the number of characters participating in the alignment and clustering, similar to a small λ . The temperature factor τ adjusts the smoothness of character features. A small τ focuses more on challenging characters but sharpens the feature distribution. As shown in Table. 9, changes in these parameters do not significantly affect model performance. To achieve relatively better performance, we set λ = 0 . 1 , η cua = 0 . 7 , and τ = 0 . 1 as our default configuration.

B.6 Qualitative Analysis

Fig. 4(a) displays several challenging examples. The baseline can correctly recognize simple images with multiple directions, benefiting from URF. However, it fails to recognize complex characters due to a lack of adaptation to character morphology. The supervised method ParSeq and semi-supervised method TRBA-cr have difficulty recognizing these challenging text images. Fig. 4(b) and (c) show the distribution of character features from t-SNE [38]. Compared with CC Loss, CUA Loss better clusters characters in the same class and makes character features in different classes more distinguishable.

B.7 Limitations

Our method has difficulty recognizing scene text images with extreme aspect ratios. As shown in Fig. 5, when the text in a scene image is very long, it usually has a large aspect ratio. However, all images are resized to a fixed size before being fed into the text recognizer, which results in drastic changes in the aspect ratio of long text images. The information loss caused by resizing the images impairs the extraction of character features. This is a common problem in scene text recognition task and can be alleviated by splitting long texts into multiple segments for recognition and then concatenating them together.

panda

-and-

pares

ponds

argentina

argentina

apr

the

mecca

mecca

2229

date

genius

genius

sunctor

sender

christm

-h-isty

-h-istm

-h-istm

cola

gola

gold

gola

lakers

jakers

jakers

takers

ta

e

ex

ey

dots

dots

5200

2100

(a)

(b)

(c)

tfboys

tifboys

tifboys

teboys

NI

GENIUS

TFBOYS

ChRistM

LAKERS

d

a

e

e

r

guangdongnanyuebank

guangdongnanyuzbank

qingwuluanrenglaji

o_ngwulve_renglas_

tiantianartspace

tiantianartspact

mansengsauvignon

mansengsauvignom

semiskimmed

semiskimme_

semi skimme

GUANGDONG NANYUZ BANK

MANSENG-SAUVIGNON

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations are discussed in Appendix B.7.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The full set of assumptions and a complete proof for the theoretical result is shown in Appendix A.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The information needed to reproduce the main experimental results is shown in Sec. 4.1, 4.2 and Appendix B.1.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We will release the code after the paper is accepted, which is also claimed in Abstract.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The paper specify all the training and test details, which can be found in Sec. 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The paper does not include error bars.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The paper provide sufficient information on the computer resources in Appendix B.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The paper does not involve societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The paper cite all the original papers.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The code and its corresponding documentation will be released together after the paper is accepted.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper proposes a ViSu framework to enhance text recognition by viewing and summarizing. For the viewing process, the authors generate various text images to lead the model to focus on different text styles. For the summarizing process, they analyze the drawback in existing character consistency loss and propose a CUA loss to cluster the character under different styles. As a result, the proposed framework outperforms other existing methods on multiple datasets.

STRENGTHS:
1. The design of OGS can obtain promising improvements without additional human cost.
2. The authors point out that previous consistency loss mistakenly regards some positive samples as negative samples, which is an important theoretical drawback in existing metric learning methods. And the proposed CUA loss provides a solution for this issue.
3. The proposed method gets SOTA performance on both common and challenging datasets.
4. The proposed framework can be applied on multiple text recognition models seamlessly.

WEAKNESSES:
1. To further verify the effectiveness of OGS. Authors should compare the performance of replacing OGS with normal synthetic data. 
2. Tab.5 lacks the comparison with the baseline model. 
3. In Tab.5, it seems that MAERec-B performs better on the Union-14m dataset.

QUESTIONS:
1. As the semi-supervised framework can use large-scale unlabeled real data for training, why does ViSu require OGS to generate synthetic data?
2. To verify the fairness, do other semi-supervised methods in Tab. 1 and 5 have the same implementation details as ViSu?

LIMITATIONS:
Please refer to Weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-06-30

comments: SUMMARY:
This paper focuses on the character morphologies and proposes to boost the scene text recognizer through viewing and summarizing paradigm. In the viewing process, the Mean Teacher framework is used to train with unlabeled data. In the summarizing process, the proposed method theoretically proves the mistakes in the previous method and proposes a new loss function.

I have read the author responses and the comments of other reviewers, I would recommend the 'accept' score.

STRENGTHS:
1. The proposed method achieves good results, especially on challenging benchmarks.
2. The paper is well written. It clearly points out the problem in the previous method and provides a detailed explanation. The proposed OGS and Mean Teacher framework effectively increase the diversity of character glyphs. The CUA loss corrects the previous error and better achieves clustering of identical characters.
3. There are lots of experiments to prove the effectiveness of the proposed method.

WEAKNESSES:
1. There is a lack of clear description in the paper on how CRNN-ViSu and TRBA-ViSu in Table 1 are set up and trained.
2. Although the proposed method achieves high average accuracy on common benchmark and Union14B-Benchmark, it is not SOTA on certain benchmarks, such as SVT, SVTP, CUTE, Curve, Contextless, MultiWords.
3. When the proposed method recognizes text with vertical reading order, how to decide whether to rotate it 90 degrees clockwise or counterclockwise.

QUESTIONS:
See Weaknesses.

LIMITATIONS:
The authors adequately addressed the limitations and potential negative societal impact
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nExisting scene text recognition (STR) methods struggle to recognize challenging texts, which originates from the insufficient exploration of character morphologies. To address the issues, the paper proposes to facilitate the contrastive learning-based STR framework in a self-motivated manner by leveraging synthetic and real unlabeled data without any human cost. An Online Generation Strategy is proposed to enrich the diversity of characters in training data. Besides, a new Character Unidirectional Alignment Loss is proposed for aligning the character features in the student model with the reference features in the teacher model. Extensive experiment results show the effectiveness of the proposed method.\n\nSTRENGTHS:\n1) The paper proposes to improve the recognition for challenging texts, especially for artistic and severely distorted characters within a mean-teacher framework without the requirement for real labeled data.\n2) An Online Generation Strategy is proposed to enrich the diversity of characters in training data without need of psuedo labeling real data explicitly. \n3) A Character Unidirectional Alignment Loss is proposed to improve the existing Character Contrastive (CC) Los [45] in character representation learning. \n4) Extensive experiments on Common Benchmarks, Union14M-Benchmark, and other challenging benchmarks demonstrate the effectiveness of the proposed method.\n\nWEAKNESSES:\n1) The overall framework follows a mean teacher framework, which is not new for semi-supervised scene text recognition.\n2) In Table 1, the baseline model without RU data has achieved a high performance than existing SOTA. Is there any special design in the baseline model?\n3) Why not simply remove the second item in equation, i.e., exclude the positives in the denominator?\n4) Absense of comparison between the proposed method and other methods in Sec. 2.3.\n\nQUESTIONS:\nPlease refer to the weakness part.\n\nLIMITATIONS:\nYes. The limiation is well descirbed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper addresses the problem of insufficient exploration of character morphology in scene text recognition and proposes a new framework comprising an Online Generation Strategy (OGS) and Character Unidirectional Alignment (CUA) Loss to enable the model to learn from unlabeled real data. OGS mitigates the issue of data scarcity, while CUA aids in clustering characters. Comprehensive experiments demonstrate the effectiveness of the proposed method.

STRENGTHS:
1. The proposed method effectively addresses the issue of excessive representation methods for scene images through Unified Representation Forms.
2. The errors in the previous Character Contrastive Loss method are corrected.
3. The idea of utilizing character morphology is novel.

WEAKNESSES:
1. The writing quality needs improvement.
2. There is a lack of detailed explanation regarding the Online Generation Strategy (OGS) and Unified Representation Forms (URF).

QUESTIONS:
The proposed method struggles with handling images with extreme aspect ratios. The authors attempt to address this by splitting the image, recognizing each part, and then concatenating them. However, this splitting operation may interfere with character recognition and degrade the overall recognition quality.

LIMITATIONS:
N.A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-18