Hierarchical Multi-label Learning for Incremental Multilingual Text Recognition

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 16:21:01

Abstract: Multilingual text recognition (MLTR) is increasingly essential for facilitating cultural communication. However, existing methods often struggle with retaining previous language knowledge when learning new languages. A straightforward solution is performing incremental learning (IL) on MLTR tasks. However, it ignores the shared words and characters across incremental languages, which we first term as an incremental sharing problem. Motivated by this observation, we propose a HierArchical Multi-label learning framework for Multilingual tExt Recognition, termed HAMMER. An online knowledge analysis is designed to identify shared knowledge and provide corresponding multi-label language supervision. Specifically, only words and characters appearing simultaneously in multiple languages are considered shared knowledge. Additionally, to further capture language dependencies, we introduce a hierarchical language evaluation mechanism to predict language scores at word and character levels. These scores, supervised by the knowledge analysis, guide the specific recognizers to effectively utilize both old and new language knowledge, thereby mitigating catastrophic forgetting caused by imbalanced rehearsal sets. Extensive experiments conducted on benchmark datasets, MLT17 and MLT19, show that HAMMER exhibits remarkable results and outperforms other state-of-the-art approaches.

Main paper: 
 introduction: 
 Scene text recognition (STR) [10, 34, 46, 51] involves reading character sequences from scene text images and is a challenging task in computer vision. Recent advancements in deep learning have facilitated the progress of text recognition technology, making it widely applicable in various real-world scenarios such as autonomous driving, machine translation, and image-text retrieval. However, most existing methods primarily focus on Latin script [4, 43, 45] and struggle to handle multilingual scenarios effectively. As crosscultural communications become more prevalent, there is a growing need for multilingual text recognition (MLTR) [13, 14, 42] . A robust MLTR model should be capable of recognizing multiple languages simultaneously and continuously learning new languages. Existing MLTR methods often train a multilingual recognizer by mixing samples from all languages together [5, 24] . When encountering a new language, these methods incorporate the new samples into the original old data to retrain the recognizer, leading to increased computational complexity [27, 41] and potential storage constraints.Incremental learning (IL) [11, 18, 22, 49] is a powerful paradigm for continuously learning new data while retaining knowledge of old data. It is promising to yield fine recognizers when incremental methods are applied to MLTR tasks. However, like some incremental learning dilemmas, incremental multilingual recognizers also face the issue of catastrophic forgetting, where the recognizer tends to forget the old languages while learning new ones.In incremental methods, replay-based ones [6, 28, 37] have shown promising performance by obtaining a small amount of old data, referred to as the rehearsal set. Conventional replay-based methods typically sample old classes uniformly to create a fixed-capacity rehearsal set. Nevertheless, this operation is not directly applicable to MLTR tasks. In MLTR, languages are treated as incremental tasks, and the characters composing each word are considered incremental classes. Unlike other tasks where classes are sampling units (e.g., image classification), in MLTR tasks, the sampling unit is the word itself, not the character classes. Consequently, there is no guarantee that the rehearsal set covers all character classes of old languages, resulting in a rehearsal-imbalance problem [52] . This imbalanced rehearsal set would exacerbate catastrophic forgetting, significantly impacting recognition performance in old languages.Existing replay-based incremental MLTR methods [38, 52] maintain awareness of old knowledge by loading corresponding specific parameters after determining the belonging language class of the entire words. An inspired work, MRN [52] , introduces a multiplexed routing network that predicts language scores of entire words to weight the various specific recognizers for decoding character sequences. Although these methods alleviate catastrophic forgetting caused by the imbalanced rehearsal set to some extent, they still suffer from two limitations. (1) Fine-grained information at the character level is ignored. A word is composed of multiple characters, and in addition to the whole word, individual characters also contain valuable information. (2) Shared knowledge among incremental languages is neglected. In MLTR tasks, defining a sample as belonging strictly to a specific language is inaccurate since a sample may appear in multiple languages simultaneously. For instance, words composed solely of digits belong to both English and Chinese. Simply assuming these words belong exclusively to either English or Chinese would lead to knowledge misalignment. In this paper, we innovatively refer to this phenomenon as the incremental sharing problem. Different from the standard IL setting shown in Fig. 1 (a), there is an overlap in the MLTR IL setting depicted in Fig. 1(b ). Overlap indicates the intersection between the content of incremental tasks, i.e., incremental sharing, which is mainly manifested in two aspects: 1) Incremental character classes sharing, where a character class exists in multiple languages, and 2) Incremental word instances sharing, where a word instance is present in more than one language. Particularly, we define words as shared only when each character of the word appears in multiple identical languages simultaneously. To demonstrate the incremental sharing problem, we further conduct a statistical analysis in two popular multilingual text datasets, MLT17 [25] and MLT19 [24] . As indicated in Table 1 , 20%-55% of incremental character classes are shared, and 0.3%-26% of incremental word instances are shared. Therefore, exploring the common knowledge between these shared characters and words is crucial for incremental MLTR tasks.Motivated by the above observation, we propose a HierArchical Multi-label learning framework for incremental Multilingual tExt Recognition, named HAMMER. It comprises two stages: specific language learning and multilingual learning. Initially, the former stage involves training a specific text recognizer for each incremental language, similar to standard STR models. Subsequently, in the multilingual learning stage, an online knowledge analysis module is devised to determine whether samples belong to shared or new knowledge, providing corresponding multi-labeled and single-labeled knowledge supervision. Additionally, a hierarchical language estimation mechanism is proposed to predict language scores at both the word and character levels using a DomainMLP that processes character features extracted by the frozen specific recognizers. These scores, supervised by the knowledge analysis module, guide both old and new recognizers in decoding character sequences. By leveraging multi-label learning of shared knowledge at the word and character levels, our HAMMER framework can adequately exploit the old and new knowledge, thereby alleviating catastrophic forgetting stemming from imbalanced rehearsal sets and enhancing recognition performance.Our contributions are summarized as follows:• We first identify the incremental sharing problem specific to incremental MLTR tasks, which differs from conventional incremental learning settings.  
related work 2.1 deep learning-based str: 
 Mainstream deep learning-based STR methods typically employ three types of decoders: CTC [9, 12, 30] , RNN [3, 4, 33] , and transformer [8, 23, 26, 35, 40] decoders. TRBA [3] , for instance, is a representative STR baseline with TPS [31] transformation. ABINet [8] is a transformer-based baseline explicitly exploring language context. Recently, with the advances in large language models (LLM), CLIP-based methods [1, 36, 50] have shown promising results on various text-related tasks. Multilingual text recognition [13, 24, 25, 42] , a sub-field of STR, aims to recognize multiple languages simultaneously and continuously learn new languages. Early solutions [52] involve training models on all language data together, leading to biases towards samples-rich languages and poor performance for samples-scarcity languages. Alternatively, some methods [14, 38] introduce an auxiliary language prediction network to determine the language class, and then specific recognizers or private parameters are loaded to predict character sequences. While similar to our pipeline, these methods do not address the incremental sharing problem, an essential property of MLTR tasks.Incremental learning [21, 22, 32, 49] is crucial for continuously processing new data while retaining previous knowledge, thus overcoming catastrophic forgetting. Mainstream incremental learning methods fall into three categories. Regularization-based methods prevent the overwriting of old knowledge by imposing constraints on the loss of new tasks. For instance, LwF [18] ensures similar predictions between old and new models on new tasks During incremental multilingual learning for task i, we perform online knowledge analysis at word and character levels to determine whether samples are shared and provide corresponding multi-label supervision. Then, the hierarchical language estimation predicts the belonging language scores at word and character levels. These scores guide specific recognizers in predicting character sequences.through knowledge distillation. An improved algorithm, EWC [15] , introduces a generalized parameter constraint method based on the Bayesian framework, adding an additional parameter regular loss. Replay-based methods retain a representative portion of old data to review previously learned knowledge. iCaRL [27] , for example, employs distillation loss to update model parameters while allowing the use of old data. CLEAR [28] dynamically adjusts the amount of old data retained, avoiding the linear growth of computational cost seen in LwF. Expansion-based methods expand a set of task-specific parameters for each incremental task. PGN [29] creates a task-specific network and transfers knowledge among them through horizontal connections. However, as the number of tasks increases, memory usage grows linearly. To mitigate memory overhead, DEN [44] and RCL [39] only expand network width when capacity is insufficient. This paper employs the incremental learning paradigm to perform MLTR tasks. 
our method 3.1 problem definition: 
 Given incremental languages D = {D 𝑖 } 𝐼 𝑖=1 , we aim to improve performance on the current language D 𝑖 and maintain perception of old languages D𝑖-1 = {D 𝑘 } 𝑖-1 𝑘=1 , where D 𝑖 = {(𝑥 𝑖,𝑗 , 𝑦 𝑖,𝑗 )} 𝑁 𝑖 𝑗=1 is the dataset of task 𝑖, with 𝑁 𝑖 being the number of training samples. Each 𝑦 𝑖,𝑗 = {𝑦 𝑖,𝑗,1 , 𝑦 𝑖,𝑗,2 , • • • , 𝑦 𝑖,𝑗,𝑇 } ∈ D 𝑖 is a sequence label within the label set C 𝑖 , where 𝑇 denotes the pre-defined maximum decoding length. The incremental sharing problem identified innovatively by us differs from conventional incremental learning settings in two ways: (1) incremental character classes sharing, i.e., C 𝑖 ∩ C𝑖-1 ≠ ∅, where C𝑖-1 = ∪ 𝑖-1 𝑘=1 C 𝑘 denote the label space of all character classes up to task 𝑖-1, and (2) incremental word instances sharing, i.e., D 𝑖 ∩ D𝑖-1 ≠ ∅. Due to the presence of shared characters and words, the replay-based incremental paradigm is naturally chosen for its access to some of the old data, referred to as the rehearsal set M 𝑖 = 𝑠𝑎𝑚𝑝𝑙𝑒𝑟 ( D𝑖-1 ), thus facilitating the exploration of dependency relationships between the old and the new languages.From the perspective of language classes, these shared characters and words are multi-labeled due to their presence in multiple languages. Therefore, multi-label learning is utilized to optimize the belonging language classes of shared samples. Furthermore, each character composing a word contains valuable information besides the entire word. Consequently, hierarchical analysis at the word and character levels benefits recognition performance. Based on the motivations above, we propose a two-stage hierarchical multi-label learning framework, HAMMER. An overview of the HAMMER is illustrated in Fig. 2 .In stage 1, a specific recognizer is trained using the language dataset D 𝑖 corresponding to the task 𝑖. Specifically, we denote the feature extractor and classifier as R 𝑖 and 𝜑 𝑖 , respectively, and P 𝑖 as the predicted probability distribution. Given the (𝑥, 𝑦) ∈ D 𝑖 , we obtain P 𝑖 (𝑥) = 𝜑 𝑖 (R 𝑖 (𝑥)). EQUATIONwhere P (𝑦 𝑡 |𝑥) represents the predicted probability of the output being 𝑦 𝑡 at time step t.In MLTR tasks, shared characters and words appear in multiple incremental languages. As mentioned previously, we term this phenomenon in MLTR tasks as incremental sharing problem. Thoroughly exploring the relationship between shared characters and words is crucial for enhancing MLTR performance. Due to the lack of comprehensive belonging language supervision of the shared characters and words in the original datasets, we first design an online knowledge analysis to distinguish between new and shared knowledge. New knowledge refers to characters or words unique to the current task, whereas shared ones encompass those that also appear in other task(s). Precisely, as illustrated in Fig. 3 (a), we classify a word instance as shared only where each constituent character is also present simultaneously in other identical task(s). Formally, let (𝑥, 𝑦) ∈ D 𝑖 ∪ M 𝑖 denote an image-text pair to be recognized in incremental task i. Initially, the character memory is updated dynamically by adding current character classes C 𝑖 to the memory. Then, the character 𝑦 𝑡 in the sequence 𝑦 = {𝑦 1 , 𝑦 2 , • • • , 𝑦 𝑇 } can be encoded based on its associated language classes. If its encoding length is 1, the character is new, existing solely in the current task. Conversely, the character is deemed shared if the encoding length is greater than 1, indicating multiple languages. The knowledge type of character 𝑦 𝑡 is formulated as,EQUATIONwhere E (•) denotes the language encoding, and 𝑙𝑒𝑛(•) is a calculating length fucntion. A word encoding is obtained by intersecting the encodings of its constituent characters. Similarly, this word is new if its encoding length is 1 and shared if it exceeds 1. The knowledge type of word 𝑦 is formulated as,EQUATIONwhereE (𝑦) = J (E (𝑦 1 ), • • • , E (𝑦 𝑇 )). The J (•)is the intersection of all character encodings. As depicted in Fig. 3 (b), if any new character is present in a word, the word encoding after character intersection must be of length 1, signifying that the word is new.In essence, a character unique to the current language implies that the word is also new, which is consistent with our design.Based on the determined knowledge types above, this module plays a crucial role in providing corresponding knowledge supervision. For new knowledge, the supervision labels are one-hot encodings. Multi-label optimization aligns precisely with the purpose of shared knowledge learning. Thus, multi-label learning is employed for shared knowledge optimization. It is worth noting that multiple labels require transformation based on the principles of multi-label learning. The language class labels of character 𝑦 𝑖 and word 𝑦 are defined as,EQUATIONEQUATIONwhere N * and H * represent the new and shared knowledge label sets, respectively. The 𝑂𝑛𝑒𝐻𝑜𝑡 (•) is the one-hot encoding, and 𝑇𝑟𝑎𝑛𝑠𝑀𝑢𝑙𝑡𝑖 (•) denotes the multi-label transformation.Replay-based incremental methods alleviate catastrophic forgetting by preserving a rehearsal set. Typically, this rehearsal set uniformly selects samples from each incremental class to ensure adequate coverage of all old classes. However, achieving even selection across all character classes is challenging in incremental MLTR tasks, where word instances are the sample objects. This selection way may lead to some old character classes being excluded from the rehearsal set. This imbalanced rehearsal set may hinder the robust recognition. Compared to a large number of character classes, language classes are relatively tiny. Therefore, training a language class predictor might be feasible for enhancing the performance. Existing incremental MLTR methods design a language predictor solely at the word level. We argue that this word-level prediction is inadequate for sequential tasks, particularly text recognition. Firstly, considering the shared words between multiple languages, it is premature to assume that a word solely belongs to a specific language, as discussed in Section 3.3. Secondly, judging the language only at the word level disregards the abundant information at the fine-grained character level. For instance, as shown in Fig. 3(b) , if we simply assume the entire word is Japanese (encoding [3] ), the Japanese recognizer will be favored, disregarding that the first and second characters belong to both Chinese and Japanese (encoding [1, 3] ), which results in under-utilization of the feature extraction capability of the Chinese recognizer. Therefore, it is essential to analyze the language classes of each character in addition to those of the entire word. To achieve this, we introduce a hierarchical language estimation mechanism to analyze the belonging languages of words and characters, respectively. Technically, the features {R 1 (𝑥), • • • , R 𝑖 (𝑥)} extracted by the frozen specific recognizers are simultaneously fed into a DomainMLP. The architecture of the DomainMLP is detailed in Fig. 4 . By capturing the latent correlation between features in both spatial and channel dimensions, the DomainMLP predicts the word score S 𝑤 (𝑥) ∈ R 1×𝑖 and the character score S 𝑐 (𝑥) ∈ R 𝑇 ×𝑖 , where 𝑇 is the maximum decoding length. These scores are optimized by cross-entropy loss L 𝐶𝐸 for new knowledge and multilabel loss L 𝑚𝑢𝑙 for shared knowledge. Thus, we define the mixed loss L 𝑚𝑖𝑥 as,EQUATIONEQUATIONEQUATIONwhere * = {𝑤, 𝑐}, denoting word and character levels. N * and H * represent language class labels for the new and shared knowledge obtained from Eq. 4 and Eq. 5. In addition to being supervised by the knowledge analysis module, these language scores also serve as guidance for the specific recognizers trained in stage 1.The initial probabilities of specific recognizers can be considered generalizations to other languages. Typically, a specific recognizer performs better on words or characters seen in their own languages but worse on unseen ones. Given the shared words and characters, there may be multiple specific recognizers with better generalization rather than just one. Fully leveraging the feature extraction capabilities of these recognizers and mining the dependencies between languages to collaborate on recognition are crucial. Consequently, based on the language scores from the DomainMLP, we propose unified guided prediction for word level and specific guided prediction for character level. The unified indicates that all characters within a word have the same weight to recognizers, while the specific indicates that the attention weight varies among characters.More formally, given (𝑥, 𝑦) ∈ D 𝑖 ∪ M 𝑖 , we obtain a set of initial probabilities {P 1 (𝑥) ∈ R 𝑇 ×C 1 , • • • , P 𝑖 (𝑥) ∈ R 𝑇 ×C 𝑖 }. These probabilities are padded with zeros to align with the dimensions C𝑖 of the union of all classes, denoted as C𝑖 = ∪ 𝑖 𝑘=1 C 𝑘 . Thus, the aligned probabilities become{P 1 (𝑥) ∈ R 𝑇 × C𝑖 , • • • , P 𝑖 (𝑥) ∈ R 𝑇 × C𝑖 }.Then, the word-level probability P𝑤 (𝑥) after being guided by the word score S 𝑤 (𝑥) ∈ R 1×𝑖 in unified guided prediction is,EQUATIONSimilarily, the guided character-level probability P𝑐 (𝑥) by the character scoreEQUATIONwhere superscript (t) indicates the operation is performed at each time step t. In essence, both Eq. 9 and Eq. 10 employ a soft voting way to obtain the final guided probabilities. More discussion about the voting mechanism is provided in Sec. 4.3.2. These guided probabilities are optimized using standard crossentropy to enhance recognition performance,EQUATIONEQUATIONwhere P𝑤 (𝑦 𝑡 |𝑥) and P𝑐 (𝑦 𝑡 |𝑥) represent the word and character probabilities of the output being 𝑦 𝑡 at time step 𝑡.As shown in Eq. 1, only L 𝑐𝑙 𝑓 is used to optimize the specific language learning in stage 1. In stage 2, the loss comprises two components: the classification loss, L 𝑤 𝑐𝑙 𝑓 and L 𝑐 𝑐𝑙 𝑓 , and the language prediction loss, L 𝑤 𝑚𝑖𝑥 and L 𝑐 𝑚𝑖𝑥 . Therefore, the total training loss in the multilingual learning stage is defined as,EQUATION)where 𝜆 is a trade-off parameter.To better illustrate the training process of stage 2, the pseudocode is summarized in Algorithm 1. 
datasets and implementation details: 
 Datasets: Current mainstream multilingual text recognition datasets are MLT17 [25] and MLT19 [24] . MLT17 is a natural scene dataset with blur, occlusion, and distortion challenges. It comprises 6 scripts: Implementation Details: Three representative STR methods with different decoding ways, CRNN [30] , SVTR [7] , and TRBA [3] , are selected to deploy our framework. The default parameters are kept consistent with those in the original papers. The size of the rehearsal set M is set to 2,000, and the decoding length T is set to 25. Stages 1 and 2 are trained for 20,000 iterations with a batch size of 128. To ensure a fair comparison, we set the default incremental order as Chinese→Latin→Japanese→Korean→Arabic→Bangla. All experiments are conducted using PyTorch on two NVIDIA GeForce RTX 2080Ti GPUs.We conduct a comprehensive comparison between our proposed HAMMER and other incremental methods. Similar to MRN [52] , we deploy HAMMER on three STR methods with different decoding ways: CTC-based CRNN [30] , RNN-based TRBA [3] , and transformer-based SVTR [7] . Specifically, we utilize the output before CTC of CRNN, the output of TRBA after attention, and the output of the penultimate layer of the transformer decoder of SVTR as character features, respectively. Meanwhile, four popular IL methods, i.e., LwF [18] , EWC [15] , WA [48] , and DER [41] , are chosen to perform MLTR tasks, where the recognition results are derived from MRN. We establish an UpperBound model trained on all samples, representing the performance upper limit of our task. The Baseline model is trained solely on the current task without any incremental optimization and is tested directly on both old and new tasks. The results of task i (i>1) represent the average performance over both old and new tasks. The results of task 1 solely indicate the recognition ability of the current STR models (CRNN, SVTR, and TRBA) and cannot be used as an indicator to evaluate the incremental method. The AVG results represent the average performance across all tasks. The experimental results are summarized in Table 2 , demonstrating that:• The poor performance of the baseline model highlights significant differences in incremental languages. Consequently, models trained solely on new tasks experience degradation when tested on old tasks, leading to low average results. • The sub-optimal results of CRNN-based HAMMER in task 2 may be attributed to insufficient discrimination in character features between old and new tasks, mainly when the number of incremental tasks is small. However, as the number of tasks increases, HAMMER effectively handles incremental sharing between old and new tasks, improving average performance. • HAMMERs, deployed across the three STR methods, exhibit substantial improvement compared to other incremental methods. Stage 1 enables the exploration of specific knowledge within each language, while stage 2 delves into incremental sharing between languages. This cooperation facilitates deeper mining of old knowledge, thereby mitigating catastrophic forgetting.Due to the effectiveness of TRBA architecture, it is selected to perform ablation experiments. For brevity, only the results for Task6 and AVG are provided in the following experimental analysis unless otherwise specified.. Some experiments are conducted to validate the effectiveness of each proposed module. The results are summarized in Table 3 . We design several variants of HAMMER, indexed from top to bottom as 1-7. Word-level and Char-level indicate whether the language evaluation module predicts language scores at the word and character levels, respectively. New denotes considering all words and characters as new knowledge, optimized using standard cross-entropy, i.e., without considering incremental sharing situations. Shared indicates optimizing shared words and characters using multi-label loss. The baseline model (index-1) yields a relatively low average performance of 37.08% on MLT17 and 34.95% on MLT19. Comparing index-2, 3 and index-4, 5, we can see that predicting the belonging language scores only at the word level and only at the character level can improve the average performance, demonstrating the necessity of analyzing characters in text recognition tasks. Analyzing shared words (index-3) and shared characters (index-5) can facilitate mining the dependencies between the old and the new language, thus minimizing the catastrophic forgetting of the old language, as observed when comparing index-2 with index-3 and index-4 with index-5. Ultimately, the HAMMER (index-7) performance is further improved when all the proposed sub-modules are optimized.In both the unified and specific guided prediction modules, a soft voting mechanism is employed to guide the final prediction. Soft voting involves using language scores as weights for individual text recognizers, which are then weighted and summed to obtain the final output. We also test the impact of hard voting on model performance, where the class with the largest probability of the DomainMLP output is taken as an indicator to select the corresponding specific recognizer. From Table 4 , it can be observed that on MLT17 and MLT19, the results using soft voting at the word and char levels are much better than those using hard voting. Hard voting assumes that a word or a character belongs to a specific language, which contradicts multi-label optimization, resulting in poor performance. In contrast, the soft voting mechanism is consistent with the idea of multi-label optimization. It allows  
 
 Appendix:  


Decision: This paper proposes HAMMER, a novel Hierarchical Multi-label learning framework designed to address Multilingual Text Recognition (MLTR) by leveraging shared words and characters across languages to mitigate catastrophic forgetting. HAMMER's innovative approach, combined with its hierarchical evaluation strategy, shows promising results on benchmark datasets ML17 and ML19, outperforming state-of-the-art methods. While the framework's novelty and comprehensive analysis are commendable, the paper would benefit from addressing several reviewer concerns. Specifically, clarifying the mechanism of knowledge transfer between languages with different character styles, providing more insights into the effectiveness of shared knowledge, and enhancing the clarity and organization of the presentation are necessary. Additional experiments with various learning orders and comparisons with recent models would further validate the approach. Addressing these points in the camera-ready version will strengthen the paper's contributions and impact on the MLTR field.

Reviews: review: paper_summary: paper_summary: The paper focuses on the problem of Multilingual Text Recognition (MLTR), which is essential for facilitating cultural communication. Existing methods often struggle with retaining previous knowledge when learning new languages, which leads to the problem of catastrophic forgetting. HAMMER aims to mitigate this by using shared knowledge across languages, particularly through words and characters that appear simultaneously in multiple languages.

Specifically, HAMMER introduces a hierarchical approach that leverages shared words and characters across multiple languages. This shared knowledge is used to predict language scores at both word and character levels, which helps in retaining previously learned knowledge and acquiring new language knowledge.

The paper presents extensive experiments conducted on benchmark datasets ML17 and ML19, showing that HAMMER performs better than existing state-of-the-art approaches.
strength_weakness: strength_weakness: Novel Framework: HAMMER's hierarchical approach is innovative and addresses a significant challenge in MLTR by leveraging shared knowledge across languages.

Comprehensive Analysis: The paper provides a thorough analysis of shared knowledge and how it can be effectively used to predict language scores.

Promising Results: The experimental results on benchmark datasets demonstrate the robustness and effectiveness of the proposed method.
limitations: limitations: Though the idea of knowledge sharing sounds reasonable, the reviewer feels unclear of why it can work, especially for languages that have completely different styles of characters, i.e., English and Chinese. It will be helpful if the authors can share more insights or illustrations of how knowledge is transferred.

The studies in Table 6 are interesting; the order of learning matters for the final performance. Such an observation might partially prove the difficulty of transferring knowledge between languages of different characters. The reviewer suggests adding more orders and analyses.

Overall, I think this paper is solid and can bring value to the community. But the presentation should be improved, for example:

In Equations (4) and (5) on line 427, the authors use superscripts w and c to represent word-level and character-level, while the definition of them is introduced on line 509. This is confusing.

The authors are encouraged to put summarized sentences in table/figure captions, rather than simply drawing/listing the results.
paper_summary: paper_summary: This work conducts research on scene text recognition tasks in multi-language text recognition. The authors observe that effective learning of words and characters shared between languages has been neglected in previous work. Inspired by this observation, this work proposes a hierarchical multi-label learning framework for multilingual text recognition, called HAMMER. Online knowledge analysis aims to identify shared knowledge and provide corresponding multi-label language supervision. Specifically, only words and characters that appear simultaneously in multiple languages are considered shared knowledge. Furthermore, to further capture language dependence, this work introduces a hierarchical language assessment mechanism to predict word and character level language scores. These scores, under the supervision of knowledge analysis, guide specific recognizers to effectively exploit old and new language knowledge, thereby mitigating catastrophic forgetting due to imbalanced rehearsal sets.
strength_weakness: strength_weakness: This work starts from an observed phenomenon, gets inspiration, and then designs a model.
From the perspective of model design, this job is a good one, with its own ingenuity and uniqueness.
From the perspective of experimental implementation, this work has produced a large number of experiments, and the experimental design is reasonable and comprehensive.
limitations: limitations: The shortcomings of this work include:
1. In the era of deep learning, the ambiguity of shared words and characters will be alleviated due to contextual information. Is it appropriate to treat it as a multi-label learning task?
2. The statistics of words and characters in the dataset are given in Table 1. Except for Chinese, the proportion of tasks 2, 3, and 4 in the training set and test set is greater than that of tasks 5 and 6, but this pattern does not appear in the experimental results.
3. The experimental comparison models are relatively old, and two of the five comparison models are 16 years old.
paper_summary: paper_summary: This paper addresses the challenge of Multilingual Text Recognition (MLTR), particularly focusing on incremental learning scenarios where systems often grapple with preserving acquired knowledge when acquiring new languages. Introducing the concept of the "incremental sharing problem," the authors highlight the oversight of shared words and characters across languages in traditional incremental learning methodologies. To tackle this issue, they propose HAMMER, a Hierarchical Multi-label learning framework for Multilingual Text Recognition. HAMMER incorporates an online knowledge analysis system to identify and leverage shared language components, thereby enhancing the retention of old knowledge while assimilating new. By further implementing a hierarchical language evaluation strategy that operates at word and character levels, the framework promotes balanced learning and reduces catastrophic forgetting. Empirical validations on MLT17 and MLT19 datasets affirm the efficacy of HAMMER, establishing it as a new state-of-the-art solution in the field.
strength_weakness: strength_weakness: 1. Novelty in Addressing Incremental Sharing: The paper identifies and defines a new problem domain within incremental learning specific to MLTR, contributing a fresh perspective to the field.
2. Hierarchical Multi-Label Learning Framework: HAMMER's design, integrating multi-level recognition and shared knowledge exploitation, is innovative and comprehensive, enhancing the model's adaptability and robustness.
3. Practical Impact: The proposed solution directly tackles a real-world challenge in global communication, making MLTR more efficient and adaptive to evolving multilingual contexts.
4. Empirical Validation: Strong experimental results on main benchmark datasets validate HAMMER's superiority, underlining its practical significance and advancing the SOTA.
limitations: limitations: 1. Scalability Concerns: The effectiveness of HAMMER with an extensive number of languages or in scenarios with rapidly growing shared vocabulary sizes is not thoroughly explored.
2. Generalizability to Low-Resource Languages: The paper does not explicitly discuss how HAMMER would perform with low-resource languages that may lack sufficient shared words or characters with previously learned languages.
3. Complexity of Implementation: The hierarchical and multi-label learning components introduce added complexity, which may impact computational requirements and potentially hinder swift adoption or implementation by researchers or developers with limited resources.
4. Assessment of Shared Knowledge Identification: While the knowledge analysis module is a key feature, the paper does not delve deeply into its precision or robustness, leaving room for further scrutiny of its effectiveness in varied linguistic contexts.