General Detection-based Text Line Recognition

Authors: raphael  baena
syrine  kalleli
mathieu  aubry
encoder  layer
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 11:31:17

Abstract: We introduces a general detection-based approach to text line recognition, be it printed (OCR) or handwritten text (HTR), with latin, chinese or ciphered characters. Detection based approaches have until now largely been discarded for HTR because reading characters separately is often challenging, and character-level annotation is difficult and expensive. We overcome these challenges thanks to three main insights: (i) synthetic pre-training with diverse enough data enables to learn reasonable characters localization in any script; (ii) modern transformer-based detectors can jointly detect a large number of instances and, if trained with an adequate masking strategy, leverage consistency between the different detections; (iii) once a pre-trained detection model with approximate character localization is available, it is possible to fine-tune it with line-level annotation on real data, even with a different alphabet.  Our approach thus builds on a completely different paradigm than most state-of-the-art methods, which rely on autoregressive decoding, predicting character values one by one, while we treat a complete line in parallel. Remarkably, our method demonstrates good performance on range of scripts, usually tackled with specialized approaches: latin script, chinese script, and ciphers, for which we significantly improve state-of-the-art performances.

Main paper: 
 introduction: 
 A popular approach in early Optical Character Recognition (OCR) was to localize individual characters before processing them independently [7] . Such an approach, while still used for specific cases such as Chinese script [56] , cipher [41] , or scene text recognition [58] , has been largely discarded for Latin script Handwritten Text Recognition (HTR) in favor of implicit segmentation approaches. In this paper, we revisit character detection for text line recognition, demonstrate its effectiveness for HTR, and show results on a diverse range of datasets that we believe have not been demonstrated in prior work.Designing a general detection-based approach for handwritten text recognition is challenging. Individual characters are often not well separated in handwritten texts, and they are not always readable independently of their context. Thus, most datasets provide only line-level annotations, without character bounding boxes. This comes in addition to the common challenges of text recognition -such as diversity in writers, very rare characters, different kinds of noise and degradation -and challenges unique to certain datasets, for which specific approaches have been designed, e.g., Chinese script which has a very large alphabet, or ciphers for which the amount of annotated data is limited. Most recent methods for HTR for Latin script rely on recurrent or autoregressive models, and very few are devoid of any recurrence [10] . However, we believe that a detection-based approach could be general, tackle HTR with any script, and provide potential advantages in terms of interpretability -since the position of each character is clearly identified, and errors can be explained by mis-localization or Figure 1 : Our model is general and can be used on diverse datasets, including challenging handwritten script, Chinese script and ciphers. From left to right and top to bottom we show results on Google1000 [46] , IAM [29] , READ [44] , RIMES [39] , CASIA [27] , Cipher [41] datasets. mis-classification of ambiguous characters -as well as computational cost, since all characters can be processed in parallel.We design such an approach, dubbed DTLR, by leveraging three critical insights. First, training with diverse and sufficiently challenging synthetic data enables the localization part of a detection network to generalize to characters unseen during training. Second, modern transformer-based detectors [6, 57] can detect all characters in a text line in parallel and allow for detections to interact with each other, especially if encouraged by an adapted masking strategy. Third, we introduce an approach to fine-tune a detection network end-to-end using only line-level annotations and demonstrate that it can be used even with a completely new alphabet.Our contribution can be summarized as follows:• we present the first transformer-based character detection approach for text line recognition.• we demonstrate that our approach performs well on a wide range of datasets, including handwritten text in English, French, German, and Chinese text as well as ciphers. • we significantly improve state-of-the-art performances for cipher recognition.Our code and models are available at: https://github.com/raphael-baena/DTLR. 
related work: 
 Text recognition is the process of transcribing the text depicted in an image using computer algorithms. In this paper we focus solely on text line recognition, where a document image cropped around a line is given as input, which is the most standard setting in document analysis. In this section, we first give an overview of approaches for text line recognition, separating works focusing on Latin script, on Chinese script, and on ciphers, for which specific methods have been developed. We then discuss briefly the use of language models for text line recognition.Text line recognition for Latin script. In the 1990s many methods leveraged character segmentation for OCR [18, 4] and later in the 2000s for HTR [16, 45, 2] . They involved several steps: (i) character segmentation, (ii) character feature extraction, and (iii) character classification [7] . The final predictions were refined, for example using hidden Markov models [16, 2] or grouping characters into words and matching them with a lexicon [18, 31] . Because explicit segmentation in HTR poses significant challenges [11] , implicit segmentation techniques were developed, first using hidden Markov models [61, 8] , then the Connectionist Temporal Classification (CTC) loss [21] .Leveraging the CTC loss [21] became the dominant approach in the field. Indeed, it provides a solution to one of the important challenge in text line recognition, the disparity in the lengths and the misalignment between the predicted output sequences and the ground truth sequences. Hybrid system combining Convolutional Neural Network and Recurrent Neural Network [40] were Figure 2 : Architecture. Our architecture is based on DINO-DETR [57] . Given as input CNN image features, a transformer encoder predicts initial anchors and tokens, that are used by a transformer decoder to predict, for each token, a character bounding box and a probability for each character in the alphabet, including white space.arguably the most successful, with many variations, for example using Long Short-Term Memory (LSTM) [20, 35, 5] or Multidimensional LSTM layers [47, 36] . Fewer studies used CTC with non-recurrent schemes [10, 55, 9] , and are typically associated to lower performance [10] .More recently, still performing implicit character detection, the use of attention and transformers architecture enabled to complement [30] or replace [24, 13, 15, 14, 25, 19] the CTC loss with crossentropy, using an autoregressive decoder: given the beginning of a transcription, the decoder predicts the next character using cross-attention with the image features. Wick et al. [51] extended this concept by fusing the outputs of a forward-reading transformer decoder with a backward reading decoder. Autoregressive decoders can handle text at various levels of tokenization, such as characters [13] or sub-words [25, 19] . Predicting one token at a time seems computationally sub-optimal, but these methods currently achieve the state-of-the-art performances [19] .In contrast, our approach performs explicit character detection and is neither recurrent nor autoregressive. It revisits an early OCR paradigm with modern transformer-based detection architectures.Text line recognition for Chinese script. Explicit character segmentation, although no longer prevalent for Latin alphabets, remains an important approach for other languages, particularly Chinese HTR [53, 52, 33, 56] . Inspired by the YOLO framework [37] , Peng et al. [33] proposed a detection network for character bounding box and class prediction in text line recognition. Building on this work, Peng et al. [34] introduced a weakly supervised method to train the detection network, using synthetic data with character-level annotations for pre-training, and generating pseudo labels for weakly labeled real data. Closer to our approach Yu et al. [56] trained a detection model on a synthetic dataset in a fully supervised way and used CTC loss for examples from their target dataset with only line level annotations, CASIA [27] . Note that most of these methods pretrain on a very large synthetic dataset of Chinese characters, while we use a general pre-training and learn Chinese characters only on a limited real training set with line-level annotations.Text line recognition for ciphers. Recently, text recognition techniques have been used for transcribing historical ciphers, which can use different scripts, esoteric symbols, or diacritics. It is particularly challenging because for each cipher the number of documents and annotations is very limited, the underlying language is often unknown and no language model can be applied to the predictions [41] . Souibgui et al. [41] compared different approaches from HTR, including LSTM and transformer-based methods. Other approaches tried to explicitly segment or detect the characters. For example, Yin et al. [54] segmented the symbol and clustered them with a Gaussian Mixture Model, while Baró et al. [3] used K-Means. Antal and Marák [1] treated the problem as a character detection task but required character-level annotations for training.Language Model. Models that incorporate recurrent schemes, such as Handwritten Text Recognition (HTR) with RNN layers [40, 36] or autoregressive decoders [13, 25] , learn an implicit language bias that is crucial in HTR. Some models leverage very large amount of data to learn this language prior. For instance, TrOCR [25] used 684 million text lines, and DtrOCR [19] two billion to train the decoders. Instead, we complement smaller scale training with a masking strategy, similar to, e.g., Lyu et al. [28] . The masking strategy most similar to ours is used in Chaudhary and Bali [10] , where it is interpret solely as a form of data augmentation. Language models can also be used after the predictions have been made. Complex models can be trained for denoising, or with a masking strategy, e.g., Fang et al. [17] , Wang et al. [49] . We used instead a simple N-gram [15, 42, 43] approach, which we train only on the training dataset. For common languages, better results could be expected by training more complex language models at a much larger scale, but this is out of the scope of our study, where we instead aim at generality. 
method: 
 Given an input text-line image, our goal is to predict its transcription, i.e., a sequence of characters. We tackle this problem as a character detection task and build on the DINO-DETR architecture [57] , shown in Figure 2 , to simultaneously detect all characters. The rest of this section is organized as follows. First, in Section 3.1, we discuss how we pretrain a character-detection model using synthetic data with character-level supervision. Then, in Section 3.2, we explain how we finetune our model over real images from the target dataset with only line-level supervision and make final predictions.Data generation and masking strategy. To generate our synthetic data, we start by defining an alphabet A 0 and sampling sentences. We use two alphabets: one for Latin scripts and another for Chinese. The Latin alphabet contains 167 characters, including uppercase and lowercase Latin letters, whitespace, common symbols, and accented characters, while the Chinese alphabet includes 7,356 characters from the CASIA v1 dataset.To simulate text line images, we build on the synthetic generation pipeline of Monnier and Aubry [32] . For the Latin alphabet, we sample a font from a set of publicly available fonts1 , using a font labeled as "handwriting" with a 50% probability, and a random background from a set of empty page photographs. We use the font to render the text and blend it with the background by using a random color and adding structured noise to both the background and font layers. For the Chinese alphabet, we use the CASIA v1 dataset [27] , which contains various handwritten Chinese characters from different writers, to create synthetic sentences. We also apply significant blur data augmentation. This process results in challenging samples, as shown in the left column of Figure 3 .We sample sentences based on the application scenario. For known Latin-script languages, we sample sentences from Wikipedia in the target language. For ciphers, we generate random sentences by uniformly sampling Latin characters. For Chinese, we create random character sequences, since the limited alphabet of the CASIA v1 dataset makes it difficult to sample real Chinese text. This approach results in five models, for English, German, French, Chinese, and ciphers.In addition, we use random erasing, masking complete vertical blocks, and small horizontal blocks completely, similar to Zhong et al. [59] , Chaudhary and Bali [10] . This results in the text lines shown in the right column of Figure 3 . Masking increases the robustness of the model, but also has additional motivations. First, masking vertical blocks can make a character unreadable or even completely hide it. In cases where the text is sampled from natural language, our network is thus encouraged to learn an implicit language model to predict detections for unreadable or even non-visible characters. Since our training time is modest, this model is far simpler than Large Language Models, but we still found it significantly improved predictions.Second, masking small horizontal blocks will typically not make a character unreadable, but will prevent the model from focusing only on a discriminative part of a character. We found this to be especially important during the fine-tuning stage, where no character bounding box annotation is available.Architecture and loss Since character labels and bounding boxes are known in synthetic data, we can simply leverage a standard DINO-DETR architecture [57] , depicted in Figure 2 and the associated loss. Multi-scale image features are extracted through a CNN backbone and are then further refined in the N e transformer encoder layers through deformable attention [60] . The transformer decoder is composed of N d decoder layers which are fed a set of Q character queries. Each character query q is the concatenation of a content query and a positional query, initialized by the output of the encoder. In each decoder layer, the character queries first interact through self-attention and then attend to the encoder image features via deformable cross-attention. The decoder outputs Q bounding boxes bq ∈ R 4 and for each one a probability vector ĉq ∈ R |A0| , where |A 0 | is the number of characters in the alphabet A 0 and q is the query index.For each image, the loss compares these Q predictions to a set of N ground truth bounding boxes b n ∈ R 4 and their associated class label c n ∈ R |A| , where n is the index of the ground truth character. The box loss L box (b n , bq ) measures the distance between a ground truth box b n and a predicted box bq and is defined as a weighted sum between the L 1 distance and the generalized intersection over union (GIoU) :L box (b n , bq ) = λ 1 b n -bq 1 + λ iou GIoU (b n , bq ).The number of queries Q is typically much larger than the number of ground-truth characters N , so the set of ground truth is completed by Q -N objects associated to the "no object" class ∅. The Hungarian algorithm is used to find a permutation σ between queries and ground truth minimizing a matching cost:EQUATIONwhere L cls is the focal loss [26] and λ cls and λ box are two scalar hyperparameters. Once this matching σ has been found, the network is trained to minimize the loss:EQUATIONwhere λ ′ cls and λ ′ box are two scalar hyperparameters.Implementation details. We follow Zhang et al. [57] , and uses N e = 6 encoder layers, N d = 6 decoder layers, Q = 900 queries, and as hyperparameters λ cls = 2, λ box = 5, λ ′ cls = 1 and λ ′ box = 5. We generate synthetic datasets of 100k text lines, and train the networks for 225k iterations with batch size of 4, using the ADAM optimizer with β 1 = 0.9, β 2 = 0.999, a fixed learning rate of 10 -4 , and a weight decay of 10 -4 .Target alphabet. The alphabet of the target dataset A, might be different from the pretraining alphabet, A 0 . In our experiments, this is for example the case for ciphers. In this case, we replace the last linear layer responsible for character class prediction, adapting its size to the size of the target alphabet, and initialize each line, corresponding to a new character prediction, with random line of the pre-trained weight matrix, which we empirically found to work better than completely random initialization. Note the layers responsible for the bounding box coordinates predictions, which good initialization is critical for our approach, are unchanged. We actually found that thanks to the large variety of our training fonts, the character bounding boxes where reasonably good even for unseen characters. To train the new class prediction layer without destroying the rest of the network parameters, we fine-tune our network in two steps. First, for short a number of iterations (20k), we only optimize the re-initialized parameters, freezing the rest of the network. Then, we fine-tune the network end-to-end, thus also optimizing bounding box predictions. From detection to sequences of character probabilities. While most approaches to text recognition directly produce a sequence of character probabilities, our method outputs a set of detection without specific ordering and independent probabilities for all classes. For ordering the predictions, we simply sort them by the minimum x coordinates of the predicted boxes. While this could cause issues with slanted scripts, we did not see any in practice. For simplicity, we assume in the rest of the paper that the queries q are sorted with this order, which differs for each line.The fact that our probability predictions are independent for each class, similar to DINO-DETR, makes it non-trivial to decide when to predict the "no object" class and makes it harder to compute CTC and combine our prediction with language models using standard tools such as Tarride and Kermorvant [42] . Given a predicted class probability ĉq ∈ R |A| , we thus define a new probability vector cq ∈ R |A|+1 by:c|A|+1 q = 1 -i ĉi q , if i ĉi q < 1 -ε ε, otherwise and ci q = (1-ε)ĉ i q i ĉi q , if c|A|+1 q = ε ĉi q , otherwise, where ϵ is a scalar hyper-parameter, we use upper indices to refer to vector coordinates and the sums are between 1 and the size of the alphabet |A|. cq can now be interpreted as a joint probability vector for all classes, and c|A|+1 q as the probability of the "no object" class.Fine-tuning with adapted CTC. Unlike synthetic data, real-world datasets typically do not provide ground truth bounding boxes, but only line-level transcriptions. This makes it impossible to supervise character bounding boxes and to perform a Hungarian matching to associate tokens to characters. Thus, we instead adapt the CTC loss to fine-tune our network on real datasets. To do so, we leverage the sequence of character probabilities c defined in the previous paragraph. The standard CTC loss merges two successive identical characters, and requires them to be separated by a specific class, referred to as the blank symbol, to be separated. This is problematic for us since we might not predict empty bounding boxes between each character, and not necessary, since we detect each character independently and can directly use the bounding boxes to remove duplicate prediction using standard non-max suppression. We thus remove this behavior from the CTC loss. In practice, one can use standard implementations of the CTC loss and simply insert a blank symbol between each prediction.Note that even with a novel alphabet, the network pre-training is critical, because if the predicted bounding boxes do not coarsely correspond to characters, training with the CTC is not enough for the network to learn meaningful character localization. Predictions refinement. Once training is complete, the performance of the network can be slightly improved either by using non max suppression or by incorporating an explicit language model in the final decoding stage. We use the N-gram approach presented in [42] which combines N-gram probabilities with the CTC logits to include a prior on the likelihood of each text sequence. Similar to them, we find character-level models to be most suitable for our task as opposed to word-level models. However, unlike them, we found it beneficial to apply the N-gram word per word in each sentence, thereby making it similar to a spell-checking algorithm.Implementation details. We use ϵ = 0.003 to compute joint letter probabilities, and the same random erasing as for pre-training. We fine-tune our networks with the same parameters as for pre-training, except the learning rate for which we use 10 -5 for 1200k iterations and then 10 -6 for 800k iterations. For HTR in English, French and German, we train character-level N -grams on the text of the training set of each dataset with the KenLM [23] library and assign the N-gram a weight of 0.3 for each dataset. Otherwise, we use non-max suppression with an IoU threshold of 0.4. 
general text-line recognition: 
 Optical Character Recognition We performed OCR on the Google1000 dataset [22] , which contains scanned historical printed books, using our pre-trained model. We use the English Volume 0002, with 5,097 training lines, 567 validation lines, and 630 testing lines. Prior to fine-tuning on real data, our model has only 3.61% in Character Error Rate (CER) and this rate is further reduced to 2.04% after fine-tuning. Most of the "errors" are actually due to mislabeled samples. An example is shown in Figure 4 . In Appendix, we also show qualitative results on the ICDAR 2024 Competition on Multi Font Group Recognition (Figure 8 ), a dataset that includes multiple fonts and languages, showcasing our method's ability to handle various languages and printing styles.Handwritten Text Recognition on Latin Alphabets We evaluate our approach on text line HTR in various languages with latin script: IAM (English) [29] , RIMES [39] (French), and READ [44] (Old German). For IAM, we follow common practice [13, 25, 19] and use the unofficial Aachen partition 3 To assess the quality of our prediction, we use the Character Error Rate (CER) as defined by [13] . In Table 1 , we present the results of our experiments across all three datasets.On IAM, our results are satisfactory but clearly lower than state-of-the-art. The main reason seems to be that the dataset includes ambiguous samples which favors methods incorporating a more complex language model trained on large scale data [25, 19, 15] . Similarly, on the READ dataset, our performance is reasonable but still lower than the state-of-the art. This could be due to the challenges posed by the dataset: it has a small number of samples, a difficult handwriting style, and an old German language. In [13] , the authors initially trained their model on synthetic lines generated from the READ dataset, which helps the model to learn the language prior. On the RIMES dataset, our performance is more competitive, and only outperformed by Diaz et al. [15] which relies on a large internal dataset for training. We observe that the simple addition of a character-level N-gram, trained solely on the training corpus, results in noticeable improvements for all datasets.Handwritten Text Recognition on Chinese To showcase the versatility of our approach, we also evaluate our method on CASIA v2 [27] a benchmark for handwritten Chinese text-line recognition, using Accurate Rate (AR) and Correct Rate (CR) as defined in [56] . The alphabet for this database has 2,703 characters, significantly more than Latin script. The training set consists of 41,781 text lines, which we further divide into a new training set (80%) and a validation set (20%) while the test set has 10,449 text lines. We report our results in Table 2 . Our method outperforms the current state of the art by Yu et al. [56] in both accurate and correct rate.Ciphers We evaluate our approach on the Borg and Copiale cipher5 , using the line segmentation and splits outlined in [3] . The Borg cipher dataset 6 , derived from a 17th-century manuscript, has 34 unique characters. Qualitative analysis Most of our results are visually meaningful, as can be seen in Figure 1 . In Figure 4 , we show the worst results for each dataset, which outlines the limitations of our model and the difficulty or problems in the datasets. On the Google1000 sample, the annotation is inaccurate since it is provided by an OCR model. On the IAM dataset, our model struggles to distinguish lowercase from uppercase letters in a specific portion of the test set where the test set letters are mostly uppercase, because the training set does not include such lines. On the RIMES sample, the input image is an example of a badly cropped line, a recurring problem in this dataset. The model misses some of the letters in couvert and relies on a misleading visual cue for é instead of ê. On both READ and CASIA v2, the model misses a few characters with high variation in writing style such as the D in READ or digits in CASIA v2 which are sparse in the training set of the model. On For the line(s) with the highest error on each dataset (Google1000 [46] , IAM [29] , RIMES [39] , READ [44] , CASIA [27] , and Copiale [41] ) we show, our detections, the predicted text (P) and the ground-truth text (GT). Best seen in color.CASIA v2 where the fine-tuning process is much longer than for other datasets, we observed that the bounding boxes degenerate while the Character Error Rate (CER) continues to improve. This can be understood by the fact that no specific loss is used to constrain bounding boxes, and that features have large receptive fields. Thus, in Figure 1 , we show the prediction of our model in middle of the training process, before the bounding boxes degenerate. On the Copiale dataset, the error comes from the labels, the last dozen characters are present in the image but not in the annotation.Ablation We conduct an ablation study in Table 4 . We first examine the impact of the pretraining dataset on IAM, which uses latin script. Using a model trained on English and random erasing both improve performances. We then study the effects of the finetuning strategy as well as the presence of random erasing during finetuning on the IAM and Borg datasets.Freezing the network and learning only the character classification layer significant boost performances on IAM, and results in a model that is better than random on Borg, but both still remain low quality, and end-to-end fine-tuning is necessary to obtain good performances. The results highlights the significant benefit of the random erasing strategy during finetuning, hinting that it helps our method to learn an implicit language model.Inference speed We compare our inference speed to TrOCr [25] and FasterDAN [14] for which public code is available, on text lines from the RIMES dataset [44] with a batch size of 1 and using an A6000 GPU. Our method requires 67ms for inference, while TrOCR takes 284ms and FasterDAN takes 140ms. However, FasterDAN is designed to recognize efficiently the lines of an entire document in parallel. When evaluating it on whole documents from RIMES and dividing the inference time by the average number of lines, one finds it requires only 7ms per line. While we can use a batch  
conclusion: 
 We presented a character detection approach to text line recognition. Although it is conceptually different from most modern HTR approaches, we demonstrated that it performs well on a wide diversity of benchmarks and achieves state-of-the-art performance for Chinese scripts and ciphers recognition. We hope that our general approach will revive detection-based approaches to text recognition and encourage the evaluation of future approaches on more diverse data. 
 
 Appendix: https://github.com/google/fontshttps://www.openslr.org/56/https://zenodo.org/records/10805048https://pages.cvc.uab.es/abaro/datasets.htmlhttps://digi.vatlib.it/view/MSS_Borg.lat.898https://rrc.cvc.uab.es/?ch=27 


Decision: The paper introduces a novel transformer-based handwritten character detection for text line recognition. The authors use a diverse set of synthetic data to enable localization part of the detection network to generalize to unseen characters during training. The transformer-based detectors can identify all characters in a text line in parallel and a masking strategy has been adopted to encourage interactions  between detections. The methods proposed by the authors also includes a process to fine-tune the detection network using only line-level annotations.

All reviewers support weak accept (6), some of them having increased their scores during the rebuttal.

Most reviewers agree that this paper is well written and supported by solid evaluations that compare well to the state-of-the art. While it revisits an earlier OCR paradigm, this approach is very original when compared to current mode holistic handwritten recognition approaches based on auto-regressive models. Compared to those, it seems much more efficient in terms of inference cost and also training cost. Synthetic data could make training on low resource languages more accessible.

The main weakness mentioned by 2 reviewers (9jmv, hupu) is the limited technical contribution. The author solid rebuttal, showing that they use the computer vision DINO-DETR model in a novel way in combination with CTC, caused one of the reviewers to increase his score.
Another limitation is the absence of real world applications, going from single lines to full documents. This would have allowed the authors to show how more cost effective their approach is.

This paper deserves a weak accept due to the limited technical contribution, but it could have a significant impact on the field given the originality and efficiency of the method

Reviews: review: paper_summary: paper_summary: This paper treats text line recognition as an object detection task and proposes a two-stage training approach based on DINO-DETR. In the first stage, synthetic data with bounding box information is used to predict the bounding boxes and categories of text, due to the absence of character-level annotations for text line datasets. In the second stage, real text line data is employed to sequentially fine-tune the classifier and the entire model. The proposed method achieves state-of-the-art performance in cipher text.
strength_weakness: strength_weakness: - Abalation study is easy. 
- Utilizes more engineering skills than methodological innovations.
questions: questions: All results in the paper were pre-trained on Latin text and fine-tuned on several datasets. What if we pre-trained and fine-tuned on data in the same language? For example, pre-training on synthetic Chinese text lines followed by fine-tuning on real Chinese datasets.
limitations: limitations: The proposed method performs well only on the cipher text recognition task.
paper_summary: paper_summary: This paper presents a novel detection-based approach to text line recognition for both printed (OCR) and handwritten text (HTR), covering Latin, Chinese and cipher characters. Traditional detection-based methods have been largely neglected in HTR due to the difficulty of reading characters separately and the high cost of character-level annotation. The authors propose a solution to these challenges through three main insights: (i) using synthetic pre-training with diverse data for character localisation across different scripts; (ii) using modern transformer-based detectors to handle multiple character instances simultaneously, using a masking strategy to ensure consistency; and (iii) fine-tuning a pre-trained detection model with approximate character localisation using line-level annotations on real data, even with different alphabets.
strength_weakness: strength_weakness: - Lack of new technical contributions: Despite the originality of the approach, the paper lacks novel technical contributions. The model, training strategy, fine-tuning with CTC, data augmentation and synthetic data generation are not novel.
- Performance on handwritten documents: The method is not competitive on Latin handwritten documents and is outperformed by existing methods on Chinese handwritten documents. It does outperform on cipher recognition, but this is a more specialised and less researched area.
- Scope of evaluation: The framework is only evaluated on perfectly segmented lines of text, raising questions about its performance on full pages. Full page processing would require additional steps such as text line detection and reading order retrieval, which could impact performance.
- Impact of line detection: There are concerns about how the quality of line detection would affect character detection and recognition, particularly in cases where vertical or horizontal lines merge.
questions: questions: - Error cases: Can you provide examples of challenging real-world examples, such as rotated, upside-down or slanted lines; blank lines; vertically merged lines; strikethrough text; translucent paper; and mixed printed/handwritten characters?
- Pre-training: How long was the model trained, on what hardware (GPUs), and why was pre-training limited to 100k lines of text? Was this value determined experimentally?
- Input image size: What was the input image size used for training and evaluation?
- Computational cost: How do you explain the difference in computational cost between DINO-DETR (25ms/line) and FasterDAN (7ms/line)? Why were comparisons made using a batch size of 1? Please provide a more comprehensive comparison, including batch sizes, CPU vs. GPU performance, impact of language modelling, input image sizes, and model parameters.
- Language model decoding: What is the impact of a language model on inference speed? Can character recognition/classification still be parallelized with a language model? Can you decode on GPU with a KenLM language model, and what size of N-gram language model do you use?
- Model release: Will the model be released to the public?
limitations: limitations: - Conclusion: The conclusion lacks depth and provides no insight into future improvements to the method.
- Paper and writing:
    - Figure 2: The figure is unclear and needs to be made clearer.
    - Table 1: The table is misplaced, it appears on page 6 but is referenced on page 7.
paper_summary: paper_summary: The paper introduces a novel transformer-based character detection for text line recognition. The authors use a diverse set of synthetic data to enable localization part of the detection network to generalize to unseen characters during training. The transformer-based detectors can identify all characters in a text line in parallel and a masking strategy has been adopted to encourage detection interactions. The methods proposed by the authors also includes a process to fine-tune the detection network using only line-level annotations.
strength_weakness: strength_weakness: W1: Several typos throughout the paper. Ex: typo in the Fig 1 caption: “Our model is general can be”, 
W2: The authors can provide additional information on the fine-tuning process and the adaptability of the masking strategy. Did they explore other strategies?
W3: The performance on real-world datasets that contain some-to-large amount(s) of noise has not been extensively discussed by the authors.
questions: questions: Q1: How does the model handle noisy datasets or texts with degraded quality?
Q2: What were the computational resource requirements? Can this approach be extended to real-world problems in a different setting?
Q3: How would LLMs affect or change the impact of this approach? Have the authors considered this aspect, and if the proposed method can do better than some existing LLMs?
Q4: Can this approach be extended to multi-line text detection and recognition?
limitations: limitations: L1: The efficiency of the proposed method in real-world datasets with noises has not been fully explored.
paper_summary: paper_summary: This manuscript provides a new approach to line recognition of offline handwriting and printed documents with regard to multilingualism.

This method can recognize a line based on its simultaneously recognized features based on transformers.

This approach is interesting for document analysis.

Promising results have been obtained for different datasets.
strength_weakness: strength_weakness: This is not a whole system for the field of document analysis. This task is a part of document analysis that depends on line segmentation. This means that if the previous part (line segmentation) of the process has errors, these are transferred to the next steps. Therefore, this is an essential task for the work. What happens if you have a whole document for the task? What happens to your results if you use existing methods that segment lines? 

This method is not generally suitable for multilingual contexts. If there are several characters in different languages, this paper has no clear solution. You should therefore have several models for each language.

It is better to explain the method for your synthetic data in more detail in the Appendix. It is better to clarify this with an illustration.

Please check your text again for some errors, e.g. line 263 (annotations annotations).

Is line 230 correct? "the English Volume 0002"
questions: questions: When I compared your transformer architecture to the original [57], Figure 2 is different from the original. They use a query selection
 and matching. Why do not you have these parts? I can see them in your description, but not in your illustration.

You cite Figure 1 in subsection 4.2, why do you give the figure as Figure 1 without citing it earlier?

What happens if you have a document with multiple languages? Do you test your method for the type of input? Or should you train your network for this type of data?

Why don't you create the synthetic data for other languages and use a fine-tuning model based on English? Have you considered the scenario?
limitations: limitations: They showed some limitations in Figure 4. But it needs to add more samples in an Appendix.

Comments: comment: Thank you for the detailed answer. I'm convinced of the value of the proposed method, mainly because of its originality compared with current methods. Without being revolutionary, it opens up interesting perspectives and I would like to see it tested further. The availability of the code and models will allow this. I'm going to raise my score.
Thank you for your response. I will increase the rating.
Thank you for addressing my questions and comments. However, I believe the work requires some major revisions. Given this, I will maintain my score of 6: Weak Accept because the work presented is novel and interesting and the authors have given good explanation in the rebuttal. It just requires some major revisions.