General Detection-based Text Line Recognition
Authors: Raphael Baena
Syrine Kalleli
Mathieu Aubry
Venue: NeurIPS 2024 poster (2024)
Keywords: Text Recognition
Handwritten Text Recognition
Optical Character Recognition
Transformer
Primary area: machine_vision
TL;DR: A transformer-based OCR/HTR approach that performs explicit character detection and we demonstrate to be very general.

Abstract: We introduce a general detection-based approach to text line recognition, be it printed (OCR) or handwritten text (HTR), with latin, chinese or ciphered characters. Detection-based approaches have until now largely been discarded for HTR because reading characters separately is often challenging, and character-level annotation is difficult and expensive. We overcome these challenges thanks to three main insights: (i) synthetic pre-training with diverse enough data to learn reasonable character localization in any script; (ii) modern transformer-based detectors can jointly detect a large number of instances and, if trained with an adequate masking strategy, leverage consistency between the different detections; (iii) once a pre-trained detection model with approximate character localization is available, it is possible to fine-tune it with line-level annotation on real data, even with a different alphabet.  Our approach thus builds on a completely different paradigm than most state-of-the-art methods, which rely on autoregressive decoding, predicting character values one by one, while we treat a complete line in parallel. Remarkably, our method demonstrates good performance on range of scripts, usually tackled with specialized approaches: latin script, chinese script, and ciphers, for which we significantly improve state-of-the-art performances. 
Our code and models are available at [https://github.com/raphael-baena/DTLR](https://github.com/raphael-baena/DTLR).

General Detection-based Text Line Recognition
Raphael Baena, Syrine Kalleli, Mathieu Aubry LIGM, Ecole des Ponts, Univ Gustave Eiffel, CNRS
Marne-la-Vallé, France
firstname. Lastname@enpc.FF
Abstract
We introduce general detection-based approach to text line recognition, be it printed (OCR) or handwritten (HTR), with Latin, Chinese, ciphered characters. Detection-based approaches have until now been largely discarded for HTR because reading characters separately is often challenging, and character-level annotation is difficult and expensive. We overcome these challenges thanks to three main insights: (i) synthetic pre-training with sufficiently diverse data enables learning reasonable character localization for any script; (ii) modern transformer-base detectors can jointly detect large number of instances, and, if trained with adequate masking strategy, leverage consistency between the different detections; (iii) once a pre-trained detection model with approximate character localization is available, it is possible to fine-tune it with line-level annotation on real data, even with different alphabet. Our approach, dubbed DTLR, builds different paradigm than state-of-the ff-th--art HTR methods, which rely autoregressive decoding, predicting character values one by one, while we treat complete line in parallel. Remarkably, we demonstrate good performance a large range of scripts, usually tackled with specialized approaches. state-of-the-ate performances for Chinese script recognition on the CASIA v2 dataset, and for eipher recognition on the Borg and Copiale datasets. Our code and www.//rappaleebaaaa//DDD
completely
particular, we improve
models are available at
Introduction
popular approach early Optical Character Recognition (OCR) was to localize individual charac- ters before processing them independently 77. Such an approach, while still used for specific cases such as Chinese script (56), cipher [41]. or scene text recognition [58), has been largely discarded for Latin script Handwritten Text Recognition (HTR) in favor of implicit segmentation approaches. In this paper, we revisit character detection for text line recognition, demonstrate its effectiveness for HTR, and show results diverse range of datasets that believe have not been demonstrated in prior work.
Designing general detection based approach for handwritten text recognition is challenging. Indi- vidual characters are often not well separated handwritten texts, and they are not always readable independently of their context. Thus, most datasets provide only line-level annotations, without char- acter bounding boxes. This comes addition to the common challenges of text recognition such as diversity in writers, very characters, different kinds of noise and degradation and challenges unique to certain datasets, for which specific approaches have been designed, e.g., Chinese script which has very large alphabet, or ciphers for which the amount of annotated data is limited. Most recent methods for HTR for Latin script rely on recurrent or autoregressive models, and very few are devoid of any recurrence [TO. However, we believe that detection-basel approach could be general, tackle HTR with any script, and provide potential advantages in terms of interpretability. since the position of each character is clearly identified, and enrors be explained by mis-localization
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

strikee 1t 5 ehowed
Bri
al tc
country, and espied more
country, and d espi ed more
strik e. it show ed
Br
SimerrEs salu FatRonas
trunk
5
rg k oc: h thr p. vônn
a
c
h3
Figure 1: Our model is general and can be used on diverse datasets, including challenging hand- written script, Chinese script and ciphers. From left to right and top to bottom we show results on
Google1000 46. IAM (29). READ 44), RIMES 39]. CASIA [27] Cipher 41 datasets.
mis-classification of ambiguous characters as well as computational cost, since characters can be processed in parallel.
We design such an approach, dubbed DTLR, by leveraging three critical insights. First, training with diverse and sufficiently challenging synthetic data enables the localization part of a detection network to generalize to characters unseen during training. Second, modern transformer-basse detectors 57 can detect all characters in a text line in parallel and allow for detections to interact with each other, especially if encouraged by an adapted masking strategy. Third, we introduce an approach to fine-tune detection network end-to-end using only line-level annotations and demonstrate that it can be used even with a completely new alphabet.
Our contribution can be summarized as follows:
present the first transformer- character detection approach for text line recognition. we demonstrate that our approach performs well on a wide range of datasets, including handwritten text in English, French, German, and Chinese text well ciphers. we significantly improve state-of-the-art performances for cipher recognition.
Our code and models are available at: cowpppppaa.ccmmmmpaaaaaa..aaaall bae.aaaaaaaaaaaaaa baeRa/DTLR DTLR
2 Related Work
Text recognition is the process of transcribing the text depicted in an image using computer algorithms. In this paper we focus solely on text line recognition, where a document image cropped around a line is given as input, which is the most standard setting in document analysis. In this section, we first give an overview of approaches for text line recognition, separating works focusing on Latin script, on Chinese script, and on ciphers, for which specific methods have been developed. We then discuss briefly the use of language models for text line recognition.
Text line recognition for Latin script. In the 1990s many methods leveraged character segmen- tation for OCR [18] 4 and later in the 2000s for HTR [I6] 45 2). They involved several steps: (i) character segmentation, (ii) character feature extraction, and (iii) character classification 77]. The final predictions were refined, for example using hidden Markov models [6] or grouping characters into words and matching them with lexicon [8 31. Because explicit segmentation in HTR poses significant challenges [T], implicit segmentation techniques were developed, first using hidden Markov models [6, then the Connectionist Temporal Classification (CTC) loss 21).
Leveraging the CTC loss 21 became the dominant approach the field. Indeed, it provides a solution to one of the important challenge in text line recognition, the disparity in the lengths and the misalignment between the predicted output sequences and the ground truth sequences. Hybrid system combining Convolutional Neural Network and Recurrent Neural Network [40] were

---

(6)
institutin
CNN
(
Decoder Layer x.N
Encoder Layer x
Feed-Forward Network
Feed-Foward Network
Query Soleeiio
Values
Keys
Scale-Scale Deformable Self Attention
Multi-Scale Deformable Self
Positional
Attention
Self Allention
Encoding
Learnalee Content
Queries
Init Anchor
Figure 2: Architecture. Our architecture based on DINO-DETR (57) Given input CNN image features, a transformer encoder predicts initial anchors and tokens, that are used by transformer decoder to predict, for each token, character bounding box and probability for each character in the alphabet, including white space.
arguably the most successful, with many variations, for example using Long Short-Term Memory (LSTM) 20
35 5 or Multidimensional LSTM layers [47 36
Fewer studies used CTC with 55 9. and are typically associated to lower performance [10]
non-recurrent schemes
T0
More recently, still performing implicit character detection, the use of attention and transformers architecture enabled complement [30] replace [24
14 25 [9] the CTC loss with cross- entropy, using an autoregressive decoder: given the beginning of a transcription, the decoder predicts the next character using cross-attention with the image features. Wick al. [5]] extended this concept by fusing the outputs of forward -raaiigg transformer decoder with a backward reading decoder. Autoregressive decoders can handle text at various levels of tokenization, such characters [3] or sub-words [25] 19.. Predicting one token at time seems computationally sub-optimal, but these methods currently achieve the state-of-the-ar art performances
13
15
In contrast, our approach performs explicit character detection and is neither recurrent nor autoregres- sive. It revisits early OCR paradigm with modern transformer-based detection architectures.
Text line recognition for Chinese script. Explicit character segmentation, although no longer prevalent for Latin alphabets, remains an important approach for other languages, particularly Chinese (37], Peng et al. 33 proposed a detection network for character bounding box and class prediction in text line recognition. Building on this work, Peng et al. (34) introduced a weakly supervised method to train the detection network, using synthetic data with character-level annotations for pre-training and generating pseudo labels for weakly labeled real data. Closer to our approach Yu et al. [56) trained a detection model on synthetic dataset in a fully supervised way and used CTC loss for examples from their target dataset with only line level annotations, CASIA (27). Note that most of these methods pretrain on very large synthetic dataset of Chinese characters, while we use general pre- pre-training and learn Chinese characters only on a limited real training set with line-level annotations.
HTR [53 52, 33 [66] Inspired by the YOLO framework
Text line recognition for ciphers. Recently, text recognition techniques have been used for tran- scribing historical ciphers, which can use different scripts, esoteric symbols. or diacritics. It is particularly challenging because for each cipher the number of documents and annotations is very limited, the underlying language is often unknown and no language model can be applied to the predictions [41] Souibgui al. [41] compared different approaches from HTR, including LSTT and censformer-based methods. Other approaches tried to explicitly segment or detect the characters. For example, Yin al. [54) segmented the symbol and clustered them with a Gaussian Mixture Model, while Baró et al. [3] used K-Means. Antal and Marák [] treated the problem as a character detection task but required character-level annotations for training.
Language Model. Models that incorporate recurrent schemes, such as Handwritten Text Recogni- tion (HTR) with RNN layers 440 36) or autoregressive decoders [13] 25), learn an implicit language
bias that is crucial in HTR. Some models leverage very large amount of data to learn this language
3

---

prior. For instance, TrOCR [25] used 684 million text lines, and DtrOCR [9 two billion to train the decoders. Instead, we complement smaller scale training with masking strategy, similar to. e.g., Lyu et al. 28 The masking strategy most similar to ours is used in Chaudhary and Bali [00] where it is interpret solely as a form of data augmentation. Language models can also used after the predictions have been made. Complex models can be trained for denoising, or with masking strategy, e.g., Fang et al. [[7] Wang et al. [49]. We used instead a simple N-gram [1S] 42 43 approach, which we train only the training dataset. For common languages, better results could be expected by training more complex language models at a much larger scale, but this is out of the scope of our study, where we instead aim generality.
3 Method
Given an input text-line image, our goal is to predict its transcription, i.e., a sequence of characters. We tackle this problem as a character detection task and build on the DINO-DETR architecture [57], shown in Figure 2 to simultaneously detect all characters. The rest of this section is organized as follows. First, in Section 3.1 we discuss how pretrain character-detection model using synthetic data with character over real images from the target dataset with only line-level supervision and make final predictions.
supervision. Then, in Section 3.2 we explain how we finetune our model
3.1 Synthetic pre-training
Data generation and masking strategy. To generate our synthetic data, we start by defining an alphabet Ao and sampling sentences. We use two alphabets: one for Latin scripts and another for Chinese. The Latin alphabet contains 167 characters, including uppercase and lowercase Latin letters, whitespace, common symbols, and accented characters, while the Chinese alphabet includes 7,356 characters from the CASIA vI dataset.
To simulate text line images, we build on the synthetic generation pipeline of Monnier and Aubry [32]. For the Latin alphabet, we sample a font from set of publicly available fonts" using font labeled as "handwriting" with a 50% probability, and a random background from a set of empty page photographs. We use the font render the text and blend it with the background by using a random color and adding structured noise to both the background and font layers. For the Chinese alphabet, we use the CASIA v1 dataset (27) which contains various handwritten Chinese characters from different writers, to create synthetic sentences. We also apply significant blur data augmentation. This process results in challenging samples, as shown the left column of Figure 3
We sample sentences based on the application scenario. For known Latin-script languages, we sample sentences from Wikipedia in the target language. For ciphers, we generate random sentences by uniformly sampling Latin characters. For Chinese, we create random character sequences, since the limited alphabet of the CASIA v1 dataset makes difficult to sample real Chinese text. This approach results in five models, for English, German, French, Chinese, and ciphers.
In addition, use random erasing, masking complete vertical blocks, and small horizontal blocks completely, similar to Zhong et al. [59]. Chaudhary and Bali [00. This results in the text lines shown in the right column of Figure 3 Masking increases the robustness of the model, but also has additional motivations. First, masking vertical blocks can make a character unreadable or even completely hide it. In cases where the text is sampled from natural language, our network is thus encouraged to learn an implicit language model to predict detections for unreadable or even non-visible characters. Since our training time modest, this model is far simpler than Large Language Models, but we still found it significantly improved predictions.
Second, masking small horizontal blocks will typically not make character unreadable, but will prevent the model from focusing only on a discriminative part of a character. We found this to be especially important during the fine-tuning stage, where no character bounding box annotation is available.
Architecture and loss can simply leverage a standard DINO-DETR architecture [57]. depicted in Figure the associated loss. Multi-scale image features extracted through CNN backbone and are then further refined
Since character labels and bounding boxes are known in synthetic data,
hanpss//gttubbcco//ogg///ttttttt

---

BNZDdSEBe G38AyPP
1989 désintégration
apire sovie
Snjee of REP Patthagens s.
DAS DECTSCHE SHE W WADDELGRA
AUCH AUSDAUSH
OER L
CH Loo
SO aune TOO COSTICETCIE KLUE
ODNI soverss
Soaipa
Rvd
eep Figure Samples from our synthetic datasets without (left) and with masking (right).
M
x
in the Ne transformer encoder layers through deformable attention [60] The transformer decoder is composed Na decoder layers which are fed set of Q character queries. Each character query q is the concatenation of a content query and positional query, initialized by the output of the encoder. In each decoder layer, the character queries first interact through self-attention and then attend to the encoder image features via deformable cross-attention. The decoder outputs Q bounding boxes by E IR and for each one a probability vector 6, E IR Aol, where | is the number of characters in the alphabet Ag and q is the query index.
For each image, the loss compares these Q predictions to set of N ground truth bounding boxes bn E R and their associated class label C, E TR/ TRILAI, where n is the index of the ground truth character. The box loss Loox (b,, b²) measures the distance between ground truth box b, and predicted box by and is defined as a weighted sum between the I distance and the generalized intersection over union (GIoU): Loox (b,, b,,) = A b, by + GLoU (b,,, byy) The number of queries Q is typically much larger than the number of ground truth characters N, the set of ground truth is completed by Q - objects associated to the "no object" class (). The Hungarian algorithm is used to find permutation ā between queries and ground truth minimizing matching cost:
ô arg min Let (o()) + 1 (cr Amox Lrox (b., (o)):
where Loss the focal loss (26) and dets and Xbox are two sealar hyperparameters. Once this matching & has been found, the network is trained to minimize the loss:
EXaLas(Cr, + (c,700 Yoax Loox (b,, ba(i))
(2)
where Xis and Xoxx are two scalar hyperparameters.
Implementation details. We follow Zhang et al. 157. and uses N, = 6 encoder layers, Na = 6 decoder layers, Q 900 queries, and as hyperparameters A 2, Abox 5, X/ and X 5. We generate synthetic datasets of 100k text lines, and train the networks for 225k iterations with batch size of 4, using the ADAM optimizer with 31 0.9, 32 0.999, fixed learning rate of 10 and weight decay of 10-4.
3.2 Finetuning with line-level annotations.
Target alphabet. The alphabet of the target dataset A, might be different from the pretraining alphabet, Ap. In our experiments, this is for example the case for ciphers. In this case, we replace the last linear layer responsible for character class prediction, adapting its size to the size of the target alphabet, and initialize each line, corresponding to a new character prediction, with random line of the pre-trained weight matrix, which we empirically found work better than completely random initialization. Note the layers responsible for the bounding box coordinates predictions, which good initialization critical for our approach, are unchanged. We actually found that thanks to the large variety of our training fonts, the character bounding boxes where reasonably good even for unseen characters. To train the new class prediction layer without destroying the rest of the network parameters, we fine-tune our network in two steps. First, for short number of iterations (20k), we only optimize the re- iniitiaaiized parameters, freezing the rest of the network. Then, we fine-tune the network end-to-end thus also optimizing bounding box predictions.
5

---

Table 1: Character Error Rate (CER, in %6) on standard HTR datasets.
Method
IAM 29
READ 44
RIMES [39
Sanchez et al.
38
5.10
Michael et al.
30
4.87
4.70
3.04
Diaz
15
3.53
2.48
Diaz et al.
15
with LM
2.75
1.99
PyLai
8.44
4.57
PyLaia+ N-gram 42
7.50
3.79
VAN 12
3.04
DAN
13
5.01
4.10
2.63
DAN
13 + N-gram
4.38
3.15
TrOCR
25
2.89
DTrOCR Fujitake
19
2.38
Ours (DTLR) w/o. N-gram
5.99
5.53
3.51
Ours (DTLR) w. N-gram
5.46
5.35
2.53
From detection to sequences of character probabilities. While most approaches to text recog- nition directly produce sequence of character probabilities, our method outputs a of detection without specific ordering and independent probabilities for all classes. For ordering the predictions, we simply sort them by the minimum x coordinates of the predicted boxes. While this could cause issues with slanted scripts, we did not see any in practice. For simplicity, we assume in the rest of the paper that the queries are sorted with this order, which differs for each line.
The fact that our probability predictions are independent for each class, similar to DINO-DETR, makes non-trivial to decide when to predict the "no object" class and makes it harder to compute CTC and combine our prediction with language models using standard tools such as Tarride and Kermorvant (2). Given a predicted class probability č E RRAd, we thus define a new probability vector C2 E glA++1 by:
Σ E/p if E <1- otherwise
g|+++
E,
(1-ejé) Σ, if 14++
and c
=
otherwise
where is scalar hyper-parameter, we upper indices to refer vector coordinates and the sums are between and the size of the alphabet |A]. C22 can now be interpreted as a joint probability vector for all classes, and C2
|A|+1 the probability of the "no object" class.
Fine-tuning with adapted CTC. Unlike synthetic data, real-world datasets typically do not provide ground truth bounding boxes, but only line-level transcriptions. This makes it impossible to supervise character bounding boxes and to perform Hungarian matching to associate tokens to characters. Thus, we instead adapt the CTC loss to fine-tune our network on real datasets. To do we leverage the sequence of character probabilitics ē defined in the previous paragraph. The standard CTC loss merges two successive identical characters, and requires them to be separated by specific class, referred as the blank symbol, to be separated. This is problematic for us since we might not predict empty bounding boxes between each character, and not necessary, since we detect each character independently and can directly use the bounding boxes to remove duplicate prediction using standard non-max suppression. We thus remove this behavior from the CTC loss. In practice, standard implementations of the CTC loss and simply insert a blank symbol between each prediction.
Note that even with a novel alphabet, the network pre-training is critical, because if the predicted bounding boxes do not coarsely correspond to characters, training with the CTC not enough for the network to learn meaningful character localization.

---

Table 2: Accurate Rate (AR) and Correct Rate (CR) [56] for Chinese HTR on CASIA
Method
Training Data
AR (%)
CR (%) 1
Wang al.
48
CASIA vI CASIA v2
90.20
90.80
Wang et al.
48
CASIA v1 + CASIA v2
90.23
90.80
Peng al. 33
Synthetic (Chinese) CASIA
90.52
89.61
Wang et al. [50]
CASIA v1 + CASIA v2
92.18
90.77
Yu al. [56
Synthetic (Chinese) + CASIA
96.77
96.77
Ours (DTLR)
Synthetic (Chinese) + CASIA 96.83
97.34
Table 3: Symbol Error Rates (SER) and Word Accuracy (WA) for cipher recognition
Copiale cipher
Borg cipher
Method
SER (%)
WA (%)
SER (%) 4
WA (%) 1
Text-DIAE 41
4.1
81.7
10.5
15.6
Seq2Seq Attention
3.6
82.8
10.7
54.7
LSTM (VGG) 41
3.3
81.7
9.4
59.4
LSTM
41
4.6
78.9
13.8
44.2
Ours (DTLR)
2.2
84.3
8.5
59.8
Predictions refinement. Once training is complete, the performance of the network can be slightly improved either by using non max suppression by incorporating an explicit language model in the final decoding stage. We use the N-gram approach presented 42 which combines N-gram probabilities with the CTC logits to include a prior on the likelihood of each text sequence. Similar to them, we find character-level models to most suitable for our task as opposed to word-level models. However, unlike them, we found it beneficial to apply the N-gram word per word in each sentence, thereby making it similar spell-checking algorithm.
Implementation details. We random erasing as for pre-training. We fine-tune our networks with the same parameters as for pre-training, except the learning rate for which 800k iterations. For HTR English, French and German, we train character-level N-grams on the text of the training set of each dataset with the KenLM [23] library and assign the N-gram weight of 0.3 for each dataset. Otherwise, we use non-max suppression with an loU threshold of 0.4.
0.003 to compute joint letter probabilities, and the same
10-5 for 1200k iterations and then 10-6 for
4 Experiments
4.1 General text-line recognition
Optical Character Recognition We performed OCR the Google1000 dataset (22] which contains scanned historical printed books, using our pre-trained model. We use the English Volume 0002, with 5,097 training lines, 567 validation lines, and 630 testing lines. Prior fine-tuning on real data, our model has only 3.61% in Character Error Rate (CER) and this rate is further reduced to 2.04% after fine-tuning. Most of the "errors" actually due to mislabeled samples. An example is shown in Figure 4 In Appendix, we also show qualitative results on the ICDAR 2024 Competition on Multi Font Group Recognition (Figure 8). a dataset that includes multiple fonts and languages, showcasing our method's s ability to handle various languages and printing styles.
Handwritten Text Recognition on Latin Alphabets We evaluate our approach on text line HTR in various languages with latin script: IAM (English) (29), RIMES 39) (French), and READ 44
(Old German). For IAM, we follow common practice [13 25 19 and use the unofficial Aachen partition It includes 6, 161 training lines, 966 validation lines, and 2,915 testing lines. The READ 2016 dataset consists of Early Modern German handwritten pages from the Ratsprotokolle collection.
https://www http:///wwwooossl...c 1rt.ssorroro/5566///

---

It includes 8,367 training lines, 1,043 validation lines, and 1,140 test lines. The RIMES dataset is composed of administrative documents written in French. We use the RIMES-201 version which includes only text lines from the letters' body 4 The training set has 10, 1 188 lines, which we split into a training set (80%) and a validation set (20%). The test set includes 778 lines. To assess the quality of our prediction, we use the Character Error Rate (CER) as defined by 13] In Table [ we present the results of our experiments across all three datasets.
On IAM, our results are satisfactory but clearly lower than state-of-the-art. The main reason seems to be that the dataset includes ambiguous samples which favors methods incorporating a more complex language model trained on large scale data [25] 13). Similarly, on the READ dataset, our performance is reasonable but still lower than the state-ofthhe the art. This could be due to the challenges posed by the dataset: it has a small number of samples, difficult handwriting style, and an old German language. In [13, the authors initially trained their model on synthetic lines generated from the READ dataset, which helps the model to learn the language prior. On the RIMES dataset, our performance more competitive, and only outperformed by Diaz et al. [L5] which relies on large internal dataset for training. We observe that the simple addition of a character-level N- N-gram, trained solely on the training corpus, results in noticeable improvements for all datasets.
Handwritten Text Recognition on Chinese To showcase the versatility of our approach, we also evaluate our method on CASIA v2 [27] a benchmark for handwritten Chinese text-line recognition, using Accurate Rate (AR) and Correct Rate (CR) as defined in [56]. The alphabet for this database has 2,703 characters, significantly more than Latin script. The training set consists of 41,781 text lines, which we further divide into a new training set (80%) and validation set (20%) while the test set has 10,449 text lines. We report our results in Table Our method outperforms the current state of the art by Yu et al. [56] in both accurate and correct rate.
Ciphers We evaluate our approach on the Borg and Copiale cipher f using the line segmentation and splits outlined in 33. The Borg cipher dataset [ derived from a 17th-century manuscript, has 34 unique characters. The dataset is divided into training set of 195 lines, a validation set of 31 lines, and a test set of 273 lines. The Copiale cipher, from 7700-1660, has 99 symbols and includes 71 training lines, 156 validation lines, and 908 test lines. As reported in Table we achieve better results than the state of the art on both data sets. Figure in the appendix presents challenging examples from the Copiale Cipher, where our method performs well.
DTLR also ranked first in the ICDAR 2024 Competition on Handwriting Recognition of Historical Ciphers [ for the Borg and Ramanacoil ciphers. The competition featured five ciphers: a digit- based cipher, updated versions of the Borg and Copiale ciphers, enciphered documents from the Bibliothèque Nationale de France (BNF), and the Ramanacoil manuscript. The digit cipher uses 76 symbols, mostly digits, with various diacritics. The BNF cipher contains 37 unique graphical symbols. The Ramanacoil cipher uses 24 Latin-based symbols and special characters for seven key words. In Appendix, we report the competition results in Table S and show visual examples of our results on the Ramanacoil dataset in Figure 7
4.2 Ablation and analysis
Qualitative analysis Most of our results are visually meaningful, as can be seen in Figure [I In Figure we show the worst results for each dataset, which outlines the limitations of our model and the difficulty problems in the datasets. On the Google 1000 sample, the annotation is inaccurate since it is provided by an OCR model. On the IAM dataset, our model struggles to distinguish lowercase from uppercase letters in specific portion of the test where the test set letters are mostly uppercase, because the training set does not include such lines. On the RIMES sample, the input image is an example of a badly cropped line, recurring problem in this dataset. The model misses some of the letters in couvert and relies on a misleading visual cue for é instead of ê. On both READ and CASIA v2, the model misses a few characters with high variation in writing style such as the D in READ or digits in CASIA v2 which are sparse in the training set of the model. On
https: bstps://pages..ee
http::///aaooocccuubbbbbee wab. wwtp::uwwww...aaaaaaaaaaaaeao
html lat, 898
https://rrc.ccc.ubb 2/
8

---

Sit mihi fas audita loqui sit numine vestro
s
H 11000
STEIn 10
sit mihi das audita loqui 5 eit nutine veetro
GT:
Sit imilli fas aujita Loqus; sit nunine vestro
Wampec m0
DE DOE
GAE
GSU
PEEN
ME
StoUGG
EN
Nook
ONE
GE
pa hts
P: HYE SteuCCC DiAFIADEENTUS, I 10L1s ETHE Nook. GT: he ahrugged diffidently,
ollnE GRSs pakats One geta plenty
I like the work.
ubois J SO so erais étre cav ut
it a ponnaabilit
N* Dubeis. Je soubaitorain étre
raponsabilité responsabilité
GT: Duboia je souhalterais couvert
yar le Deuon
Te Declm Zaadamary
AB1 B1 I
201 Ins
alt.. 8. din abrüary on
P: das Dewn
P: Sa en caanas Zhaing GT: To Doun Laudamus 2üsing.
GT: aSt 8. diz. Febrüarý
It 89
18
GT:
100 100
t Cima 8 x
ycm
a cet 2oA +2ajs yon n 8 ibsbu G
c.cm by
8r
nojjrbézz de. the Antji aagjjeooppppsssssssssseese GT: oryymm uojrbbk/(le fuc. : thd Aotja + wwwmmippaaaaeeeeueeeeeee
Figure 4: Failure cases. For the line(s) with the highest error each dataset (Google 1000 [46). IAM (29), RIMES [39], READ 44), CASIA [27), and Copiale [T] we show, our detections, the predicted text (P) and the ground ground-truth text (GT). Best seen in color.
CASIA where the fine-tuning process is much longer than for other datasets, we observed that the bounding boxes degenerate while the Character Error Rate (CER) continues to improve. This can be understood by the fact that no specific loss is used to constrain bounding boxes, and that features have large receptive fields. Thus, in Figure [ we show the prediction of our model in middle of the training process, before the bounding boxes degenerate. On the Copiale dataset, the error comes from the labels, the last dozen characters are present in the image but not in the annotation.
Ablation We conduct ablation study in Table We first examine the impact of the pretraining dataset on IAM, which uses latin script. Using a model trained on English and random erasing both improve performances. We then study the effects of the finetuning strategy as well as the presence of random erasing during finetuning on the IAM and Borg datasets.
Freezing the network and learning only the character classification layer significant boost perfor- mances on IAM, and results a model that is better than random on Borg, but both still remain low quality, and end-to-end fine-tuning is necessary to obtain good performances. The results highlights the significant benefit of the random crasing strategy during finetuning, hinting that it helps our method to learn implicit language model.
Inference speed We compare our inference speed TrOCr 25 and FasterDAN 14 for which public code is available, on text lines from the RIMES dataset 44 with a batch size of and using an A6000 GPU. Our method requires 67ms for inference, while TrOCR takes 284ms and FasterDAN takes 140ms. However, FasterDAN is designed to recognize efficiently the lines of an entire document in parallel. When evaluating on whole documents from RIMES and dividing the inference time by the average number of lines, one finds it requires only 7ms per line. While
use batch

---

Table 4: Ablation studies on the IAM and Borg dataset
Method
IAM CER (%) 4
Borg SER (%) 4
General model
39.72
English model w/o. erasing
37.20
English model w. erasing
36.70
Finetuning only class embedding w/o. erasing
20.48
49.87
6.93
13.1
erasing
5.99
8.5
size 8, our inference speed decreases only to 25 ms, but one could imagine further improvements by adapting it to entire pages.
Conclusion
We presented character detection approach text line recognition. Although it is conceptually different from most modern HTR approaches, we demonstrated that performs well on a wide diversity of benchmarks and achieves state-of-the-ard performance for Chinese scripts and ciphers recognition. We hope that our general approach will revive detection- approaches to text recognition and encourage the evaluation of future approaches on more diverse data.
6
Acknowledgments
This work was funded by ANR project EIDA ANR-22-CE38-00 ANR project VHS ANR-21- CE38-0008. ANR project sharp DISCOVER funded by the European Union's 's Horizon Europe Research and Innovation program under grant agreement No. 101076028, We thank Ségolène Albouy, Zeynep Sonat Baltaci, loannis Siglidis, Elliot Vincent and Malamatenia Vlachou for feedback and fruitful discussions.
11----3-PAA--ONE in the context of the PEPR IA. and ERC project
References
[1] Eugen Antal and Pavol Marák. Automated transcription of historical encrypted manuscripts. Tatra Mountains Mathematical Publications, 82(2))65-86, 2022.
[2] Nafiz Arica and Fatos Marman-Vural. Optical character recognition for cursive handwriting.
IEEE transactions pattern analysis and machine intelligence, 24(6):801-811 2002.
[3] Arnau Baró, Jialuo Chen, Alicia Fornés, and Beáta Megyesi. Towards generic unsupervised method for transcription encoded manuscripts. In Proceedings of the 3rd International Conference Digital Access Textual Cultural Heritage, pages 73-78, 2019.
[4] Yoshua Bengio, Yann LeCun, Craig Nohl, and Chris Burges. Lerec: A nn/hmm hybrid for on-line handwriting recognition. Neural computation, 7(6):1289-1300 1995.
[5] Théodore Bluche and Ronaldo Messina. Gated convolutional recurrent neural networks for multilingual handwriting recognition. In 2017 14th IAPR international conference on document analysis and recognition (ICDAR), volume 1, pages 646-651. IEEE, 2017.
[6] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213 229. Springer, 2020.
[7] Richard G Casey and Eric Lecolinet. A survey of methods and strategies in character seg- mentation. IEEE transactions on pattern analysis and machine intelligence, 18(7):690-706, 1996.
[8] Paulo Rodrigo Cavalin, Alceu de Souza Britto Jr, Flávio Bortolozzi, Robert Sabourin, and Luiz E Soares Oliveira. An implicit segmentation-basee method for recognition of handwritten
10

---

strings of characters. In Proceedings of the 2006 ACM symposium on Applied computing, pages 836- 866-800.22 2006.
[9] Kartik Chaudhary and Raghav Bali. Easter: Simplifying text recognition using only 1d convolu- tions, In Canadian Conference on AI, 2021.
[10] Kartik Chaudhary and Raghav Bali. Easter2. 0: Improving convolutional models for handwritten text recognition. arXiv preprint arXiv:2205. 14879,, 2022.
[11] Amit Choudhary. A review of various character segmentation techniques for cursive handwritten words recognition. International Journal of Information & Compulation Technology. 4(6): 559-564, 2014.
[12] Denis Coquenet, Clément Chatelain, and Thierry Paquet. End-to-end handwritten paragraph text recognition using a vertical attention network. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(1):508-525 2022.
[13] Denis Coquenet, Clément Chatelain, and Thierry Paquet. Dan: a segmentation-free document attention network for handwritten document recognition. IEEE transactions on pattern analysis and machine intelligence, 2023.
[14] Denis Coquenet, Clément Chatelain, and Thieny Paquet. Faster dan: Multi-target queries with document positional encoding for end-to-end handwritten document recognition. In International Conference on Document Analysis and Recognition, pages 182-199. Springer, 2023.
[15] Daniel Hernandez Diaz, Siyang Qin, Reeve Ingle, Yasuhisa Fujii, and Alessandro Bissacco. Rethinking text line recognition models. arXiv preprint (xXi:22004.07755 2021.
[16] El- Yacoubi, Michel Gilloux, Robert Sabourin, and Ching Y. Suen. An hmm-based approach for off-line unconstrained handwritten word modeling and recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(8)7552-760 1999.
[17] Shancheng Fang. Hongtao Xie, Yuxin Wang, Zhendong Mao, and Yongdong Zhang. Read like humans:
Autonomous, bidirectional and iterative language modeling for scene text recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7098-7107, 2021.
[18] Hiromichi Fujisawa, Yasuaki Nakano, and Kiyomichi Kurino. Segmentation methods for character recognition: from segmentation document structure analysis. Proceedings of the IEEE, 80(7): 1992.
[19] Masato Fujitake. Dtroer: Decoder-only transformer for optical character recognition. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pages 8025 8035, 2024.
[20] Alex Graves and Jürgen Schmidhuber. Framewise phoneme classification with bidirectional Istm and other neural network architectures. Neural networks, 18(5-6):602-610, 2005.
[21] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pages 369-376, 2006.
[22] Ankush Gupta, Andrea Vedaldi, and Andrew Zisserman. Learning to read by spelling: Towards unsupervised text recognition. In Proceedings of the 11th Indian Conference Computer Vision, Graphics and Image Processing, pages 1-10, 2018
[23] Kenneth Heafield. KenLM: Faster and smaller language model queries. In Proceedings of the Sixth Workshop Statistical Machine Translation, pages 187-197, Edinburgh, Scotland, July 2011. Association for Computational Linguistics. URL httph::////// :/wp:::///www.. aclweb. org/
anthology/Wi1--122
[24] Lei Kang, Pau Riba, Marçal Rusiñol, Alicia Fornés, and Mauricio Villegas. Pay attention to what you read: non-recurrent handwritten text-line recognition. Pattern Recognition, 129: 108766, 2022.
11

---

[25] Minghao Li, Tengchao Lv, Jingye Chen, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, and Furu Wei. Troer: Transformer-base based optical character recognition with pre-trained models. In Proceedings of the AAAI Conference on Artificial Intelligence, 2023.
[26] Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection, Proceedings of the IEEE international conference on computer vision, pages 2980-2988, 2017.
[27] Cheng-Lin Liu, Fei Yin, Da-Han Wang, and Qiu-Feng Wang. Online and offline handwritten chinese character recognition: benchmarking on new databases. Puttern Recognition, 46(1): 155-162, 2013.
(28) Pengyuan Lyu, Chengquan Zhang, Shanshan Liu, Meina Qiao, Yangliu Xu, Liang Wu, Kun Yao, Junyu Han, Errui Ding, and Jingdong Wang. Maskocr: Text recognition with masked encoder-decoder pretraining, arXiv preprint arXiv:2206.0000 00311,, 2022.
[29] U-V Marti and Horst Bunke. A full english sentence database for off-line handwriting recog- nition. In Proceedings of the Fifth International Conference on Document Analysis and Recognition, ICDAR 99 (Cat. No. PR00318), pages 705-708. IEEE, 1999.
[30] Johannes Michael, Roger Labahn, Tobias Grüning, and Jochen Zöllner. Evaluating sequence- to-sequence models for handwritten text recognition. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 1286-1292 IEEE, 2019.
[31] Magdi Mohamed and Paul Gader. Handwritten word recognition using segmentation-fiee hidden markov modeling and segmentation-bees based dynamic programming techniques. /EEE transactions pattern analysis and machine intelligence, 18(5):548-555) 1996.
[32] Tom Monnier and Mathieu Aubry. docExtractor: An off-the-shelf historical document element extraction. In ICFHR, 2020.
[33] Dezhi Peng, Lianwen Jin, Yaqiang Wu, Zhepeng Wang, and Mingxiang Cai. A fast and accurate fully convolutional network for end-to-end handwritten chinese text segmentation and recognition. In 20/9 International Conference on Document Analysis and Recognition (ICDAR), pages 25-30. IEEE, 2019.
[34] Dezhi Peng. Lianwen Jin, Weihong Ma, Canyu Xie, Hesuo Zhang, Shenggao Zhu, and Jing Li. Recognition of handwritten chinese text by segmentation: a segmenntannnoaation--ee approach. IEEE Transactions on Multimedia, 2022.
[35] Vu Pham, Théodore Bluche, Christopher Kermorvant, and Jérôme Louradour. Dropout improves recurrent neural networks for handwriting recognition. In 2014 14th international conference on frontiers in handwriting recognition, pages 285-290. IEEE, 2014.
[36] Joan Puigcerver. Are multidimensional recurrent layers really necessary for handwritten text recognition? In 2017 14th IAPR international conference on document analysis and recognition (ICDAR). volume 1, pages 67-72. IEEE, 2017.
[37] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 779-788, 2016
[38] Joan Andreu Sanchez, Verónica Romero, Alejandro H Toselli, and Enrique Vidal. lcfhr2016 competition on handwritten text recognition on the read dataset. In 2016 15th International conference on frontiers in handwriting recognition (ICFHR), pages 530-633. IEEE, 2016.
[39] Joan Andreu Sanchez, Verónica Romero, Alejandro H Toselli, Mauricio Villegas, and Enrique Vidal. lcdar2017 competition on handwritten text recognition on the read dataset. In 2017 14th IAPR international conference on document analysis and recognition (ICDAR), volume 1, pages
IEEE, 2017.
[40] Baoguang Shi, Xiang Bai, and Cong Yao. An end-to-end trainable neural network for image- based sequence recognition and its application to scene text recognition. IEEE transactions on pattern analysis and machine intelligence,
2016.
12

---

[41] Mohamed Ali Souibgui, Pau Torras, Jialuo Chen, and Alicia Fornés. An evaluation of hand- written text recognition methods for historical ciphered manuscripts. In Proceedings of the 7th International Workshop on Historical Document Imaging and Processing, pages 7-12, 2023.
[42] Solène Tarride and Christopher Kermorvant. Revisiting n-gram models: Their impact in modern neural networks for handwritten text recognition. arXiv preprint arXiv:2404. (TtE:::244..9311,,, 2024.
[43] Vasiliki Tassopoulou, George Retsinas, and Petros Maragos. Enhancing handwritten text recog- nition with n-gram sequence decomposition and multitask learning. In 2020 25th International Conference Pattern Recognition (ICPR), pages 10555-10560. IEEE, 2021.
[44] Alejandro H. Toselli, HTR Dataset ICFHR 2016. Zenodo, 2018.
[45] Brijesh Verma and Michael Blumenstein. Fusion of segmentation strategies for off-line cursive handwriting recognition. Pattern Recognition Technologies and Applications: Recent Advances, pages 1-16, 2008.
[46] Luc Vincent. Google book search: Document understanding on a massive scale. In Ninth International Conference on Document Analysis and Recognition (ICDAR 2007), volume pages 819-823, IEEE, 2007.
[47] Paul Voigtlaender, Patrick Doetsch, and Hermann Ney. Handwriting recognition with large multidimensional long short-term memory recurrent neural networks. In 2016 15th international conference on frontiers in handwriting recognition (ICFHR), pages 228-233. IEEE, 2016.
[48] Qiu-Feng Wang, Fei Yin, and Cheng-Lin Liu. Handwritten chinese text recognition by integrat- ing multiple contexts. IEEE transactions on pattern analysis and machine intelligence, 34(8): 1469-1481, 2011.
[49] Yuxin Wang, Hongtao Xie, Shancheng Fang, Jing Wang. Shenggao Zhu, and Yongdong Zhang. From two to one: A new scene text recognizer with visual language modeling network. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14194-
14203, 2021.
[50] Zhen-Xing Wang, Qiu-Feng Wang. Fei Yin, and Cheng-Lin Liu. Weakly supervised learning for over-segmentation based handwritten chinese text recognition. In 2020 17th International Conference on Frontiers in Handwriting Recognition (ICFHR), pages 157-162. IEEE, 2020.
[51] Christoph Wick, Jochen Zöllner, and Tobias Grüning. Transformer for handwritten text recogni- tion using bidirectional post-decoding. In International Conference on Document Analysis and Recognition, pages 11 112-126. Springer, 2021.
[52] Yi-Chao Wu, Fei Yin, and Cheng-Lin Liu. Improving handwritten chinese text recognition using neural network language models and convolutional neural network shape models. Pattern Recognition, 65:251-264. 2017.
[53] Fei Yin, Qiu-Feng Wang, Ya Zhang, and Cheng-Lin Liu. Icdar 2013 chinese handwriting recognition competition. In 20/3 12th international conference on document analysis and recognition, pages 1464-1470. IEEE, 2013.
[54] Xusen Yin, Nada Aldarrab, Beáta Megyesi, and Kevin Knight. Decipherment of historical manuscript images. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 85. IEEE, 2019.
[55] Mohamed Yousef, Khaled F Hussain, and Usama S Mohammed. Accurate, data-efficient, unconstrained text recognition with convolutional neural networks. Pattern Recognition, 108: 107482, 2020.
[56] Ming-Ming Yu, Heng Zhang, Fei Yin, and Che Liu. An approach for handwritten chinese text recognition unifying character segmentation and recognition. Pattern Recognition, page 1 110373, 2024.
[57] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung- Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint arXiv:2203.03603 2022.
13

---

[58] Xiang Zhang, Yongwen Su, Subarna Tripathi, and Zhuowen Tu. Text spotting transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9519-99 9528, 2022.
[59] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 13001-13008, 2020.
[60] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020.
[61] Matthias Zimmermann and Horst Bunke. Hidden markov model length optimization for handwriting recognition systems. In Proceedings Eighth International Workshop on Frontiers Handwriting Recognition, pages 369-374. IEEE, 2002.
14

---

Appendix
Table 5: Symbol Error Rate (SER, in %) on ICDAR 2024 of Historical Ciphers.
Competition on Handwriting Recognition
Method
Digits
Borg
Copiale
BNF
Ramanacoil
Organizer A
7.83
7.91
4.33
1.09
6.07
Organizer
11.91
7.42
4.69
0.89
6.25
S. Corbille
7.60
1.62
0.89
6.08
Jiaqianwen 9.25
7.10
3.22
Ours (DTLR)
11.88
6.76
2.73
1.73
5.61
abgsiill Wel dh EK
Ba Jel
bussic I.
BIS DET IS
TRE
BEMDEIS
dGn
BEER
S I
DE
ll, helay
266
RUIDS RRI TRRTE sollan
118
Figure 5: Challenging Data. Qualitative examples from the READ dataset with slightly slanted lines, degraded characters and translucent paper. Spaces are omitted for better visualization.
adp Alinalifcccc Less 1eix+hov- man yo yo
Zargg tot ud wtt. WCC21
nEw DNER oa8
papti hecbbyyrr bhu papcr aC purpp2+
m mnigh Adsman mov Priggi ci me
Figure 6: Challenging Data. Qualitative examples from the Copiale Cipher dataset with slightly slanted lines and translucent paper. Spaces are omitted better visualization.
Will Barebi pestted apds regeitee ggtu prottii (wor Beilie
worrh of ofte Aohal wounel aufle
Bordd prll dl Jalaage
ret wed eam upry
Figure 7: Challenging Data. Qualitative examples from the Cipher Ramanacoil dataset in the ICDAR2024 Competition on Handwriting Recognition of Historical Ciphers, where our model achieves perfect predictions.
Gen biejer Befaunt it, iu Bio
colummae inferibi Bn fifferunty Beli nosech Lattewaa
and Diun Der Bilt linguá verncculá baptizatum, revt- 888 tos Sel 8 did Bracn ire becdl soraloat mil Eoct
Beeld bea HEl B2
Sam. Aamfi multi gauifi fume ibi le ba Hils a HURI BM BB Hal 18 ba
Curiofirés & d'hiftoire naturelle de I'aca- 8 886 H 883
mB B
Bi Bil Mt Bull a a b 8 ineiiia
Di Minder mngerfirent bep fuungen Farren rufin Elnao sneet aee BOg
fansfrembl annd anbeBand jumor
Bis
and
BEER
1OB 8 88B8 ERESHEEE BESSE im
Figure 8: Multilingual Predictions Qualitative examples from the ICDAR 2024 Competition on Multi Font Group Recognition and OCR dataset, using single model. The first row shows two examples of lines with mixed languages (German and Latin) and fonts. Subsequent rows display multiple languages (French, Latin, German) and fonts (Antiqua, Bastarda, Fraktur, Schwabacher) from left to right, top to bottom.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We depict failure cases all datasets with explanation. Additional results shown in the appendix.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide explicit explanation or our implementation and the full code will be released publicly.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All details to reproduce the experiments are provided the paper. Datasets used are public, and the full code will be released publicly.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We describe and will provide the split used for each dataset.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: We explain which GPU is used and the speed inference.
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:///w//.aaaeeppp.oooo///aaaa
16

---

Answer: [Yes]
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Each dataset referenced correctly.
New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
17


Reviews:
[{"review_id": "MfRL0GoW2Z", "reviewer": "Reviewer_hupU", "summary": "This paper treats text line recognition as an object detection task and proposes a two-stage training approach based on DINO-DETR. In the first stage, synthetic data with bounding box information is used to predict the bounding boxes and categories of text, due to the absence of character-level annotations for text line datasets. In the second stage, real text line data is employed to sequentially fine-tune the classifier and the entire model. The proposed method achieves state-of-the-art performance in cipher text.", "questions": "All results in the paper were pre-trained on Latin text and fine-tuned on several datasets. What if we pre-trained and fine-tuned on data in the same language? For example, pre-training on synthetic Chinese text lines followed by fine-tuning on real Chinese datasets.", "limitations": "The proposed method performs well only on the cipher text recognition task.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper is well-written and easy to follow. \n- The evaluation is well done and compares against several strong baselines.", "weaknesses": "- Abalation study is easy. \n- Utilizes more engineering skills than methodological innovations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed review and positive feedback on the paper, in particular positive comments on the clarity of the paper and the thorough evaluation against strong baselines. We address the concerns and questions raised below.\n\n## Weakness\n### Abalation study is easy.\n\nWe evaluate in our ablation the impact of the pre-training dataset, the impact of erasing during pre-training and fine-tuning, and we demonstrate the importance of fine-tuning the complete network, including the bounding box prediction, despite the fact there is no supervision on bounding boxes during fine-tuning. We will happily add any other ablation that the reviewer thinks would clarify aspects of our method.\n\n### Utilizes more engineering skills than methodological innovations. \nWe respectfully disagree with this assessment. As explained in the common rebuttal, building an OCR/HTR method based on a transformer detection architecture requiered methodological innovations and not mere engineering, which we believe is the reason why it was never demonstrated. While the pre-training losses are the same as DINO, the fine-tuning with CTC for such an architecture is novel and requires (i) ordering the queries differently for each text line according to the bounding boxes, (ii) adapting the class probabilities produced by the DINO architecture (iii) adapting the CTC loss, which can be done by introducing additionnal blank tokens, as detailed in our methodology section.\n\nWe believe that it is due to these methodological contributions, and not heavy engineering, that our method is, to the best of our knowledge, the first method competitive on many datasets, performing jointly text recognition and character segmentation, and processing characters in parallel rather than in an autoregressive way.\n\n## Questions \n### All results in the paper were pre-trained on Latin text and fine-tuned on several datasets. What if we pre-trained and fine-tuned on data in the same language? For example, pre-training on synthetic Chinese text lines followed by fine-tuning on real Chinese datasets.\n\nWe used pre-trained models for each language (French, German, English) and one general model trained on random text for the cipher and Chinese datasets. We noticed that this information was missing from Section 3.1. We will include all the information in the camera-ready version. \nAs shown in Table 4 of the ablation study, pretraining the model and the target language results with a masking strategy results to better results on IAM. We will compare performance with generic and language specific pre-training for all datasets in the final version of the paper.\n\nRegarding Chinese, we have pre-trained a model on Chinese characters and are currently fine-tuning it on CASIA. We achieved an AR of 94.3 and a CR of 95.3, surpassing the performance reported in the paper by  2.1\\%. \n\n\n## Limitations\n### The proposed method performs well only on the cipher text recognition task.\n We refer the reviewer to the common rebuttal where we address this limitation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rZ6tXpmGGk", "reviewer": "Reviewer_9JMV", "summary": "This paper presents a novel detection-based approach to text line recognition for both printed (OCR) and handwritten text (HTR), covering Latin, Chinese and cipher characters. Traditional detection-based methods have been largely neglected in HTR due to the difficulty of reading characters separately and the high cost of character-level annotation. The authors propose a solution to these challenges through three main insights: (i) using synthetic pre-training with diverse data for character localisation across different scripts; (ii) using modern transformer-based detectors to handle multiple character instances simultaneously, using a masking strategy to ensure consistency; and (iii) fine-tuning a pre-trained detection model with approximate character localisation using line-level annotations on real data, even with different alphabets.", "questions": "- Error cases: Can you provide examples of challenging real-world examples, such as rotated, upside-down or slanted lines; blank lines; vertically merged lines; strikethrough text; translucent paper; and mixed printed/handwritten characters?\n- Pre-training: How long was the model trained, on what hardware (GPUs), and why was pre-training limited to 100k lines of text? Was this value determined experimentally?\n- Input image size: What was the input image size used for training and evaluation?\n- Computational cost: How do you explain the difference in computational cost between DINO-DETR (25ms/line) and FasterDAN (7ms/line)? Why were comparisons made using a batch size of 1? Please provide a more comprehensive comparison, including batch sizes, CPU vs. GPU performance, impact of language modelling, input image sizes, and model parameters.\n- Language model decoding: What is the impact of a language model on inference speed? Can character recognition/classification still be parallelized with a language model? Can you decode on GPU with a KenLM language model, and what size of N-gram language model do you use?\n- Model release: Will the model be released to the public?", "limitations": "- Conclusion: The conclusion lacks depth and provides no insight into future improvements to the method.\n- Paper and writing:\n    - Figure 2: The figure is unclear and needs to be made clearer.\n    - Table 1: The table is misplaced, it appears on page 6 but is referenced on page 7.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Originality: The method is highly original, proposing a detection/classification approach to character recognition that differs from most state-of-the-art methods, which typically rely on autoregressive decoding from images of lines or pages.\n- Synthetic training data: The authors propose a way to train models using fully synthetic data, eliminating the need for actual character-level annotations.\n- Model efficiency: The model is relatively small (~40M parameters), requiring only 100k synthetic line samples for training.\nError analysis: This approach allows for a better understanding of errors, distinguishing between detection and classification errors.\n- Computational cost: Characters can be independently predicted in parallel, reducing computational cost, especially without a language model.\n- Adaptability: The method can be easily adapted to any alphabet with minimal training data.\n- Code release: The authors are committed to releasing the code, facilitating further research and application.", "weaknesses": "- Lack of new technical contributions: Despite the originality of the approach, the paper lacks novel technical contributions. The model, training strategy, fine-tuning with CTC, data augmentation and synthetic data generation are not novel.\n- Performance on handwritten documents: The method is not competitive on Latin handwritten documents and is outperformed by existing methods on Chinese handwritten documents. It does outperform on cipher recognition, but this is a more specialised and less researched area.\n- Scope of evaluation: The framework is only evaluated on perfectly segmented lines of text, raising questions about its performance on full pages. Full page processing would require additional steps such as text line detection and reading order retrieval, which could impact performance.\n- Impact of line detection: There are concerns about how the quality of line detection would affect character detection and recognition, particularly in cases where vertical or horizontal lines merge.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thorough review. We are glad they found the paper highly original, efficient, and adaptable. We address the concerns and questions raised below.\n\n## Weakness\n### Lack of new technical contributions.\nWe respectfully disagree with this assessment, as detailed in the common rebuttal and the answer to R-hupU.\n\n### Performance on handwritten documents.\nWe respectfully disagree as answered in the common rebuttal. \n\n\n### Impact of line detection.\nWe agree that the method used to extract text lines has an impact on performance. However, evaluating on cropped text lines is the most common practice in OCR and HTR, rather than evaluating an entire pipeline. Very few papers  (e.g., DAN, FasterDAN) address both tasks.\n\nIn cases where the line detection quality is poor and multiple lines appear in one crop, our method can still learn to detect only the characters of the line of interest, as shown in Figure 4 of the rebuttal PDF.\n\n## Question\n### Error cases.\n\nIn the paper, our method has been evaluated on several challenging real-world datasets with such issues. In Table 1 of the paper, we report results on the READ and Copiale datasets, which include translucent paper and slanted lines, as shown in Figures 2 and 3 of the rebuttal PDF. IAM and RIMES datasets include strikethrough text, which we will include in an appendix.\n\nOur method should perform well on mixed printed and handwritten text, as it has been effective on both printed (Google 1000) and handwritten datasets (IAM, READ, RIMES, HWDB), and on datasets mixing multiple fonts and languages, as shown in Figure 5 of the rebuttal PDF.\n\nWe see \"rotated, upside-down\" text as out of the scope of the paper. We consider that correctly detecting the orientation and shape of such lines is the role of the line detector. Lines can be slightly slanted, as is the case in several of our datasets and the one reported in Figure 4 of the rebuttal pdf.\n\n### Pre-training.\n\nThe pre-training took approximately one week, conducted on an RTX A6000 with 32 GB of memory, involving 225k iterations with a batch size of 4, as detailed in Section 3.1. The generation of synthetic data is expensive, accounting for 20% of the total time. Finetuning lasted for two days (number of iterations are given in section 3.2).\n\nThe number of synthetic lines was not tuned, we simply generated a high number of lines. We believe that using a very large number of text lines during pre-training is not necessary since our method only learns a limited implicit language model. However, we will explore the impact of the synthetic training dataset size systematically in the final version (since this requires re-training several models, this was not doable during the time of the rebuttal).\n\n\n### Computational cost.\n\nAs explained in Section 4.2 we evaluated the inference speed of FasterDAN on whole documents, and the inference time was divided by the average number of lines. It is not a fair comparison for our method since our model does not work on whole documents while FasterDAN does. As explained in the common rebuttal, we aim to adapt our model for whole documents by using an additional model to guide attention, thereby reducing inference time like Faster DAN.\n\nWhen we launch both methods on individual lines, our method requires 67ms for inference, while FasterDAN takes 140ms.\n\n**Batch Size:** \nComparisons were made using a batch size of 1 to ensure a fair evaluation of inference time. Below, we provide a table comparing inference speeds in ms across different batch sizes:\n\n| Batch Size | 1 | 2 | 4 |  8 | 16 | 32 |\n|-|-|-|-|-|-|-| \n| Ours | 74 | 44 |34 |31 |28 |32 |\n| TrOCR | 271 | 208 | 111 |59 |49 |46 |\n| FasterDAN Document / average number of lines| 56 | 35 |21 |16 |16 |21 |\n| FasterDAN Lines | 140 | 74 |28 |16 |8 |7 |\n\n**Image Sizes:**\nWe agree that image size impacts speed. However, text lines are approximately the same size and are resized to a maximum width of 1330 pixels.\n\n**Model Parameters:**\nWe agree that model parameters impact both inference time and performance. But, analyzing these effects in detail would require extensive experimentation. Our focus was on demonstrating the effectiveness of our method in handling text recognition and character segmentation efficiently.\n\n### Language model decoding. \n\nWe employ the KenLM library and pytorch CTC decoder which do not support GPU acceleration. We have measured the inference speed on the RIMES dataset on a single CPU to be 960 ms per example. While it could be parallelized using CPU multiprocessing, this optimization is beyond the scope of our current work. Our aim was to illustrate that our predictions can be further refined through the use of a language model. One could perfectly employ a LLM that supports GPU decoding. \n\nThe N-gram model used in our experiments is of size 6, tuned on the validation set.\n### Model release: Will the model be released to the public?\n\nYes, as stated in the introduction, all models will be released publicly, including pre-trained and fine-tuned models.\n\n## Limitations\n###  The conclusion lacks depth and provides no insight into future improvements to the method.\nWe can extend the conclusion by adding the following future directions:\n\n1. **Text Recognition on whole documents**:  detailed in the common rebuttal.\n2. **Advanced Language Models**: Improving recognition by integrating more advanced language models, such as a GPT decoder.\n3. **Unsupervised Learning**: Implementing character segmentation with a reconstruction loss, allowing the model to learn the characters (alphabet) of a dataset and perform text recognition without any annotations, similary to [A].\n\n[A] Siglidis, Ioannis, et al. \"The Learnable Typewriter: A Generative Approach to Text Analysis.\" arXiv preprint arXiv:2302.01660 (2023).\n### Paper and writing\nAs addressed in our response to RjpUH, we will clarify Figure 2 and have corrected the placement of Table 1.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0aPfGF4w0y", "reviewer": "Reviewer_KLkw", "summary": "The paper introduces a novel transformer-based character detection for text line recognition. The authors use a diverse set of synthetic data to enable localization part of the detection network to generalize to unseen characters during training. The transformer-based detectors can identify all characters in a text line in parallel and a masking strategy has been adopted to encourage detection interactions. The methods proposed by the authors also includes a process to fine-tune the detection network using only line-level annotations.", "questions": "Q1: How does the model handle noisy datasets or texts with degraded quality?\nQ2: What were the computational resource requirements? Can this approach be extended to real-world problems in a different setting?\nQ3: How would LLMs affect or change the impact of this approach? Have the authors considered this aspect, and if the proposed method can do better than some existing LLMs?\nQ4: Can this approach be extended to multi-line text detection and recognition?", "limitations": "L1: The efficiency of the proposed method in real-world datasets with noises has not been fully explored.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "S1: The approach is novel for character detection for text line recognition using transformer-based models.\nS2: The authors demonstrate strong performance by the model across several datasets and outperformance for cipher recognition.\nS3: The model can generalize well to unseen characters and variations in text as the authors utilize synthetic data.", "weaknesses": "W1: Several typos throughout the paper. Ex: typo in the Fig 1 caption: “Our model is general can be”, \nW2: The authors can provide additional information on the fine-tuning process and the adaptability of the masking strategy. Did they explore other strategies?\nW3: The performance on real-world datasets that contain some-to-large amount(s) of noise has not been extensively discussed by the authors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thorough review and insightful questions. We appreciate their recognition of the originality of our method. Below, we address each of the reviewer's concerns and provide additional information to clarify our approach.\n\n## Weakness\n### W1: Several typos throughout the paper. Ex: typo in the Fig 1 caption: “Our model is general can be”,\nWe thank the reviewer for pointing out these issues. We have carefully reviewed the paper and corrected the typos.\n\n### W2: The authors can provide additional information on the fine-tuning process and the adaptability of the masking strategy. Did they explore other strategies?\n\nRegarding the masking strategy, we found empirically that combining both horizontal and vertical masking yielded the best results, similar to the approaches in [59, 10]. In addition, we used blur and random scaling as data augmentation strategies.\n\nFine-tuning details, including the number of iterations and learning rate, are provided at the end of Section 3.2. Additional details on pretraining and finetuning will be included in an appendix, and the code will be released publicly.\n\n### W3: The performance on real-world datasets that contain some-to-large amount(s) of noise has not been extensively discussed by the authors.\nWe respectfully disagree with this assessment, as addressed in the common rebuttal. We have provided additional visual examples of READ and Copiale in Figures 2 and 3 of the rebuttal PDF, illustrating the challenges posed by these datasets.\n\nTo further demonstrate the robustness of our model, we have included additional qualitative results for another cipher, Ramanacoil, in Figure 4 of the rebuttal PDF. Our model performs well on this dataset, that includes very slanted lines, leading to multiple lines being cropped jointly.\n\nWe will include more examples for each dataset in the appendix. This appendix will also include visual examples from five other ciphers from the ICDAR2024 Competition on Handwriting Recognition of Historical Ciphers, which we did not include in the\npaper.\n\n\n# Questions\n### Q1: How does the model handle noisy datasets or texts with degraded quality? Can this approach be extended to real-world problems in a different setting?\nAs explained in response to W3, our method has been evaluated on several challenging real-world datasets, and we show additional results on challenging data in the rebuttal pdf. The method has been successfully applied to various settings, including printed text (google1000), handwritten text(IAM, READ, RIMES), multiple languages, and noisy data (ciphers, RIMES). It outperforms the state-of-the-art in cipher recognition and remains competitive on historical data (READ) that contains degraded quality text.\n\n\n### Q2: What were the computational resource requirements?\nThe pre-training took approximately one week, conducted on an RTX A6000 with 32 GB of memory, involving 225k iterations with a batch size of 4, as detailed in Section 3.1. The generation of synthetic data is expensive, accounting for 20% of the total time. Fine-tuning lasted 2 days, with details provided in Section 3.2. Iterations are faster during fine-tuning since no synthetic data is generated and the CTC loss is not costly to compute.\n\n\n### Q3: How would LLMs affect or change the impact of this approach?\n1. Multimodal LLMs capable of OCR cannot fully replace this approach. They will struggle with complex documents such as handwritten historical documents (e.g., READ) or ciphers that include rare symbols.\n\n2. LLMs can complement our approach in several ways. For example, they can be used as external language models for post-processing to improve results, as demonstrated in [A]. However, this requires specific fine-tuning of the models. Indeed, we attempted to use ChatGPT 3.5 on the IAM dataset to refine predictions, but this approach was not successful due to the LLM hallucinating words, as the IAM dataset tends to split sentences and words. It is also possible to use an LLM to initialize part of the architecture, such as the decoder, as done in TrOCR. However, end-to-end training requires significant computational resources.\n\n[A] Thomas, Alan, Robert Gaizauskas, and Haiping Lu. \"Leveraging LLMs for Post-OCR Correction of Historical Newspapers.\" Proceedings of the Third Workshop on Language Technologies for Historical and Ancient Languages (LT4HALA)@ LREC-COLING-2024. 2024.\n\n### Q4: Can this approach be extended to multi-line text detection and recognition?\n\nWe appreciate the reviewer's question regarding the extensibility of our approach to multi-line text detection and recognition. As detailed in the common rebuttal, we have begun to implement a model similar to ours, based on DINO-DETR, which is capable of detecting lines (Figure 1 of the Rebuttal PDF) with accuracy close to state-of-the-art in baseline detection, as shown in Table 1 of the rebuttal PDF. We use this model to detect lines on IAM and run our text recognition model. Performance is reported in Table 2.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PhVLFABt7W", "reviewer": "Reviewer_jpUH", "summary": "This manuscript provides a new approach to line recognition of offline handwriting and printed documents with regard to multilingualism.\n\nThis method can recognize a line based on its simultaneously recognized features based on transformers.\n\nThis approach is interesting for document analysis.\n\nPromising results have been obtained for different datasets.", "questions": "When I compared your transformer architecture to the original [57], Figure 2 is different from the original. They use a query selection\n and matching. Why do not you have these parts? I can see them in your description, but not in your illustration.\n\nYou cite Figure 1 in subsection 4.2, why do you give the figure as Figure 1 without citing it earlier?\n\nWhat happens if you have a document with multiple languages? Do you test your method for the type of input? Or should you train your network for this type of data?\n\nWhy don't you create the synthetic data for other languages and use a fine-tuning model based on English? Have you considered the scenario?", "limitations": "They showed some limitations in Figure 4. But it needs to add more samples in an Appendix.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper is well written and organized. Since the method recognizes the characters simultaneously, this approach is efficient. This paper explores the impact of using a synthetic dataset to improve accuracy in practice.", "weaknesses": "This is not a whole system for the field of document analysis. This task is a part of document analysis that depends on line segmentation. This means that if the previous part (line segmentation) of the process has errors, these are transferred to the next steps. Therefore, this is an essential task for the work. What happens if you have a whole document for the task? What happens to your results if you use existing methods that segment lines? \n\nThis method is not generally suitable for multilingual contexts. If there are several characters in different languages, this paper has no clear solution. You should therefore have several models for each language.\n\nIt is better to explain the method for your synthetic data in more detail in the Appendix. It is better to clarify this with an illustration.\n\nPlease check your text again for some errors, e.g. line 263 (annotations annotations).\n\nIs line 230 correct? \"the English Volume 0002\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thorough review and insightful questions. We appreciate the comments on the organization and clarity of the paper, as well as the recognition of the efficiency of our method in recognizing characters simultaneously. We address the concerns and questions raised below.\n\n## Weaknesses\n### This is not a whole system for the field of document analysis. [...] What happens to your results if you use existing methods that segment lines?\n\nThe most common practice in OCR and HTR is to evaluate text recognition on cropped text lines, not the entire pipeline. Very few papers address both tasks (e.g., DAN, FasterDAN).\n\nHowever, as detailed in the common rebuttal, we have begun to implement a model similar to ours, based on DINO-DETR, which is capable of detecting lines (Figure 1 of rebuttal PDF) with accuracy close to state-of-the-art in baseline detection, as shown in Table 1 of the rebuttal PDF. We use this model to detect lines on IAM and run our text recognition model. Performance is reported in Table 2.\n\n\n\n### Please check your text again for some errors, e.g. line 263 (annotations annotations). Is line 230 correct? \"the English Volume 0002\"\nWe thank the reviewer for noticing these errors. We have reviewed the text and made the necessary corrections. \n\n## Questions \n### When I compared your transformer architecture to the original [57], Figure 2 is different from the original. They use a query selection and matching. Why do not you have these parts? I can see them in your description, but not in your illustration.\n Indeed, we use query selection at the end of the encoder as well as a denoising task for the decoder, similar to DINO-DETR. We initally removed these from Figure 2 for simplicity, but recognize that it can create confusion and will add them. We will correct the figure for the camera-ready version.\n\n### You cite Figure 1 in subsection 4.2, why do you give the figure as Figure 1 without citing it earlier?\nThank you for pointing this out. This Figure illustrates the capabilities of our method and the idea of our approach, we will refer to it explicitly in the introduction.\n\n### What happens if you have a document with multiple languages?.\nSince our method does not heavily rely on language modelling (especially our general model), it is possible to use or fine-tune it on datasets that encompass multiple languages. We tested training a single model on a mix of Latin, French and Germanic printed manuscripts with 8 different historical fonts, and it works effectively with a cer of 1.68 % (note that better performances can likely be obtained by learning different models for different fonts, since different letters can have similar shapes in different fonts). We provide visual examples of results in Figure 5 of the rebuttal PDF,  and we will include more in the appendix.\n\n\n## Limitation\n### They showed some limitations in Figure 4. But it needs to add more samples in an Appendix.\n\nWe will add an appendix to the paper with more examples of failure cases. We will also add more qualitative results similar to Figure 2, 3, 4 and 5 in the rebuttal pdf.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their thoughtful feedback. We are pleased that our approach was found to be highly original (R9JMV), novel (RKLkw), efficient (R9JMV, RjpUH), rigorously evaluated (RhupU, RKLkw), and well written (RhupU, RjpUH). We address in the following main concerns, raised by several reviewers. 

## Lack of Technical Novelty (RhupU, RKLkw)

We respectfully disagree with this assessment. Building an OCR/HTR method based on a transformer detection architecture, allowing for parallel processing of characters instead of the sequential processing typical in current methods [13, 15, 19, 25, 30, 38], requiered methodological innovations and not mere engineering, which we believe is the reason why it was never demonstrated. While the pre-training losses are the same as DINO-DETR, the fine-tuning with the CTC loss for such an architecture is novel and requires several ideas and contributions: (i) ordering the queries differently for each text line according to the bounding boxes, (ii) adapting the class probabilities produced by the DINO-DETR architecture, (iii) adapting the CTC loss, which can be done by introducing additionnal blank tokens, as detailed in our methodology section.

We believe that it is due to these methodological contributions, and not heavy engineering, that our method is, to the best of our knowledge, the first method competitive on many datasets, performing jointly text recognition and character segmentation, and processing characters in parallel rather than in an autoregressive way.

##  Lack of Evaluation on Challenging Real Datasets (R9JMV, RKLkw)

Our method has been evaluated on the historical READ dataset, which includes Germanic manuscripts and two ciphers. These datasets are characterized by noise, degradation, and can have slightly slanted lines as shown in Figure 2 (READ) and 3 (Copiale) of rebutal PDF. We have also included additional examples from two extra challenging datasets: the Ramancoil cipher (Figure 4) and a collection of early-modern prints from the 
ICDAR24 Competition on Multi Font Group Recognition and OCR (Figure 5). 

The Ramancoils cipher (year 1674) includes very slanted lines and the crops can include multiple lines, as shown in Figure 4 of the rebuttal PDF. Our model successfully recognizes the correct line even despite the presence of the upper and lower lines.

The early-modern prints dataset features eight different fonts and three languages (Old French, Old German, and Latin). We trained a single model capable of handling these diverse fonts and languages, as illustrated in Figure 2. The model also processes lines containing mixed fonts and languages (German and Latin) within the same line, as shown in Figure 5. Note that better performances can likely be obtained by learning different models for different fonts, since different letters can have similar shapes in different fonts.

## Competive results limited to Ciphers

We respectfully disagree. Our approach demonstrates strong performance across the datasets mentioned in the paper. While our results on the IAM dataset are below the state-of-the-art, it is important to note that leading methods like DTRoCR and TrOCR use significantly larger architectures than ours with up to 10 times more parameters, extensive computational resources (e.g., TrOCR used 32 V100 GPUs and a batch size of 1024), and much larger-scale training datasets (hundreds of millions of printed text lines for TrOCR). We believe that given comparable computational resources, similar performances could be achieved by our model.

Our method also provides a valuable contribution by efficiently addressing both text recognition and character segmentation, which is not achieved by any current method.

Following the suggestion of RhupU, we pre-trained a new model on synthetic lines of Chinese using characters from CASIA v2. We are currently fine-tuning this model on CASIA v2 and have achieved an AR of 94.3\% and a CR of 95.3\%, which is already  2.1\% above the performance reported in the paper, reducing the gap with state-of-the-art. Moreover, the training loss has not had time to converge during the rebuttal period, and better results can be expected after convergence.

##  Text Recognition on whole documents

We acknowledge the value of a system that can handle both text detection and recognition. However, the standard in text recognition typically involves evaluating crops of individual lines [19, 25, 30, 38]. Few methods address this combined task [12, 13].

Nevertheless, we provide proof of concept results of a model similar to ours, derived from DINO which performs text line detection in the rebuttal PDF. This method predicts 8 points for the baseline of the text and 2 for the line above it to segment the line, enabling both baseline evaluation on standard datasets and bounding-box extraction. It is pre-trained on synthetic data and fine-tuned on real data. The method achieves results close to the state of the art in baseline detection, as reported in Table 1 of the Rebuttal PDF. We also provide visual examples of line predictions on complex datasets (cBAD2019) and IAM in Figure 1 of the rebuttal PDF. Finally, we provide CER for combined line detection and recognition on IAM in Table 2.

This is a promising first step, but it could be improved by fully integrating it with our approach and training it end-to-end by using the line detections to guide the attention weights of our text recognition model. Recognition for several lines could then be performed in parallel, similar to FasterDAN. Similar to our model, pretraining could be performed in a completely supervised way, while fine-tuning could be done with only page level supervision. However, we believe that demonstrating a detection-based model for text lines is a significant enough contribution for this paper, and introducing a complete pipeline would dilute the contribution and limit the diversity of datasets on which the full approach can be evaluated.

Meta-review: The paper introduces a novel transformer-based handwritten character detection for text line recognition. The authors use a diverse set of synthetic data to enable localization part of the detection network to generalize to unseen characters during training. The transformer-based detectors can identify all characters in a text line in parallel and a masking strategy has been adopted to encourage interactions  between detections. The methods proposed by the authors also includes a process to fine-tune the detection network using only line-level annotations.

All reviewers support weak accept (6), some of them having increased their scores during the rebuttal.

Most reviewers agree that this paper is well written and supported by solid evaluations that compare well to the state-of-the art. While it revisits an earlier OCR paradigm, this approach is very original when compared to current mode holistic handwritten recognition approaches based on auto-regressive models. Compared to those, it seems much more efficient in terms of inference cost and also training cost. Synthetic data could make training on low resource languages more accessible.

The main weakness mentioned by 2 reviewers (9jmv, hupu) is the limited technical contribution. The author solid rebuttal, showing that they use the computer vision DINO-DETR model in a novel way in combination with CTC, caused one of the reviewers to increase his score.
Another limitation is the absence of real world applications, going from single lines to full documents. This would have allowed the authors to show how more cost effective their approach is.

This paper deserves a weak accept due to the limited technical contribution, but it could have a significant impact on the field given the originality and efficiency of the method
Decision: Accept (poster)