Title: General Detection-based Text Line Recognition
Venue: NeurIPS 2024
Accepted: yes
Keywords: Text Recognition

Handwritten Text Recognition

Optical Character Recognition

Transformer
Abstract: We introduce a general detection-based approach to text line recognition, be it printed (OCR) or handwritten text (HTR), with latin, chinese or ciphered characters. Detection-based approaches have until now largely been discarded for HTR because reading characters separately is often challenging, and character-level annotation is difficult and expensive. We overcome these challenges thanks to three main insights: (i) synthetic pre-training with diverse enough data to learn reasonable character localization in any script; (ii) modern transformer-based detectors can jointly detect a large number of instances and, if trained with an adequate masking strategy, leverage consistency between the different detections; (iii) once a pre-trained detection model with approximate character localization is available, it is possible to fine-tune it with line-level annotation on real data, even with a different alphabet.  Our approach thus builds on a completely different paradigm than most state-of-the-art methods, which rely on autoregressive decoding, predicting character values one by one, while we treat a complete line in parallel. Remarkably, our method demonstrates good performance on range of scripts, usually tackled with specialized approaches: latin script, chinese script, and ciphers, for which we significantly improve state-of-the-art performances. 
Our code and models are available at [https://github.com/raphael-baena/DTLR](https://github.com/raphael-baena/DTLR).

General Detection-based Text Line Recognition

Abstract

We introduce a general detection-based approach to text line recognition, be it printed (OCR) or handwritten (HTR), with Latin, Chinese, or ciphered characters. Detection-based approaches have until now been largely discarded for HTR because reading characters separately is often challenging, and character-level annotation is difficult and expensive. We overcome these challenges thanks to three main insights: (i) synthetic pre-training with sufficiently diverse data enables learning reasonable character localization for any script; (ii) modern transformer-based detectors can jointly detect a large number of instances, and, if trained with an adequate masking strategy, leverage consistency between the different detections; (iii) once a pre-trained detection model with approximate character localization is available, it is possible to fine-tune it with line-level annotation on real data, even with a different alphabet. Our approach, dubbed DTLR, builds on a completely different paradigm than state-of-the-art HTR methods, which rely on autoregressive decoding, predicting character values one by one, while we treat a complete line in parallel. Remarkably, we demonstrate good performance on a large range of scripts, usually tackled with specialized approaches. In particular, we improve state-of-the-art performances for Chinese script recognition on the CASIA v2 dataset, and for cipher recognition on the Borg and Copiale datasets. Our code and models are available at https://github.com/raphael-baena/DTLR .

1 Introduction

A popular approach in early Optical Character Recognition (OCR) was to localize individual characters before processing them independently [7]. Such an approach, while still used for specific cases such as Chinese script [56], cipher [41], or scene text recognition [58], has been largely discarded for Latin script Handwritten Text Recognition (HTR) in favor of implicit segmentation approaches. In this paper, we revisit character detection for text line recognition, demonstrate its effectiveness for HTR, and show results on a diverse range of datasets that we believe have not been demonstrated in prior work.

Designing a general detection-based approach for handwritten text recognition is challenging. Individual characters are often not well separated in handwritten texts, and they are not always readable independently of their context. Thus, most datasets provide only line-level annotations, without character bounding boxes. This comes in addition to the common challenges of text recognition - such as diversity in writers, very rare characters, different kinds of noise and degradation - and challenges unique to certain datasets, for which specific approaches have been designed, e.g., Chinese script which has a very large alphabet, or ciphers for which the amount of annotated data is limited. Most recent methods for HTR for Latin script rely on recurrent or autoregressive models, and very few are devoid of any recurrence [10]. However, we believe that a detection-based approach could be general, tackle HTR with any script, and provide potential advantages in terms of interpretability - since the position of each character is clearly identified, and errors can be explained by mis-localization or

county

and

esped

more

stvwe.

hawed

BE

country,

and

espied

more

strik

e

show

e

d

Brit

a

in

47

saluallin

W

a

P

R

a

t

h

ce

es

S

a

tations

5俱他类似的一些星体民中庭为

与其他类

似的

一

些星体统一定义为

22

+rg k

Ioc:$ hthr ap.cIvonr

uac.$h3c

mis-classification of ambiguous characters - as well as computational cost, since all characters can be processed in parallel.

We design such an approach, dubbed DTLR, by leveraging three critical insights. First, training with diverse and sufficiently challenging synthetic data enables the localization part of a detection network to generalize to characters unseen during training. Second, modern transformer-based detectors [6, 57] can detect all characters in a text line in parallel and allow for detections to interact with each other, especially if encouraged by an adapted masking strategy. Third, we introduce an approach to fine-tune a detection network end-to-end using only line-level annotations and demonstrate that it can be used even with a completely new alphabet.

Our contribution can be summarized as follows:

we present the first transformer-based character detection approach for text line recognition.

we demonstrate that our approach performs well on a wide range of datasets, including handwritten text in English, French, German, and Chinese text as well as ciphers.

we significantly improve state-of-the-art performances for cipher recognition.

Our code and models are available at: https://github.com/raphael-baena/DTLR .

2 Related Work

Text recognition is the process of transcribing the text depicted in an image using computer algorithms. In this paper we focus solely on text line recognition, where a document image cropped around a line is given as input, which is the most standard setting in document analysis. In this section, we first give an overview of approaches for text line recognition, separating works focusing on Latin script, on Chinese script, and on ciphers, for which specific methods have been developed. We then discuss briefly the use of language models for text line recognition.

Text line recognition for Latin script. In the 1990s many methods leveraged character segmentation for OCR [18, 4] and later in the 2000s for HTR [16, 45, 2]. They involved several steps: (i) character segmentation, (ii) character feature extraction, and (iii) character classification [7]. The final predictions were refined, for example using hidden Markov models [16, 2] or grouping characters into words and matching them with a lexicon [18, 31]. Because explicit segmentation in HTR poses significant challenges [11], implicit segmentation techniques were developed, first using hidden Markov models [61, 8], then the Connectionist Temporal Classification (CTC) loss [21].

Leveraging the CTC loss [21] became the dominant approach in the field. Indeed, it provides a solution to one of the important challenge in text line recognition, the disparity in the lengths and the misalignment between the predicted output sequences and the ground truth sequences. Hybrid system combining Convolutional Neural Network and Recurrent Neural Network [40] were

Multi-Scale Deformable Self

Attention

Feed-Foward Network

Values

Learnable Content

Queries

2D Positional

Encoding

Decoder Layer

Self Attention

Feed-Forward Network

Multi-Scale Deformable Self

Attention

CNN

Init Anchor

Query Selection

arguably the most successful, with many variations, for example using Long Short-Term Memory (LSTM) [20, 35, 5] or Multidimensional LSTM layers [47, 36]. Fewer studies used CTC with non-recurrent schemes [10, 55, 9], and are typically associated to lower performance [10].

More recently, still performing implicit character detection, the use of attention and transformers architecture enabled to complement [30] or replace [24, 13, 15, 14, 25, 19] the CTC loss with crossentropy, using an autoregressive decoder: given the beginning of a transcription, the decoder predicts the next character using cross-attention with the image features. Wick et al. [51] extended this concept by fusing the outputs of a forward-reading transformer decoder with a backward reading decoder. Autoregressive decoders can handle text at various levels of tokenization, such as characters [13] or sub-words [25, 19]. Predicting one token at a time seems computationally sub-optimal, but these methods currently achieve the state-of-the-art performances [19].

In contrast, our approach performs explicit character detection and is neither recurrent nor autoregressive. It revisits an early OCR paradigm with modern transformer-based detection architectures.

Text line recognition for Chinese script. Explicit character segmentation, although no longer prevalent for Latin alphabets, remains an important approach for other languages, particularly Chinese HTR [53, 52, 33, 56]. Inspired by the YOLO framework [37], Peng et al. [33] proposed a detection network for character bounding box and class prediction in text line recognition. Building on this work, Peng et al. [34] introduced a weakly supervised method to train the detection network, using synthetic data with character-level annotations for pre-training, and generating pseudo labels for weakly labeled real data. Closer to our approach Yu et al. [56] trained a detection model on a synthetic dataset in a fully supervised way and used CTC loss for examples from their target dataset with only line level annotations, CASIA [27]. Note that most of these methods pretrain on a very large synthetic dataset of Chinese characters, while we use a general pre-training and learn Chinese characters only on a limited real training set with line-level annotations.

Text line recognition for ciphers. Recently, text recognition techniques have been used for transcribing historical ciphers, which can use different scripts, esoteric symbols, or diacritics. It is particularly challenging because for each cipher the number of documents and annotations is very limited, the underlying language is often unknown and no language model can be applied to the predictions [41]. Souibgui et al. [41] compared different approaches from HTR, including LSTM and transformer-based methods. Other approaches tried to explicitly segment or detect the characters. For example, Yin et al. [54] segmented the symbol and clustered them with a Gaussian Mixture Model, while Baró et al. [3] used K-Means. Antal and Marák [1] treated the problem as a character detection task but required character-level annotations for training.

Language Model. Models that incorporate recurrent schemes, such as Handwritten Text Recognition (HTR) with RNN layers [40, 36] or autoregressive decoders [13, 25], learn an implicit language bias that is crucial in HTR. Some models leverage very large amount of data to learn this language

prior. For instance, TrOCR [25] used 684 million text lines, and DtrOCR [19] two billion to train the decoders. Instead, we complement smaller scale training with a masking strategy, similar to, e.g., Lyu et al. [28]. The masking strategy most similar to ours is used in Chaudhary and Bali [10], where it is interpret solely as a form of data augmentation. Language models can also be used after the predictions have been made. Complex models can be trained for denoising, or with a masking strategy, e.g., Fang et al. [17], Wang et al. [49]. We used instead a simple N-gram [15, 42, 43] approach, which we train only on the training dataset. For common languages, better results could be expected by training more complex language models at a much larger scale, but this is out of the scope of our study, where we instead aim at generality.

3 Method

Given an input text-line image, our goal is to predict its transcription, i.e., a sequence of characters. We tackle this problem as a character detection task and build on the DINO-DETR architecture [57], shown in Figure 2, to simultaneously detect all characters. The rest of this section is organized as follows. First, in Section 3.1, we discuss how we pretrain a character-detection model using synthetic data with character-level supervision. Then, in Section 3.2, we explain how we finetune our model over real images from the target dataset with only line-level supervision and make final predictions.

3.1 Synthetic pre-training

Data generation and masking strategy. To generate our synthetic data, we start by defining an alphabet A 0 and sampling sentences. We use two alphabets: one for Latin scripts and another for Chinese. The Latin alphabet contains 167 characters, including uppercase and lowercase Latin letters, whitespace, common symbols, and accented characters, while the Chinese alphabet includes 7,356 characters from the CASIA v1 dataset.

To simulate text line images, we build on the synthetic generation pipeline of Monnier and Aubry [32]. For the Latin alphabet, we sample a font from a set of publicly available fonts 1 , using a font labeled as "handwriting" with a 50% probability, and a random background from a set of empty page photographs. We use the font to render the text and blend it with the background by using a random color and adding structured noise to both the background and font layers. For the Chinese alphabet, we use the CASIA v1 dataset [27], which contains various handwritten Chinese characters from different writers, to create synthetic sentences. We also apply significant blur data augmentation. This process results in challenging samples, as shown in the left column of Figure 3.

We sample sentences based on the application scenario. For known Latin-script languages, we sample sentences from Wikipedia in the target language. For ciphers, we generate random sentences by uniformly sampling Latin characters. For Chinese, we create random character sequences, since the limited alphabet of the CASIA v1 dataset makes it difficult to sample real Chinese text. This approach results in five models, for English, German, French, Chinese, and ciphers.

In addition, we use random erasing, masking complete vertical blocks, and small horizontal blocks completely, similar to Zhong et al. [59], Chaudhary and Bali [10]. This results in the text lines shown in the right column of Figure 3. Masking increases the robustness of the model, but also has additional motivations. First, masking vertical blocks can make a character unreadable or even completely hide it. In cases where the text is sampled from natural language, our network is thus encouraged to learn an implicit language model to predict detections for unreadable or even non-visible characters. Since our training time is modest, this model is far simpler than Large Language Models, but we still found it significantly improved predictions.

Second, masking small horizontal blocks will typically not make a character unreadable, but will prevent the model from focusing only on a discriminative part of a character. We found this to be especially important during the fine-tuning stage, where no character bounding box annotation is available.

Architecture and loss Since character labels and bounding boxes are known in synthetic data, we can simply leverage a standard DINO-DETR architecture [57], depicted in Figure 2 and the associated loss. Multi-scale image features are extracted through a CNN backbone and are then further refined

BNZpdSEBe

G38AyPP

Fpes

Pathogens.

AUCHAUSDAUE

脾体翻

戒名爸预吃起入”

in the N e transformer encoder layers through deformable attention [60]. The transformer decoder is composed of N d decoder layers which are fed a set of Q character queries. Each character query q is the concatenation of a content query and a positional query, initialized by the output of the encoder. In each decoder layer, the character queries first interact through self-attention and then attend to the encoder image features via deformable cross-attention. The decoder outputs Q bounding boxes ˆ b q ∈ R 4 and for each one a probability vector ˆ c q ∈ R |A 0 | , where |A 0 | is the number of characters in the alphabet A 0 and q is the query index.

For each image, the loss compares these Q predictions to a set of N ground truth bounding boxes b n ∈ R 4 and their associated class label c n ∈ R |A| , where n is the index of the ground truth character. The box loss L box ( b n , ˆ b q ) measures the distance between a ground truth box b n and a predicted box ˆ b q and is defined as a weighted sum between the L 1 distance and the generalized intersection over union (GIoU) : L box ( b n , ˆ b q ) = λ 1 ∥ ∥ ∥ b n -ˆ b q ∥ ∥ ∥ 1 + λ iou GIoU ( b n , ˆ b q ) . The number of queries Q is typically much larger than the number of ground-truth characters N , so the set of ground truth is completed by Q -N objects associated to the "no object" class ∅ . The Hungarian algorithm is used to find a permutation ˆ σ between queries and ground truth minimizing a matching cost:

̸

where L cls is the focal loss [26] and λ cls and λ box are two scalar hyperparameters. Once this matching ˆ σ has been found, the network is trained to minimize the loss:

̸

where λ ′ cls and λ ′ box are two scalar hyperparameters.

Implementation details. We follow Zhang et al. [57], and uses N e = 6 encoder layers, N d = 6 decoder layers, Q = 900 queries, and as hyperparameters λ cls = 2 , λ box = 5 , λ ′ cls = 1 and λ ′ box = 5 . We generate synthetic datasets of 100k text lines, and train the networks for 225k iterations with batch size of 4, using the ADAM optimizer with β 1 = 0 . 9 , β 2 = 0 . 999 , a fixed learning rate of 10 -4 , and a weight decay of 10 -4 .

3.2 Finetuning with line-level annotations.

Target alphabet. The alphabet of the target dataset A , might be different from the pretraining alphabet, A 0 . In our experiments, this is for example the case for ciphers. In this case, we replace the last linear layer responsible for character class prediction, adapting its size to the size of the target alphabet, and initialize each line, corresponding to a new character prediction, with random line of the pre-trained weight matrix, which we empirically found to work better than completely random initialization. Note the layers responsible for the bounding box coordinates predictions, which good initialization is critical for our approach, are unchanged. We actually found that thanks to the large variety of our training fonts, the character bounding boxes where reasonably good even for unseen characters. To train the new class prediction layer without destroying the rest of the network parameters, we fine-tune our network in two steps. First, for short a number of iterations (20k), we only optimize the re-initialized parameters, freezing the rest of the network. Then, we fine-tune the network end-to-end, thus also optimizing bounding box predictions.

From detection to sequences of character probabilities. While most approaches to text recognition directly produce a sequence of character probabilities, our method outputs a set of detection without specific ordering and independent probabilities for all classes. For ordering the predictions, we simply sort them by the minimum x coordinates of the predicted boxes. While this could cause issues with slanted scripts, we did not see any in practice. For simplicity, we assume in the rest of the paper that the queries q are sorted with this order, which differs for each line.

The fact that our probability predictions are independent for each class, similar to DINO-DETR, makes it non-trivial to decide when to predict the "no object" class and makes it harder to compute CTC and combine our prediction with language models using standard tools such as Tarride and Kermorvant [42]. Given a predicted class probability ˆ c q ∈ R |A| , we thus define a new probability vector ¯ c q ∈ R |A| +1 by:

where ϵ is a scalar hyper-parameter, we use upper indices to refer to vector coordinates and the sums are between 1 and the size of the alphabet |A| . ¯ c q can now be interpreted as a joint probability vector for all classes, and ¯ c |A| +1 q as the probability of the "no object" class.

Fine-tuning with adapted CTC. Unlike synthetic data, real-world datasets typically do not provide ground truth bounding boxes, but only line-level transcriptions. This makes it impossible to supervise character bounding boxes and to perform a Hungarian matching to associate tokens to characters. Thus, we instead adapt the CTC loss to fine-tune our network on real datasets. To do so, we leverage the sequence of character probabilities ¯ c defined in the previous paragraph. The standard CTC loss merges two successive identical characters, and requires them to be separated by a specific class, referred to as the blank symbol , to be separated. This is problematic for us since we might not predict empty bounding boxes between each character, and not necessary, since we detect each character independently and can directly use the bounding boxes to remove duplicate prediction using standard non-max suppression. We thus remove this behavior from the CTC loss. In practice, one can use standard implementations of the CTC loss and simply insert a blank symbol between each prediction.

Note that even with a novel alphabet, the network pre-training is critical, because if the predicted bounding boxes do not coarsely correspond to characters, training with the CTC is not enough for the network to learn meaningful character localization.

Predictions refinement. Once training is complete, the performance of the network can be slightly improved either by using non max suppression or by incorporating an explicit language model in the final decoding stage. We use the N-gram approach presented in [42] which combines N-gram probabilities with the CTC logits to include a prior on the likelihood of each text sequence. Similar to them, we find character-level models to be most suitable for our task as opposed to word-level models. However, unlike them, we found it beneficial to apply the N-gram word per word in each sentence, thereby making it similar to a spell-checking algorithm.

Implementation details. We use ϵ = 0 . 003 to compute joint letter probabilities, and the same random erasing as for pre-training. We fine-tune our networks with the same parameters as for pre-training, except the learning rate for which we use 10 -5 for 1200k iterations and then 10 -6 for 800k iterations. For HTR in English, French and German, we train character-level N -grams on the text of the training set of each dataset with the KenLM [23] library and assign the N-gram a weight of 0 . 3 for each dataset. Otherwise, we use non-max suppression with an IoU threshold of 0.4.

4 Experiments

4.1 General text-line recognition

Optical Character Recognition We performed OCR on the Google1000 dataset [22], which contains scanned historical printed books, using our pre-trained model. We use the English Volume 0002, with 5,097 training lines, 567 validation lines, and 630 testing lines. Prior to fine-tuning on real data, our model has only 3 . 61% in Character Error Rate (CER) and this rate is further reduced to 2 . 04% after fine-tuning. Most of the "errors" are actually due to mislabeled samples. An example is shown in Figure 4. In Appendix, we also show qualitative results on the ICDAR 2024 Competition on Multi Font Group Recognition (Figure 8), a dataset that includes multiple fonts and languages, showcasing our method's ability to handle various languages and printing styles.

Handwritten Text Recognition on Latin Alphabets We evaluate our approach on text line HTR in various languages with latin script: IAM (English) [29], RIMES [39] (French), and READ [44] (Old German). For IAM, we follow common practice [13, 25, 19] and use the unofficial Aachen partition 3 . It includes 6,161 training lines, 966 validation lines, and 2,915 testing lines. The READ 2016 dataset consists of Early Modern German handwritten pages from the Ratsprotokolle collection.

It includes 8,367 training lines, 1,043 validation lines, and 1,140 test lines. The RIMES dataset is composed of administrative documents written in French. We use the RIMES-2011 version which includes only text lines from the letters' body 4 . The training set has 10,188 lines, which we split into a training set (80%) and a validation set (20%). The test set includes 778 lines. To assess the quality of our prediction, we use the Character Error Rate (CER) as defined by [13]. In Table 1, we present the results of our experiments across all three datasets.

On IAM, our results are satisfactory but clearly lower than state-of-the-art. The main reason seems to be that the dataset includes ambiguous samples which favors methods incorporating a more complex language model trained on large scale data [25, 19, 15]. Similarly, on the READ dataset, our performance is reasonable but still lower than the state-of-the art. This could be due to the challenges posed by the dataset: it has a small number of samples, a difficult handwriting style, and an old German language. In [13], the authors initially trained their model on synthetic lines generated from the READ dataset, which helps the model to learn the language prior. On the RIMES dataset, our performance is more competitive, and only outperformed by Diaz et al. [15] which relies on a large internal dataset for training. We observe that the simple addition of a character-level N-gram, trained solely on the training corpus, results in noticeable improvements for all datasets.

Handwritten Text Recognition on Chinese To showcase the versatility of our approach, we also evaluate our method on CASIA v2 [27] a benchmark for handwritten Chinese text-line recognition, using Accurate Rate (AR) and Correct Rate (CR) as defined in [56]. The alphabet for this database has 2,703 characters, significantly more than Latin script. The training set consists of 41,781 text lines, which we further divide into a new training set (80%) and a validation set (20%) while the test set has 10,449 text lines. We report our results in Table 2. Our method outperforms the current state of the art by Yu et al. [56] in both accurate and correct rate.

Ciphers We evaluate our approach on the Borg and Copiale cipher 5 , using the line segmentation and splits outlined in [3]. The Borg cipher dataset 6 , derived from a 17th-century manuscript, has 34 unique characters. The dataset is divided into a training set of 195 lines, a validation set of 31 lines, and a test set of 273 lines. The Copiale cipher, from 1730-1760, has 99 symbols and includes 711 training lines, 156 validation lines, and 908 test lines. As reported in Table 3, we achieve better results than the state of the art on both data sets. Figure 7 in the appendix presents challenging examples from the Copiale Cipher, where our method performs well.

DTLR also ranked first in the ICDAR 2024 Competition on Handwriting Recognition of Historical Ciphers 7 for the Borg and Ramanacoil ciphers. The competition featured five ciphers: a digitbased cipher, updated versions of the Borg and Copiale ciphers, enciphered documents from the Bibliothèque Nationale de France (BNF), and the Ramanacoil manuscript. The digit cipher uses 76 symbols, mostly digits, with various diacritics. The BNF cipher contains 37 unique graphical symbols. The Ramanacoil cipher uses 24 Latin-based symbols and special characters for seven key words. In Appendix, we report the competition results in Table 5, and show visual examples of our results on the Ramanacoil dataset in Figure 7.

4.2 Ablation and analysis

Qualitative analysis Most of our results are visually meaningful, as can be seen in Figure 1. In Figure 4, we show the worst results for each dataset, which outlines the limitations of our model and the difficulty or problems in the datasets. On the Google1000 sample, the annotation is inaccurate since it is provided by an OCR model. On the IAM dataset, our model struggles to distinguish lowercase from uppercase letters in a specific portion of the test set where the test set letters are mostly uppercase, because the training set does not include such lines. On the RIMES sample, the input image is an example of a badly cropped line, a recurring problem in this dataset. The model misses some of the letters in couvert and relies on a misleading visual cue for é instead of ê . On both READ and CASIA v2, the model misses a few characters with high variation in writing style such as the D in READ or digits in CASIA v2 which are sparse in the training set of the model. On

P: " Sit

mihi fas audita loqui ;

sit numine vestro '

GT: " Sit

inilii fas aujita loqui;

sit numine vestro "

P: HME StouCGC

DifFIiDEENTUs, ' I lLLli

ETHE Nook.

oNnE GESs

paNnts

GT: he shrugged diffidently, " I like

the work.

One gets

plenty

P: N°

Dubois.

Je

souhaiterais étre cavut as

je souhaiterais être couvert au titre de la responsabilité

P: a¯ uf.

8.

din ebr¯ uar¯ y on

GT: a¯ uf .8.

diz.

Febr¯ uar¯ y an

P: das am

GT: Das Te

Deum

P: Se em

oaamas Z¯ using

GT: Te Deum

Laudamus Z¯ using.

P:

18

法

国

传

士

在

花

重

部

首

次

年

到

崖

座

柏本

，

此

后

的

GT:

1892

年

法

国

传

士

在

重

庆市

首

次

采

集

到

崖

柏

标

本

。

此

后

100

P:

z

˙

rγc.m

:

uo

¨

yrb

˙

xz

|

ˆ

ac.

:

†

˙

n

ˆ

ι

:

λc

ˆ

ιj

ˆ

i

+

z

ˆ

ajsγ

ˆ

oηn

ˆ

ip

˙

s

ˆ

aιbas

Y

usδ

g

\

ua

|

Λ

c.ιmbasγi

|

g

˙

neq

˙

x

ˆ

or.

GT:

ˆ

o

˙

rγ

ˆ

im

:

uo

¨

yrb

˙

xz

|

ˆ

ac.

:

†

˙

n

ˆ

ι

:

λc

ˆ

ιj

ˆ

i

+

z

ˆ

ajsγ

ˆ

oηn

ˆ

ip

˙

s

ˆ

aιbas

usδ

g

ua

ves

S

m

S

V

e

口

迎园

GEU

MES

StouCG

DifFiDENT US,

L

ETH

Nook.

onE

GES

pa

nts

N°Dubois.]e souhaiteraisétre

cavut

se

gae leDeun

TeDerimLaadamde

!pl8lne

Ie bru ar y on

d

a

S

am

S

bulsnzseweeo

18

法国

住

与花重

部首次年

本

，

此后的

onxxEr

c.um basilg r ×

or.

Y

\

CASIA v2 where the fine-tuning process is much longer than for other datasets, we observed that the bounding boxes degenerate while the Character Error Rate (CER) continues to improve. This can be understood by the fact that no specific loss is used to constrain bounding boxes, and that features have large receptive fields. Thus, in Figure 1, we show the prediction of our model in middle of the training process, before the bounding boxes degenerate. On the Copiale dataset, the error comes from the labels, the last dozen characters are present in the image but not in the annotation.

Ablation We conduct an ablation study in Table 4. We first examine the impact of the pretraining dataset on IAM, which uses latin script. Using a model trained on English and random erasing both improve performances. We then study the effects of the finetuning strategy as well as the presence of random erasing during finetuning on the IAM and Borg datasets.

Freezing the network and learning only the character classification layer significant boost performances on IAM, and results in a model that is better than random on Borg, but both still remain low quality, and end-to-end fine-tuning is necessary to obtain good performances. The results highlights the significant benefit of the random erasing strategy during finetuning, hinting that it helps our method to learn an implicit language model.

Inference speed We compare our inference speed to TrOCr [25] and FasterDAN [14] for which public code is available, on text lines from the RIMES dataset [44] with a batch size of 1 and using an A6000 GPU. Our method requires 67 ms for inference, while TrOCR takes 284 ms and FasterDAN takes 140 ms. However, FasterDAN is designed to recognize efficiently the lines of an entire document in parallel. When evaluating it on whole documents from RIMES and dividing the inference time by the average number of lines, one finds it requires only 7 ms per line. While we can use a batch

size 8 , our inference speed decreases only to 25 ms, but one could imagine further improvements by adapting it to entire pages.

5 Conclusion

We presented a character detection approach to text line recognition. Although it is conceptually different from most modern HTR approaches, we demonstrated that it performs well on a wide diversity of benchmarks and achieves state-of-the-art performance for Chinese scripts and ciphers recognition. We hope that our general approach will revive detection-based approaches to text recognition and encourage the evaluation of future approaches on more diverse data.

6 Acknowledgments

This work was funded by ANR project EIDA ANR-22-CE38-0014, ANR project VHS ANR-21CE38-0008, ANR project sharp ANR-23-PEIA-0008, in the context of the PEPR IA, and ERC project DISCOVER funded by the European Union's Horizon Europe Research and Innovation program under grant agreement No. 101076028. We thank Ségolène Albouy, Zeynep Sonat Baltaci, Ioannis Siglidis, Elliot Vincent and Malamatenia Vlachou for feedback and fruitful discussions.

References

Eugen Antal and Pavol Marák. Automated transcription of historical encrypted manuscripts. Tatra Mountains Mathematical Publications , 82(2):65-86, 2022.

Nafiz Arica and Fatos T Yarman-Vural. Optical character recognition for cursive handwriting. IEEE transactions on pattern analysis and machine intelligence , 24(6):801-813, 2002.

Arnau Baró, Jialuo Chen, Alicia Fornés, and Beáta Megyesi. Towards a generic unsupervised method for transcription of encoded manuscripts. In Proceedings of the 3rd International Conference on Digital Access to Textual Cultural Heritage , pages 73-78, 2019.

Yoshua Bengio, Yann LeCun, Craig Nohl, and Chris Burges. Lerec: A nn/hmm hybrid for on-line handwriting recognition. Neural computation , 7(6):1289-1303, 1995.

Théodore Bluche and Ronaldo Messina. Gated convolutional recurrent neural networks for multilingual handwriting recognition. In 2017 14th IAPR international conference on document analysis and recognition (ICDAR) , volume 1, pages 646-651. IEEE, 2017.

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision , pages 213-229. Springer, 2020.

Richard G Casey and Eric Lecolinet. A survey of methods and strategies in character segmentation. IEEE transactions on pattern analysis and machine intelligence , 18(7):690-706, 1996.

Paulo Rodrigo Cavalin, Alceu de Souza Britto Jr, Flávio Bortolozzi, Robert Sabourin, and Luiz E Soares Oliveira. An implicit segmentation-based method for recognition of handwritten

strings of characters. In Proceedings of the 2006 ACM symposium on Applied computing , pages 836-840, 2006.

Kartik Chaudhary and Raghav Bali. Easter: Simplifying text recognition using only 1d convolutions. In Canadian Conference on AI , 2021.

Kartik Chaudhary and Raghav Bali. Easter2. 0: Improving convolutional models for handwritten text recognition. arXiv preprint arXiv:2205.14879 , 2022.

Amit Choudhary. A review of various character segmentation techniques for cursive handwritten words recognition. International Journal of Information & Computation Technology , 4(6): 559-564, 2014.

Denis Coquenet, Clément Chatelain, and Thierry Paquet. End-to-end handwritten paragraph text recognition using a vertical attention network. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(1):508-524, 2022.

Denis Coquenet, Clément Chatelain, and Thierry Paquet. Dan: a segmentation-free document attention network for handwritten document recognition. IEEE transactions on pattern analysis and machine intelligence , 2023.

Denis Coquenet, Clément Chatelain, and Thierry Paquet. Faster dan: Multi-target queries with document positional encoding for end-to-end handwritten document recognition. In International Conference on Document Analysis and Recognition , pages 182-199. Springer, 2023.

Daniel Hernandez Diaz, Siyang Qin, Reeve Ingle, Yasuhisa Fujii, and Alessandro Bissacco. Rethinking text line recognition models. arXiv preprint arXiv:2104.07787 , 2021.

A El-Yacoubi, Michel Gilloux, Robert Sabourin, and Ching Y. Suen. An hmm-based approach for off-line unconstrained handwritten word modeling and recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence , 21(8):752-760, 1999.

Shancheng Fang, Hongtao Xie, Yuxin Wang, Zhendong Mao, and Yongdong Zhang. Read like humans: Autonomous, bidirectional and iterative language modeling for scene text recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7098-7107, 2021.

Hiromichi Fujisawa, Yasuaki Nakano, and Kiyomichi Kurino. Segmentation methods for character recognition: from segmentation to document structure analysis. Proceedings of the IEEE , 80(7):1079-1092, 1992.

Masato Fujitake. Dtrocr: Decoder-only transformer for optical character recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 8025-8035, 2024.

Alex Graves and Jürgen Schmidhuber. Framewise phoneme classification with bidirectional lstm and other neural network architectures. Neural networks , 18(5-6):602-610, 2005.

Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning , pages 369-376, 2006.

Ankush Gupta, Andrea Vedaldi, and Andrew Zisserman. Learning to read by spelling: Towards unsupervised text recognition. In Proceedings of the 11th Indian Conference on Computer Vision, Graphics and Image Processing , pages 1-10, 2018.

Kenneth Heafield. KenLM: Faster and smaller language model queries. In Proceedings of the Sixth Workshop on Statistical Machine Translation , pages 187-197, Edinburgh, Scotland, July 2011. Association for Computational Linguistics. URL https://www.aclweb.org/ anthology/W11-2123 .

Lei Kang, Pau Riba, Marçal Rusiñol, Alicia Fornés, and Mauricio Villegas. Pay attention to what you read: non-recurrent handwritten text-line recognition. Pattern Recognition , 129: 108766, 2022.

Minghao Li, Tengchao Lv, Jingye Chen, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, and Furu Wei. Trocr: Transformer-based optical character recognition with pre-trained models. In Proceedings of the AAAI Conference on Artificial Intelligence , 2023.

Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision , pages 2980-2988, 2017.

Cheng-Lin Liu, Fei Yin, Da-Han Wang, and Qiu-Feng Wang. Online and offline handwritten chinese character recognition: benchmarking on new databases. Pattern Recognition , 46(1): 155-162, 2013.

Pengyuan Lyu, Chengquan Zhang, Shanshan Liu, Meina Qiao, Yangliu Xu, Liang Wu, Kun Yao, Junyu Han, Errui Ding, and Jingdong Wang. Maskocr: Text recognition with masked encoder-decoder pretraining. arXiv preprint arXiv:2206.00311 , 2022.

U-V Marti and Horst Bunke. A full english sentence database for off-line handwriting recognition. In Proceedings of the Fifth International Conference on Document Analysis and Recognition. ICDAR'99 (Cat. No. PR00318) , pages 705-708. IEEE, 1999.

Johannes Michael, Roger Labahn, Tobias Grüning, and Jochen Zöllner. Evaluating sequenceto-sequence models for handwritten text recognition. In 2019 International Conference on Document Analysis and Recognition (ICDAR) , pages 1286-1293. IEEE, 2019.

Magdi Mohamed and Paul Gader. Handwritten word recognition using segmentation-free hidden markov modeling and segmentation-based dynamic programming techniques. IEEE transactions on pattern analysis and machine intelligence , 18(5):548-554, 1996.

Tom Monnier and Mathieu Aubry. docExtractor: An off-the-shelf historical document element extraction. In ICFHR , 2020.

Dezhi Peng, Lianwen Jin, Yaqiang Wu, Zhepeng Wang, and Mingxiang Cai. A fast and accurate fully convolutional network for end-to-end handwritten chinese text segmentation and recognition. In 2019 International Conference on Document Analysis and Recognition (ICDAR) , pages 25-30. IEEE, 2019.

Dezhi Peng, Lianwen Jin, Weihong Ma, Canyu Xie, Hesuo Zhang, Shenggao Zhu, and Jing Li. Recognition of handwritten chinese text by segmentation: a segment-annotation-free approach. IEEE Transactions on Multimedia , 2022.

Vu Pham, Théodore Bluche, Christopher Kermorvant, and Jérôme Louradour. Dropout improves recurrent neural networks for handwriting recognition. In 2014 14th international conference on frontiers in handwriting recognition , pages 285-290. IEEE, 2014.

Joan Puigcerver. Are multidimensional recurrent layers really necessary for handwritten text recognition? In 2017 14th IAPR international conference on document analysis and recognition (ICDAR) , volume 1, pages 67-72. IEEE, 2017.

Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 779-788, 2016.

Joan Andreu Sanchez, Verónica Romero, Alejandro H Toselli, and Enrique Vidal. Icfhr2016 competition on handwritten text recognition on the read dataset. In 2016 15th International conference on frontiers in handwriting recognition (ICFHR) , pages 630-635. IEEE, 2016.

Joan Andreu Sanchez, Verónica Romero, Alejandro H Toselli, Mauricio Villegas, and Enrique Vidal. Icdar2017 competition on handwritten text recognition on the read dataset. In 2017 14th IAPR international conference on document analysis and recognition (ICDAR) , volume 1, pages 1383-1388. IEEE, 2017.

Baoguang Shi, Xiang Bai, and Cong Yao. An end-to-end trainable neural network for imagebased sequence recognition and its application to scene text recognition. IEEE transactions on pattern analysis and machine intelligence , 39(11):2298-2304, 2016.

Mohamed Ali Souibgui, Pau Torras, Jialuo Chen, and Alicia Fornés. An evaluation of handwritten text recognition methods for historical ciphered manuscripts. In Proceedings of the 7th International Workshop on Historical Document Imaging and Processing , pages 7-12, 2023.

Solène Tarride and Christopher Kermorvant. Revisiting n-gram models: Their impact in modern neural networks for handwritten text recognition. arXiv preprint arXiv:2404.19317 , 2024.

Vasiliki Tassopoulou, George Retsinas, and Petros Maragos. Enhancing handwritten text recognition with n-gram sequence decomposition and multitask learning. In 2020 25th International Conference on Pattern Recognition (ICPR) , pages 10555-10560. IEEE, 2021.

Alejandro H. Toselli. HTR Dataset ICFHR 2016. Zenodo, 2018.

Brijesh Verma and Michael Blumenstein. Fusion of segmentation strategies for off-line cursive handwriting recognition. Pattern Recognition Technologies and Applications: Recent Advances , pages 1-16, 2008.

Luc Vincent. Google book search: Document understanding on a massive scale. In Ninth International Conference on Document Analysis and Recognition (ICDAR 2007) , volume 2, pages 819-823. IEEE, 2007.

Paul Voigtlaender, Patrick Doetsch, and Hermann Ney. Handwriting recognition with large multidimensional long short-term memory recurrent neural networks. In 2016 15th international conference on frontiers in handwriting recognition (ICFHR) , pages 228-233. IEEE, 2016.

Qiu-Feng Wang, Fei Yin, and Cheng-Lin Liu. Handwritten chinese text recognition by integrating multiple contexts. IEEE transactions on pattern analysis and machine intelligence , 34(8): 1469-1481, 2011.

Yuxin Wang, Hongtao Xie, Shancheng Fang, Jing Wang, Shenggao Zhu, and Yongdong Zhang. From two to one: A new scene text recognizer with visual language modeling network. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1419414203, 2021.

Zhen-Xing Wang, Qiu-Feng Wang, Fei Yin, and Cheng-Lin Liu. Weakly supervised learning for over-segmentation based handwritten chinese text recognition. In 2020 17th International Conference on Frontiers in Handwriting Recognition (ICFHR) , pages 157-162. IEEE, 2020.

Christoph Wick, Jochen Zöllner, and Tobias Grüning. Transformer for handwritten text recognition using bidirectional post-decoding. In International Conference on Document Analysis and Recognition , pages 112-126. Springer, 2021.

Yi-Chao Wu, Fei Yin, and Cheng-Lin Liu. Improving handwritten chinese text recognition using neural network language models and convolutional neural network shape models. Pattern Recognition , 65:251-264, 2017.

Fei Yin, Qiu-Feng Wang, Xu-Yao Zhang, and Cheng-Lin Liu. Icdar 2013 chinese handwriting recognition competition. In 2013 12th international conference on document analysis and recognition , pages 1464-1470. IEEE, 2013.

Xusen Yin, Nada Aldarrab, Beáta Megyesi, and Kevin Knight. Decipherment of historical manuscript images. In 2019 International Conference on Document Analysis and Recognition (ICDAR) , pages 78-85. IEEE, 2019.

Mohamed Yousef, Khaled F Hussain, and Usama S Mohammed. Accurate, data-efficient, unconstrained text recognition with convolutional neural networks. Pattern Recognition , 108: 107482, 2020.

Ming-Ming Yu, Heng Zhang, Fei Yin, and Cheng-Lin Liu. An approach for handwritten chinese text recognition unifying character segmentation and recognition. Pattern Recognition , page 110373, 2024.

Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and HeungYeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint arXiv:2203.03605 , 2022.

Xiang Zhang, Yongwen Su, Subarna Tripathi, and Zhuowen Tu. Text spotting transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9519-9528, 2022.

Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 13001-13008, 2020.

Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159 , 2020.

Matthias Zimmermann and Horst Bunke. Hidden markov model length optimization for handwriting recognition systems. In Proceedings Eighth International Workshop on Frontiers in Handwriting Recognition , pages 369-374. IEEE, 2002.

Appendix

598

CieMemorial

A

bz

es

ste

n

AIS

a

be

r

ist

E

ic

h

N

e

m

0

a

d

e

n

he

P

e

ine

Ne

e

ba

e

n

b

Yg-en.

PLel

gen

G

r

itis

Kauff

g

t.

S

e

0

Z

ug

g

ais

S

A

e

S

0

he

S

A

一

n

hoc

h

HamgznuuglNATAaoziW]

49rxv+yagw

pHntfraxLesx+hp=nply

△

Wc.△

P

63gr1nhrf n.clk

V

es

≠+hetc.=neg

yohc入Zno3k

mngshrotarObmaciomgovhoinyH+A:grghiye/m

hlpc入

y

mrznob

n*g≠hr

mncc.Tio.ng3vhoHs

+入:grg

hx入eSTm

d+

7

四

cg

TTA

domDte

Belannt

columnao InCrilbi hulfumn

und pfun Der QBelt/ lingua

baptizatum,rev-

Chem dieer

zu

mnaeincr bl urderund

Pfuy

der

verr

nacula

luriofires

& dihiftoire

de

l'aca

DRml.

C

es

&

de

atmnee

Wada

m

N

a

m

m

a

D

re

h

Sarren mn

ganc

frembd onnd pnbeland

Buuori

Die

R

der

nz

er

Ttre

ent

beynun

en

en

6

e

中

n

Z

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We depict failure cases on all datasets with explanation. Additional results are shown in the appendix.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide explicit explanation or our implementation and the full code will be released publicly.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: All details to reproduce the experiments are provided in the paper. Datasets used are public, and the full code will be released publicly.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We describe and will provide the split used for each dataset.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We explain which GPU is used and the speed inference.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Each dataset is referenced correctly.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript provides a new approach to line recognition of offline handwriting and printed documents with regard to multilingualism.\n\nThis method can recognize a line based on its simultaneously recognized features based on transformers.\n\nThis approach is interesting for document analysis.\n\nPromising results have been obtained for different datasets.\n\nSTRENGTHS:\nThis paper is well written and organized. Since the method recognizes the characters simultaneously, this approach is efficient. This paper explores the impact of using a synthetic dataset to improve accuracy in practice.\n\nWEAKNESSES:\nThis is not a whole system for the field of document analysis. This task is a part of document analysis that depends on line segmentation. This means that if the previous part (line segmentation) of the process has errors, these are transferred to the next steps. Therefore, this is an essential task for the work. What happens if you have a whole document for the task? What happens to your results if you use existing methods that segment lines? \n\nThis method is not generally suitable for multilingual contexts. If there are several characters in different languages, this paper has no clear solution. You should therefore have several models for each language.\n\nIt is better to explain the method for your synthetic data in more detail in the Appendix. It is better to clarify this with an illustration.\n\nPlease check your text again for some errors, e.g. line 263 (annotations annotations).\n\nIs line 230 correct? \"the English Volume 0002\"\n\nQUESTIONS:\nWhen I compared your transformer architecture to the original [57], Figure 2 is different from the original. They use a query selection\n and matching. Why do not you have these parts? I can see them in your description, but not in your illustration.\n\nYou cite Figure 1 in subsection 4.2, why do you give the figure as Figure 1 without citing it earlier?\n\nWhat happens if you have a document with multiple languages? Do you test your method for the type of input? Or should you train your network for this type of data?\n\nWhy don't you create the synthetic data for other languages and use a fine-tuning model based on English? Have you considered the scenario?\n\nLIMITATIONS:\nThey showed some limitations in Figure 4. But it needs to add more samples in an Appendix.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel transformer-based character detection for text line recognition. The authors use a diverse set of synthetic data to enable localization part of the detection network to generalize to unseen characters during training. The transformer-based detectors can identify all characters in a text line in parallel and a masking strategy has been adopted to encourage detection interactions. The methods proposed by the authors also includes a process to fine-tune the detection network using only line-level annotations.\n\nSTRENGTHS:\nS1: The approach is novel for character detection for text line recognition using transformer-based models.\nS2: The authors demonstrate strong performance by the model across several datasets and outperformance for cipher recognition.\nS3: The model can generalize well to unseen characters and variations in text as the authors utilize synthetic data.\n\nWEAKNESSES:\nW1: Several typos throughout the paper. Ex: typo in the Fig 1 caption: “Our model is general can be”, \nW2: The authors can provide additional information on the fine-tuning process and the adaptability of the masking strategy. Did they explore other strategies?\nW3: The performance on real-world datasets that contain some-to-large amount(s) of noise has not been extensively discussed by the authors.\n\nQUESTIONS:\nQ1: How does the model handle noisy datasets or texts with degraded quality?\nQ2: What were the computational resource requirements? Can this approach be extended to real-world problems in a different setting?\nQ3: How would LLMs affect or change the impact of this approach? Have the authors considered this aspect, and if the proposed method can do better than some existing LLMs?\nQ4: Can this approach be extended to multi-line text detection and recognition?\n\nLIMITATIONS:\nL1: The efficiency of the proposed method in real-world datasets with noises has not been fully explored.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel detection-based approach to text line recognition for both printed (OCR) and handwritten text (HTR), covering Latin, Chinese and cipher characters. Traditional detection-based methods have been largely neglected in HTR due to the difficulty of reading characters separately and the high cost of character-level annotation. The authors propose a solution to these challenges through three main insights: (i) using synthetic pre-training with diverse data for character localisation across different scripts; (ii) using modern transformer-based detectors to handle multiple character instances simultaneously, using a masking strategy to ensure consistency; and (iii) fine-tuning a pre-trained detection model with approximate character localisation using line-level annotations on real data, even with different alphabets.\n\nSTRENGTHS:\n- Originality: The method is highly original, proposing a detection/classification approach to character recognition that differs from most state-of-the-art methods, which typically rely on autoregressive decoding from images of lines or pages.\n- Synthetic training data: The authors propose a way to train models using fully synthetic data, eliminating the need for actual character-level annotations.\n- Model efficiency: The model is relatively small (~40M parameters), requiring only 100k synthetic line samples for training.\nError analysis: This approach allows for a better understanding of errors, distinguishing between detection and classification errors.\n- Computational cost: Characters can be independently predicted in parallel, reducing computational cost, especially without a language model.\n- Adaptability: The method can be easily adapted to any alphabet with minimal training data.\n- Code release: The authors are committed to releasing the code, facilitating further research and application.\n\nWEAKNESSES:\n- Lack of new technical contributions: Despite the originality of the approach, the paper lacks novel technical contributions. The model, training strategy, fine-tuning with CTC, data augmentation and synthetic data generation are not novel.\n- Performance on handwritten documents: The method is not competitive on Latin handwritten documents and is outperformed by existing methods on Chinese handwritten documents. It does outperform on cipher recognition, but this is a more specialised and less researched area.\n- Scope of evaluation: The framework is only evaluated on perfectly segmented lines of text, raising questions about its performance on full pages. Full page processing would require additional steps such as text line detection and reading order retrieval, which could impact performance.\n- Impact of line detection: There are concerns about how the quality of line detection would affect character detection and recognition, particularly in cases where vertical or horizontal lines merge.\n\nQUESTIONS:\n- Error cases: Can you provide examples of challenging real-world examples, such as rotated, upside-down or slanted lines; blank lines; vertically merged lines; strikethrough text; translucent paper; and mixed printed/handwritten characters?\n- Pre-training: How long was the model trained, on what hardware (GPUs), and why was pre-training limited to 100k lines of text? Was this value determined experimentally?\n- Input image size: What was the input image size used for training and evaluation?\n- Computational cost: How do you explain the difference in computational cost between DINO-DETR (25ms/line) and FasterDAN (7ms/line)? Why were comparisons made using a batch size of 1? Please provide a more comprehensive comparison, including batch sizes, CPU vs. GPU performance, impact of language modelling, input image sizes, and model parameters.\n- Language model decoding: What is the impact of a language model on inference speed? Can character recognition/classification still be parallelized with a language model? Can you decode on GPU with a KenLM language model, and what size of N-gram language model do you use?\n- Model release: Will the model be released to the public?\n\nLIMITATIONS:\n- Conclusion: The conclusion lacks depth and provides no insight into future improvements to the method.\n- Paper and writing:\n    - Figure 2: The figure is unclear and needs to be made clearer.\n    - Table 1: The table is misplaced, it appears on page 6 but is referenced on page 7.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

comments: SUMMARY:
This paper treats text line recognition as an object detection task and proposes a two-stage training approach based on DINO-DETR. In the first stage, synthetic data with bounding box information is used to predict the bounding boxes and categories of text, due to the absence of character-level annotations for text line datasets. In the second stage, real text line data is employed to sequentially fine-tune the classifier and the entire model. The proposed method achieves state-of-the-art performance in cipher text.

STRENGTHS:
- This paper is well-written and easy to follow. 
- The evaluation is well done and compares against several strong baselines.

WEAKNESSES:
- Abalation study is easy. 
- Utilizes more engineering skills than methodological innovations.

QUESTIONS:
All results in the paper were pre-trained on Latin text and fine-tuned on several datasets. What if we pre-trained and fine-tuned on data in the same language? For example, pre-training on synthetic Chinese text lines followed by fine-tuning on real Chinese datasets.

LIMITATIONS:
The proposed method performs well only on the cipher text recognition task.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-17