SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$\mathbf{M}^{5}$ HisDoc: A Large-scale Multi-style Chinese Historical Document Analysis Benchmark
Yongxin Shi, Chongyu Liu, Dezhi Peng, Cheng Jian, Jiarong Huang, Lianwen Jin*
yongxin_shi@foxmail.com, liuchongyu1996@gmail.com
pengdzscut@foxmail.com, eechengjian@mail.scut.edu.cn,
jiarong_huang@outlook.com, eelwjin@scut.edu.cn
Recognizing and organizing text in correct reading order plays a crucial role in historical document analysis and preservation. While existing methods have shown promising performance, they often struggle with challenges such as diverse layouts, low image quality, style variations, and distortions. This is primarily due to the lack of consideration for these issues in the current benchmarks, which hinders the development and evaluation of historical document analysis and recognition (HDAR) methods in complex real-world scenarios. To address this gap, this paper introduces a complex multi-style Chinese historical document analysis benchmark, named  $\mathbf{M}^5$  HisDoc. The  $\mathbf{M}^5$  indicates five properties of style, i.e., Multiple layouts, Multiple document types, Multiple calligraphy styles, Multiple backgrounds, and Multiple challenges. The  $\mathbf{M}^5$  HisDoc dataset consists of two subsets,  $\mathbf{M}^5$  HisDoc-R (Regular) and  $\mathbf{M}^5$  HisDoc-H (Hard). The  $\mathbf{M}^5$  HisDoc-R subset comprises 4,000 historical document images. To ensure high-quality annotations, we meticulously perform manual annotation and triple-checking. To replicate real-world conditions for historical document analysis applications, we incorporate image rotation, distortion, and resolution reduction into  $\mathbf{M}^5$  HisDoc-R subset to form a new challenging subset named  $\mathbf{M}^5$  HisDoc-H, which contains the same number of images as  $\mathbf{M}^5$  HisDoc-R. The dataset exhibits diverse styles, significant scale variations, dense texts, and an extensive character set. We conduct benchmarking experiments on five tasks: text line detection, text line recognition, character detection, character recognition, and reading order prediction. We also conduct cross-validation with other benchmarks. Experimental results demonstrate that the  $\mathbf{M}^5$  HisDoc dataset can offer new challenges and great opportunities for future research in this field, thereby providing deep insights into the solution for HDAR. The dataset is available at https://github.com/HCIILAB/M5HisDoc.
Historical documents are invaluable carriers of human cultural heritage, containing important information about human history, culture, and literary arts. Therefore, the preservation and analysis of historical documents are considered an urgent research topic. As the digital version of historical documents is easier to disseminate and retrieve, designing a digitization system [1] of historical documents is an effective way to make use of historical documents without causing irreversible damage. With the rapid development of Optical Character Recognition (OCR) technologies, significant progress [2, 3, 4] has been made in understanding historical documents, and some benchmarks [5, 6, 7] have been established. However, real-world scenarios present various challenges such as diverse page layouts,
(e)
Figure 1: An overview of the proposed M<sup>5</sup>HisDoc dataset. For better visibility, please zoom in on the image. (a) Multiple layouts. (b) Multiple document types. (c) Multiple calligraphy styles. (d) Multiple backgrounds. (e) Multiple challenges, including dense texts, distortion, rotation, damage, image blurriness, and variations in font sizes.
poor image quality, multiple font styles, and severe distortion, which are rarely considered in the existing benchmarks. Although related methods have reported promising performance, they do not perform well on the aforementioned issues.
To fill this gap and advance the development of historical document analysis in real-world scenarios, we propose a complex multi-style Chinese historical document analysis benchmark, which contains 8,000 manually annotated images of historical documents. The characteristics of our dataset are as follows. (a) It features a wide range of styles, including multiple layouts (Fig. 1a), multiple document types (Fig. 1b), multiple calligraphy styles (Fig. 1c), multiple backgrounds (Fig. 1d), and multiple challenges (Fig. 1e). (b) The images in our dataset contain dense text, with an average of over 545 characters per image. (c) The dataset has a large character set of 16,151 categories, with zero-shot recognition scenario setting in the validation and test sets. (d) Both the annotations at the character-level and text-line-level are provided, including the text bounding box, text content, and the corresponding reading order. Therefore,  $\mathbf{M}^5$  HisDoc can be applied to a wide range of tasks, including text-line/character detection, recognition, and reading order prediction. (e) To better simulate real-world scenarios for historical document analysis, we perform random rotation, random distortion, and resolution transformations on the images of  $\mathbf{M}^5$  HisDoc-R, obtaining a more challenging subset,  $\mathbf{M}^5$  HisDoc-H.
Based on  $\mathbf{M}^5$  HisDoc, we benchmark several representative methods in the following tasks: text line detection, text line recognition, character detection, character recognition, and reading order prediction. We also conduct cross-validation with other benchmarks. The detailed experimental setup and results are presented in Sec. 4.
In conclusion, this paper presents two primary contributions. Firstly, we propose a complex multi-style Chinese historical document analysis benchmark, named  $\mathbf{M}^5$  HisDoc, which contains 8,000 images in multiple styles. The benchmark exhibits features of large scale, high text density, extensive character sets, and diverse styles. To the best of our knowledge, this is the first multi-style Chinese document analysis benchmark. Secondly, we conduct thorough baseline experiments using  $\mathbf{M}^5$  HisDoc, which demonstrate  $\mathbf{M}^5$  HisDoc presents novel challenges for historical document analysis. We strongly believe that  $\mathbf{M}^5$  HisDoc benchmark holds significant potential for future academic research endeavors.
2.1 Historical document benchmarks
The development of historical document analysis algorithms is closely linked to the progress of the corresponding benchmarks. ENP [8] contains 600 images of European historical newspapers, which are annotated with Unicode encoding of full text, layout information, type labels, and reading order.
DIVA-HisDB [9] provides 150 European medieval manuscripts labeled with text lines and layout information. HJDataset [10] is a large-scale dataset of Japanese historical documents, which contains more than 250,000 hierarchy layout elements of seven types. However, all of these datasets lack character-level annotations and do not specifically address Chinese historical documents. For Chinese historical documents, Xu et al. [7] established CASIA-AHCDB with more than 2.2 million text samples from 10,350 categories for character recognition in historical document images. MTHv2 [5] is a comprehensive dataset for Chinese historical documents, which can be used for text detection, text recognition, and reading order prediction. However, its document type is relatively homogeneous and contains only Buddhist scriptures. Similar to MTHv2, ICDAR 2019 HDRC-CHINESE [6] is a large structured dataset with only one document type, i.e., genealogy. Besides, SCUT-CAB [11] is a dataset of Chinese historical documents with complex layouts, but it only provides annotation for layout analysis without any information for text recognition.
2.2 Text detection
There are three main categories of text detection methods: regression-based, segmentation-based, and connected component-based. (a) Regression-based methods are inspired by object detection techniques such as Faster R-CNN [12] and SSD [13]. EAST [14] directly regressed text instances at pixel level and applied Non-Maximal Suppression (NMS) to obtain the final results. OBD [15] discretized blocks of unordered boxes and used Mask R-CNN [16] to address the inconsistent labeling issue of regression-based methods. MOST [17] proposed a text feature alignment module to dynamically adjust the receptive field and a position-aware NMS algorithm to further refine the detections. (b) Segmentation-based methods typically perform text detection by implicitly encoding text instances with pixel masks. PSENet [18] used a progressive scale-expansion algorithm on multiscale segmentation maps to locate the text accurately. PAN [19] improved PSENet by a learning pixel aggregation algorithm to reduce computation cost. FCENet [20] predicted the Fourier signal vector of text instances and then reconstructed the text contour by Inverse Fourier Transform. DBNet++ [21] proposed an efficient adaptive scale fusion module based on DBNet [22] to enhance the accuracy of text detection. (c) Connected component-based methods typically detect individual text parts or characters first, followed by a post-processing procedure that links or groups them together to generate the final results. TextSnake [23] employed centerline and ordered disk to represent text instance. Zhang et al. [24] proposed a novel graph-based fusion network to adaptively fuse detection boxes to generate more accurate and holistic object instances for multi-oriented text detection. Besides, there are some text line detection methods specifically designed for historical documents. Mechi et al. [25] proposed an adaptive U-Net architecture for low-cost text line segmentation. Ma et al. [5] built upon Faster R-CNN [12] with two additional branches for character prediction and layout analysis, respectively. Then each individual detected character will be grouped into different text lines. Mechi et al. [4] used topological structural analysis to extract complete text lines.
2.3 Text recognition
Text recognition can be divided into three categories: CTC-based, attention-based, and segmentation-based. (a) CTC-based methods typically use the CTC algorithm to directly decode feature sequences into target sequences. CRNN [26] treated scene text recognition as a sequence recognition task. It employed CNN and RNN to extract text features, which are then decoded with the CTC algorithm. Ma et al. [5] improved CRNN by replacing the Bidirectional-LSTM with Residual-LSTM. ZCTRN [27] used a distance-based CTC decoder to match visual features with class embedding for zero-shot Chinese text recognition. SVTR [28] introduced local and global mixing blocks for extracting stroke-like features and inter-character dependence. (b) Attention-based methods use attention mechanisms to decode the target string sequence. In order to exploit the dependencies in both directions, the recognition decoder in ASTER [29] was extended into a bidirectional one. Sheng et al. proposed a self-attention-based sequence-to-sequence model NRTR [30] that does not rely on recurrence. To mitigate attention drift, Robust Scanner [31] incorporated a novel position enhancement branch and dynamically fused its outputs with those of the decoder attention module. Zhang et al. [32] proposed a cascade position attention mechanism to effectively handle the linguistic insensitive drift problem. (c) Segmentation-based methods first detect and recognize individual characters, then perform character combination with a post-processing algorithm, Liao et al. [33] and Peng et al. [34] are two typical methods. Liao et al. proposed a character attention FCN (Fully Convolutional Network) to predict the characters at pixel level, and then used a word formation module to group and arrange the pixels
to form the final word result. Considering that character-level annotation is insufficient, Peng et al. [34] proposed a weakly supervised learning paradigm, so that it can be trained with only transcript annotations.
2.4 Reading order prediction
The reading order prediction aims to arrange the text within a document in the correct order for reading. LayoutReader [35] is a sequence-to-sequence model using both textual and layout information, where it leverages the layout-aware language model LayoutLM [36] as encoder and modify the generation step in the encoder-decoder structure to generate the reading order sequence. Gu et al. [37] proposed a heuristic algorithm, Augmented XY Cut, to sort the input tokens. These methods primarily focus on modern documents. To the best of our knowledge, there is no work specializes for the reading order prediction of historical documents.
3 M5HisDoc benchmark
3.1 Data collection
The objective of  $\mathbf{M}^5$  HisDoc is to create an extensive benchmark for real-world scenarios in Chinese historical document analysis. To achieve this, we make significant efforts to gather a diverse collection of historical document images. Our data collection process consists of three main sources. Firstly, we carefully select 300 images from the training set of MTHv2 [5] and 700 images from SCUTCAB [11], under the criteria that the selected images possess certain characteristics (such as dense text or complex layout). Secondly, we gather tens of thousands of scanned images from electronic ancient books available on the Internet (such as Harvard Library, and National Archives of Japan), encompassing 131 ancient books. From this collection, we manually curate 2,799 historical document images taken from some representative books. Thirdly, we conduct realistic photo shoots to simulate photographing situations. By selecting four physical Chinese ancient books, we capture 201 images using a scanner, considering various scanning angles and lighting conditions. In total, we obtain 4,000 images to establish the  $\mathbf{M}^5$  HisDoc benchmark. The final dataset includes historical documents spanning different periods, such as the Han (206 BCE - 220 CE), Tang (618 - 907 CE), Song (960 - 1279 CE), Ming (1368 - 1644 CE), and Qing (1644 - 1912 CE) dynasties.
3.2 Data annotation
Initially, we train a character detector [12] and a character recognizer [38] using MTHv2 [5] and CASIA-AHCDB [7], respectively. We then utilize them for preliminary annotation. Afterward, we meticulously carry out the following procedures: (1) Adjust inaccurate character bounding boxes. (2) Proofread the character recognition results, which involve two rounds of checking. In the first round, two annotators proofread the character results on each image and point out the uncertain characters. In the second round, a domain expert finally corrects the inconsistencies between the two annotation employees in the first round and the uncertain characters. (3) Determine which characters belong to the same text line. (4) Sort the text lines in the correct reading order. To ensure high-quality annotations, we implement a strict three-round check after the annotating process. The entire data collection, annotation, and check process takes approximately 4,000 person-hours.
(a) Rotation.
(b) Reducing the resolution.
(c) Distortion.
Figure 2: Illustration of data processing to generate  $\mathbf{M}^5$  HisDoc-H subset.
After the annotating is completed, we process the images in three ways to make the dataset more representative of some problems in real-world scenarios, such as image rotation, distortion, and blur. Firstly, we select 834 images with white backgrounds and randomly rotate them  $\pm (5^{\circ} \sim 15^{\circ})$ . Secondly, we utilize DewarpNet [39] to randomly distort  $50\%$  of the images to simulate document distortion in the real world. Thirdly, We select 2,041 images featuring large character boxes, which are subsequently scaled down randomly, while maintaining a minimum edge size of 18 pixels. The processing scheme is shown in Fig. 2. We denote the pre-processed version and the post-processed version as  $\mathbf{M}^{5}\mathbf{HisDoc}-\mathbf{R}$  (Regular) and  $\mathbf{M}^{5}\mathbf{HisDoc}-\mathbf{H}$  (Hard), respectively. To ensure a fair evaluation, the two subsets are both divided into training, validation, and testing sets in a ratio of 2:1:1.
3.3 Dataset analysis
The comparisons of the statistics between  $\mathbf{M}^5$  HisDoc and other datasets are summarized in Table 1. We compare our dataset with previous datasets in three aspects, including text density, style types, and the number of character categories.
Table 1: Comparison with existing Chinese historical document datasets. Cls means The number of character categories. * indicates that we only consider the training set due to only the training set of ICDAR 2019 HDRC-CHINESE is available.
Dataset	Images	Text lines/im	Char/im	Cls	Char box	Layout type	Doc type
MTHv1 [40]	1,500	27.10	347.58	4,058	✓	< 10	1
MTHv2 [5]	3,199	33.0	338.13	6,733	✓	< 10	1
IC19 HDRC [6]*	11,715	35.64	211.95	8,353	×	< 10	1
CASIA-AHCDB [7]	-	-	-	10,350	-	-	2
M5HisDoc (Ours)	8,000	50.48	545.92	16,151	✓	> 20	> 20
(b)
Figure 3: Statistics of  $\mathbf{M}^5$  HisDoc. (a) The aspect ratio of the images. (b) The number of characters per category. (c) The distributions of the text line length.
Text density As shown in Table 1, it is evident that the average number of characters and text lines in each image of  $\mathbf{M}^5$  HisDoc significantly exceeds that of existing datasets. The dense texts pose a significant challenge for both detection and recognition tasks.
Style types (a) Layout types: As shown in Fig. 1a,  $\mathbf{M}^5$ HisDoc dataset contains more than 20 types of layouts, which is more diverse than the existing benchmarks. The multiple types of layout pose a significant challenge for text detection and reading order understanding tasks. (b) Document types:  $\mathbf{M}^5$ HisDoc contains more than 20 document types, as are shown in Fig. 1b. In contrast, existing datasets typically have a single document type. For example, MTHv2 [5] and ICDAR 2019 HDRC-CHINESE [6] contain only Buddhist scriptures and genealogy, respectively.  $\mathbf{M}^5$ HisDoc, however, is sourced from a wide range of materials, including medical books, anthologies, ancient dictionaries, and so on. (c)  $\mathbf{M}^5$ HisDoc dataset encompasses a diverse array of Chinese calligraphic styles, including cursive, clerical, regular, and running scripts, which are illustrated in Fig. 1c. The significant variation in calligraphy styles poses a major challenge for the recognizer, requiring a robust system to accurately process them. (d) Background types: As shown in the Fig. 1d,  $\mathbf{M}^5$ HisDoc contains various background types. (e)  $\mathbf{M}^5$ HisDoc presents various challenges, including dense texts, distortion, rotation, damage, image blurriness, and variations in font sizes, as illustrated in Fig. 1e. Therefore, we can potentially avoid overfitting the historical document digitization system to
a specific type, ensuring that our dataset is more representative and comprehensive for real-world applications.
The number of character categories  $\mathbf{M}^5$ HisDoc contains 16,151 categories, which is at least 1.6 times more than other existing datasets. The multitude of categories benefit a lot from the diverse range of our data sources, as documents of different types have their own unique character sets. In addition, as the character categories of historical document digitization systems often struggle to cover the ones in real-world applications, we set up zero-shot recognition scenarios in  $\mathbf{M}^5$ HisDoc, where some character categories in the validation and test sets are not presented in the training set. The specific distribution of category are shown in Table 2, we can see that the validation set and the test set contain 1,853 and 1,851 categories that are not presented in the train set, respectively.
Table 2: Category distribution of  ${\mathrm{M}}^{5}$  HisDoc.
Dataset	Category	Zero shot category
Training set	12,779	-
Validation set	9883	1853
Test set	9748	1831
All	16,151	-
We also calculate the aspect ratio of the images, the number of characters per category, and the distributions of the text line length. As shown in Fig. 3a, the aspect ratio of the images varies significantly, ranging from less than 0.4 to more than 1.8. This is because  $\mathrm{M}^5\mathrm{HisDoc}$  contains a variety of styles and layouts. The number of samples per category exhibits a clear long-tail distribution, as demonstrated in Fig. 3b. The category with the largest number of samples consists of over 30,000 instances, whereas the category with the fewest has samples fewer than 3. It can also be observed from Fig. 3c that there exists a notable diversity in text length in  $\mathrm{M}^5\mathrm{HisDoc}$ . About  $40\%$  of texts exhibit a length of fewer than 6 characters, while about  $7\%$  of texts surpass the threshold of 25 characters. Some texts are extremely long, with the longest text containing 58 characters.
4.1 Text line detection
Setting To provide a comprehensive evaluation of the text line detection on  $\mathbf{M}^5$  HisDoc, we benchmark three categories of methods, including regression-based [16, 41, 15], segmentation-based [18, 19, 20, 21] and connected component-based [23] approaches. We use ResNet-50 [42] with a FPN pre-trained on ImageNet [43] as our backbone. All models are optimized using AdamW [44] with an initial learning rate of 1e-4. We train the models with a batch size of 8 for 160 epochs, and the learning rate decays by a factor of 0.1 at epochs 80 and 128. Various data augmentation techniques are applied, including random scale, color jitter, horizontal flip, random crop, and rotation. During training, the long sides of the training images are scaled to a fixed size of 1,333, while the short sides are randomly scaled to different scales (704, 736, 768, 800, 832, 864, 896). During testing, the long and short sides are resized to 1,333 and 800, respectively. The experimental evaluation metrics included precision, recall, and F1 score at different IoU thresholds (0.5, 0.6, 0.7). In Table 3, we also report the evaluation results of end-to-end performance using the normalized edit distance (1-NED) metric [45], by passing the detected text through the CRNN [26] model described in Sec. 4.2.
Results and analysis (1) The extensive presence of lengthy and densely texts in  $\mathbf{M}^5$  HisDoc poses challenges for segmentation-based methods. As shown in Table 3, all methods can achieve considerable performance under the 0.5 IoU threshold. However, as the IoU threshold increases, segmentation-based models exhibit a substantial decline in text line detection. This because the presence of long texts introduces inaccuracies in the segmentation process, while dense texts make these methods prone to predicting fused results. (2) The inclusion of rotated or distorted text in  $\mathbf{M}^5$  HisDoc-H substantially diminishes the recall performance of regression-based methods. The reason is that he proposals generated by the conventional Region Proposal Network (RPN) in regression-based methods remain horizontal and rectangular, which exhibit substantial overlap
with surrounding text instances. They will be easily suppressed by NMS and subsequently increase missing detection. (3) High IoU detection is crucial for historical document analysis. The last columns of Table 3 demonstrate that while the detection models show relatively good performance, the  $1 - NED$  in the end-to-end systems falls short of expectations. This is largely due to the absence of accurate detection of texts, which leads to the omission of certain characters or the inclusion of surrounding characters.