SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 3: Results of text line detection, in the format of  ${\mathrm{M}}^{5}$  HisDoc-H/M  ${}^{5}$  HisDoc-R test set.
Type	Method	Venue	IoU thres	Precision↑	Recall↑	F1-score↑	1-NED↑
Regression-based	Mask R-CNN [16]	CVPR'16	0.5	88.66/97.73	80.46/92.84	84.36/95.22
0.6	88.04/97.36	79.89/92.50	83.77/94.87	66.17/83.85
0.7	86.61/96.44	78.60/91.62	82.41/93.97
Cascade R-CNN [41]	CVPR'18	0.5	91.61/98.41	82.37/92.88	86.74/95.57
0.6	91.07/98.10	81.89/92.59	86.24/95.26	69.68/84.43
0.7	89.94/97.35	80.86/91.88	85.16/94.54
OBD [15]	IJCV'21	0.5	94.73/97.31	81.52/91.35	87.63/94.24
0.6	91.91/96.55	79.10/90.63	85.02/93.49	71.56/82.71
0.7	85.99/94.66	74.00/88.85	79.55/91.66
Segmentation-based	PSENet [18]	CVPR'19	0.5	80.49/95.31	85.91/94.51	83.11/94.90
0.6	76.14/91.03	81.26/90.26	78.61/90.65	63.77/80.60
0.7	65.60/79.08	70.01/78.42	67.73/78.75
PAN [19]	ICCV'19	0.5	92.83/93.59	80.48/88.58	86.22/91.02
0.6	88.70/86.91	76.91/82.27	82.39/84.53	68.91/75.10
0.7	77.03/67.90	66.79/64.27	71.55/66.03
FCENet [20]	CVPR'21	0.5	87.92/91.82	84.67/87.70	86.27/89.71
0.6	80.17/86.16	77.21/82.30	78.66/84.19	67.15/72.84
0.7	66.80/75.67	64.33/72.28	65.55/73.94
DBNet++ [21]	T-PAMI'22	0.5	91.81/89.78	89.76/86.83	90.77/88.28
0.6	63.92/65.15	62.49/63.01	63.20/64.06	73.75/70.97
0.7	39.01/38.10	38.14/36.84	38.57/37.46
Connected component-based	TextSnake [23]	ECCV'18	0.5	95.46/93.25	86.67/88.74	90.85/90.94
0.6	93.34/89.99	84.74/85.63	88.83/87.75	75.01/74.76
0.7	89.06/86.15	80.86/81.98	84.76/84.01
4.2 Text line recognition
Setting To assess the performance of text line recognition methods on  $\mathbf{M}^5$  HisDoc, we select several representative methods from three categories: (1) CTC-based, including CRNN [26], Ma et al. [5], and ZCTRN [27]. (2) Attention-based, such as ASTER [29], NRTR [30] and Robust Scanner [31]. (3) Segmentation-based, such as Peng et al. [34]. The models are optimized by AdamW [44] with a base learning rate of 4e-4 and decay to 4e-6 using CosineAnnealing. The models are trained for 50 epochs with a batch size of 32. Thin Plate Spline transformation (TPS) algorithm is applied to each text line image for pre-processing. The height of each image is normalized to 96 while ensuring that the width is limited to a maximum of 1,536 pixels. Evaluation metrics are correct rate (CR) and accurate rate (AR) [34].
Table 4: Results of text line recognition, in the format of  ${\mathrm{M}}^{5}$  HisDoc-H/M  ${}^{5}$  HisDoc-R test set. CR: correct rate, AR: accurate rate. * indicates the annotation of the character bounding box is used in training.
Type	Method	Venue	CR↑	AR↑
CTC-based	CRNN [26]	T-PAMI'16	90.22/91.49	90.09/91.36
Ma et al. [5]	ICFHR'20	91.29/92.29	90.83/92.10
ZCTRN [27]	ICDAR'21	87.19/88.79	86.96/88.50
Attention-based	ASTER [29]	T-PAMI'18	86.95/87.98	86.63/87.70
NRTR [30]	ICDAR'19	84.50/85.43	82.03/83.02
Robust Scanner [31]	ECCV'20	88.55/90.99	88.33/90.78
Segmentation-based	Peng et al. [34]	TMM'22	88.16/88.38	88.06/88.35
Peng et al. [34]*	TMM'22	89.97/93.03	89.24/92.01
Results and analysis (1) The large number of character categories (16,151) and relatively long texts make attention-based methods perform poorly on this task. These characteristics specifically hinder the effective modeling of contextual relationships by attention-based methods, and leading to significant attention drift issues[46]. Contrarily, CTC-based methods autonomously explore decoding paths and are less impacted by the aforementioned challenges [47], resulting in better performance. (2) The style diversity presented in  $\mathbf{M}^5$  HisDoc results in a significant performance gap between weakly supervised segmentation-based methods and fully supervised methods. The existence of
various backgrounds and calligraphy styles introduces challenges for segmentation-based methods to achieve accurate character detection performance using weak supervision, a characteristic not commonly observed in other historical document datasets[5].
4.3 Character detection
Setting We evaluate the performance of a typical two-stage method, Faster R-CNN [12], and two one-stage methods, YOLOv3 [48] and YOLOX [49] on character detection. The models are optimized using SGD with an initial learning rate of 0.001. We train the models for 160 epochs with a batch size of 8, and the learning rate decays by a factor of 0.1 at the 128th and 144th epochs. In the training and testing phases, the longer and shorter sizes are adjusted to 1,333 and 800, respectively. The evaluation metrics of this task are the same as Sec. 4.1. Moreover, we crop the detected characters and pass them through the SwinTransformer-based [50] character recognizer in Sec. 4.4 to evaluate the end-to-end character recognition performance.
Results and analysis As depicted in Table 5, there is minimal disparity in performance between the one-stage and two-stage methods for character detection. (1) The performance gap between character-level detection schemes in  $\mathbf{M}^5$  HisDoc-R and  $\mathbf{M}^5$  HisDoc-H is significantly smaller than the disparity observed in text line detection as presented in Table 3. This can be attributed to the smaller scale variations of each character and the less sensitivity of character detection to rotation and distortion. (2) The character-based methods demonstrate comparable performance on both versions of  $\mathbf{M}^5$  HisDoc in comparison to the results in Table 3.
In conclusion, the challenges posed by rotation, distortion, and other factors in  $\mathbf{M}^5$  HisDoc-H significantly impact text-line-level approaches. Conversely, character-level approaches demonstrate relatively higher robustness in tackling these challenges.
Table 5: Results of character detection, in the format of  $\mathrm{M}^{5}\mathrm{{HisDoc}} - \mathrm{H}/{\mathrm{M}}^{5}\mathrm{{HisDoc}} - \mathrm{R}$  test set. "Top-1 acc" is the metric of extracting the detected bounding box and feeding it into the SwinTransformer-based character recognizer. * indicates the accuracy of the missed and overchecked characters is set to 0 , otherwise only the characters matching on GT are considered.
Type	Method	Venue	IoU thres	Precision↑	Recall↑	F1-score↑	Top-1 acc↑	Top-1 acc↑*
Two-stage	Faster R-CNN [12]	NIPS'15	0.5	96.36/97.75	98.46/98.53	97.40/98.14
0.6	95.27/96.91	97.34/97.69	96.29/97.30	93.01/94.92	88.29/91.46
0.7	91.84/94.46	93.83/95.21	92.82/94.83
One-stage	YOLOv3 [48]	arXiv'18	0.5	99.20/99.21	94.71/95.88	96.90/97.52
0.6	98.61/98.77	94.14/95.45	96.32/97.08	93.69/95.28	88.04/90.67
0.7	96.43/97.19	92.06/93.93	94.20/95.53
YOLOX [49]	CVPR'21	0.5	96.66/97.14	98.04/99.17	97.35/98.15
0.6	93.77/96.48	95.11/98.49	94.44/97.47	93.57/95.06	88.73/91.60
0.7	84.16/94.59	85.36/96.56	84.76/95.56
4.4 Character recognition
Setting The character recognition model involves utilizing a backbone network, followed by a fully connected layer for final classification. In this study, we analyze the influence of various backbones on the accuracy of the recognition results, including CNN-based [42, 51, 52] and Vision-Transformer-based backbones[53, 50]. The models are optimized by AdamW [44] with a base learning rate of 0.001 and decay to 1e-6 using CosineAnnealing. We train the models with a batch size of 1,024 for 90 epochs. The learning rate for the first 5 epochs warmup from 1e-4 to 1e-3. All models are trained using RandAugment [54]. The input images are all scaled to  $96 \times 96$ . Evaluation metrics are based on top-1 accuracy, top-5 accuracy, and macro accuracy[55] that refers to calculating the accuracy for each category individually and then averaging them.
Results and analysis As shown in Table 6, all models demonstrate considerable top-1 accuracy. However, there are still two prominent challenges in this task.(1) Character recognition with similar morphology. All models achieve high top-5 accuracy, indicating their ability to effectively limit the recognition results within a range of five potential candidates. Nonetheless, the Top-1 accuracy exhibits a significant decline due to errors in discerning similar Chinese characters. (2) Long-tail distribution. The macro accuracy of all models is noticeably lower than the top-1 accuracy. This
observation underscores the presence of the long-tail distribution problem, where characters with limited training samples cannot be accurately recognized.
In real-world historical document analysis scenarios, numerous similar morphology and uncommon characters exist, which pose challenges to character recognition. Accurately recognizing these characters becomes crucial in addressing the aforementioned issues. Compared to existing benchmarks, our research incorporates a wider range of visually similar and uncommon character categories, expanding the class number in  $\mathbf{M}^5$  HisDoc, thereby providing a more comprehensive representation of these challenges.
Table 6: Results of character recognition, in the format of  ${\mathrm{M}}^{5}$  HisDoc-H/M  ${}^{5}$  HisDoc-R test set.
Type	Method	Venue	Top-1 acc↑	Top-5 acc↑	Macro acc↑
CNNs	ResNet50 [42]	CVPR'16	94.40/95.07	98.18/98.46	69.61/73.96
RegNet [51]	CVPR'20	94.56/95.17	98.30/98.48	69.23/72.94
ConvNeXt [52]	CVPR'22	94.61/95.19	98.28/98.52	71.14/75.06
Vision Transformer	ViT [53]	ICLR'21	94.09/94.63	98.28/98.21	65.27/68.10
SwinTransformer [50]	ICCV'21	94.68/95.22	98.33/98.56	71.08/74.34
4.5 Reading order prediction
Setting We conduct experiments on  $\mathbf{M}^5$  HisDoc for reading order prediction with both rule-based and learning-based methods. The rule-based methods include Augmented XY Cut [37] specifically designed for layout analysis in modern documents, and a heuristic algorithm by ourselves. This algorithm arranges the center coordinates of text lines in a left-to-right, top-to-bottom order (The detailed procedure is presented in the Appendix), which is designed mainly based on the fact that Chinese historical documents are typically read in such a manner. For the learning-based method, we employ LayoutReader [35] for comparison. As all of the aforementioned methods exclusively take horizontal rectangle bounding boxes as input, we utilize horizontal minimum enclosing rectangle boxes of each text as inputs. We utilize Average Relative Distance [35] (ARD) to evaluate the performance.
Table 7: Results of reading order prediction, in the format of  ${\mathrm{M}}^{5}$  HisDoc-H/M  ${}^{5}$  HisDoc-R test set. ARD: Average Relative Distance.
Type	Method	Venue	ARD↓
Rule-based	Heuristic method (Ours)	-	5.27/5.20
Augmented XY Cut [37]	CVPR'22	14.63/12.60
Learning-based	LayoutReader [35](layout only)	EMNLP'21	8.68/5.49
Results and analysis (1) The reading order prediction methods designed for modern documents face challenges in generalizing to the analysis of historical documents. The results in Table 7 demonstrate a substantial disparity in performance between augmented XY Cut and the heuristic-based algorithm. This discrepancy can be primarily attributed to the fact that augmented XY Cut is tailored specifically for modern documents. However, due to the significant gap between historical and modern documents, this method struggles to generalize effectively to historical document analysis. In contrast, the heuristic algorithm, which leverages the distinctive characteristics of Chinese historical documents, exhibits notable advantages in predicting the reading order. (2)  $\mathbf{M}^5$  HisDoc-H has presented greater challenges to existing methods for reading order prediction. We can also observe that augmented XY Cut and LayoutReader experience a notable decline in performance on  $\mathbf{M}^5$  HisDoc-H. This is because the polygonal boxes of text in  $\mathbf{M}^5$  HisDoc-H are more distorted compared to those in  $\mathbf{M}^5$  HisDoc-R, resulting in substantial differences between the horizontal minimum enclosing rectangles and the original polygons. Hence,  $\mathbf{M}^5$  HisDoc-H presents numerous cases of overlap with external horizontal rectangles, which pose challenges for these methods to model the relationships between each text line.
4.6 Cross-validation with other benchmarks
Setting We train our models on MTHv2 [5] using Cascade R-CNN [41], CRNN [26], YOLOX [49], and ConvNeXt [52] architectures, with the settings described in Sec. 4.1, 4.2, 4.3, and 4.4, respectively.
We train Cascade R-CNN and CRNN on IC19 HDRC [6], ConvNeXt on CASIA-AHCDB [7]. Subsequently, we perform cross-validation using the models trained on  $\mathbf{M}^5$  HisDoc-H.
Table 8: Cross-validation between the models trained on other datasets and  $\mathbf{M}^5$  HisDoc-H. The evaluation metric for text line/character detection is F-score at 0.5 IoU, the metrics for text line/character recognition are AR and top-1 accuracy, respectively.
Tasks	Text line det (F1-score)	Text line reg (AR)	Char det (F1-score)	Char reg (top-1 acc)
MTHv2 → M5HisDoc-H	53.88	53.95	85.77	73.71
M5HisDocH → MTHv2	97.46	95.83	99.20	96.35
IC19 HDRC → M5HisDoc-H	39.92	56.53	-	-
M5HisDocH → IC19 HDRC	70.56	85.60	-	-
CASIA-AHCDB → M5HisDoc-H	-	-	-	71.91
M5HisDocH → CASIA-AHCDB	-	-	-	92.55
Results and analysis The results presented in Table 8 demonstrate the outstanding performance of the model trained on  $\mathbf{M}^5$  HisDoc-H when evaluated on MTHv2, IC19 HDRC and CASIA-AHCDB. Conversely, the model trained on other datasets faces difficulties in generalizing to  $\mathbf{M}^5$  HisDoc-H. These are mainly due to the multiple sytles of  $\mathbf{M}^5$  HisDoc and strongly indicate that  $\mathbf{M}^5$  HisDoc-H introduces novel challenges to the field, and also reflect its greater representativeness.
Although significant efforts have been made to capture a diverse range of images from various perspectives and simulate realistic distortions through data synthesis, it is crucial to acknowledge that certain instances of distortion still remain uncovered in  $\mathbf{M}^5$  HisDoc. This serves as a part of future work.
In this paper, we propose M<sup>5</sup>HisDoc dataset, which is particularly notable for its diverse range of styles, including multiple layouts, document types, calligraphy styles, backgrounds, and challenges. Furthermore, we conduct an extensive benchmark evaluation of M<sup>5</sup>HisDoc and perform a thorough analysis of the results. Our findings demonstrate that M<sup>5</sup>HisDoc offers a more accurate representation of practical challenges encountered in historical document analysis, compared with existing benchmarks. Consequently, we firmly believe that this comprehensive dataset will provide researchers and developers with a valuable resource to advance the field of historical document analysis.
This research is supported in part by NSFC (Grant No.: 61936003) and Science and Technology Foundation of Guangzhou Huangpu Development District (Grant No.: 2020GH17).
[1] Xiaoqing Ding, Di Wen, Liangrui Peng, and Changsong Liu. Document digitization technology and its application for digital library in China. In Proc. DIAL, pages 46-53. IEEE, 2004.
[2] Dezhi Peng, Lianwen Jin, Yuliang Liu, Canjie Luo, and Songxuan Lai. PageNet: Towards end-to-end weakly supervised page-level handwritten Chinese text recognition. Int. J. Comput. Vis., 130(11):2623-2645, 2022.
[3] Oldrich Kodym and Michal Hradis. Page layout analysis system for unconstrained historic documents. In Proc. ICDAR, pages 492-506. Springer, 2021.
[4] Olfa Mechi, Maroua Mehri, Rolf Ingold, and Najoua Essoukri Ben Amara. A two-step framework for text line segmentation in historical Arabic and Latin document images. Int. J. Doc. Anal. Recognit., 24(3):197-218, 2021.
[45] Baoguang Shi, Cong Yao, Minghui Liao, Mingkun Yang, Pei Xu, Linyan Cui, Serge Belongie, Shijian Lu, and Xiang Bai. ICDAR2017 competition on reading Chinese text in the wild (RCTW-17). In Proc. ICDAR, volume 1, pages 1429–1434. IEEE, 2017.
[46] Zhanzhan Cheng, Fan Bai, Yunlu Xu, Gang Zheng, Shiliang Pu, and Shuigeng Zhou. Focusing attention: Towards accurate text recognition in natural images. In Proc. ICCV, pages 5076-5084, 2017.
[47] Haiyang Yu, Jingye Chen, Bin Li, Jianqi Ma, Mengnan Guan, Xixi Xu, Xiaocong Wang, Shaobo Qu, and Xiangyang Xue. Benchmarking Chinese text recognition: Datasets, baselines, and an empirical study. arXiv preprint arXiv:2112.15093, 2021.
[48] Joseph Redmon and Ali Farhadi. YOLOv3: An incremental improvement. arXiv preprint arXiv:1804.02767, 2018.
[49] Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, and Jian Sun. YOLOX: Exceeding yolo series in 2021. arXiv preprint arXiv:2107.08430, 2021.
[50] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical vision Transformer using shifted windows. In Proc. ICCV, pages 10012-10022, 2021.
[51] Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proc. CVPR, pages 10428-10436, 2020.
[52] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A ConvNet for the 2020s. In Proc. CVPR, pages 11976-11986, 2022.
[53] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In Proc. ICLR, pages 1-22, 2021.
[54] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proc. CVPR Workshops, pages 702-703, 2020.
[55] Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learning: A survey. IEEE Trans. Pattern Anal. Mach. Intell., 2023.