Reviewerly/exHarmony huggingface dataset:
HisDoc R-CNN: Robust Chinese Historical Document Text Line Detection with Dynamic Rotational Proposal Network and Iterative Attention Head

Text line detection is an essential task in a historical document analysis system. Although many existing text detection methods have achieved remarkable performance on various scene text datasets, they cannot perform well because of the high density, multiple scales, and multiple orientations of text lines in complex historical documents. Thus, it is crucial and challenging to investigate effective text line detection methods for historical documents. In this paper, we propose a Dynamic Rotational Proposal Network (DRPN) and an Iterative Attention Head (IAH), which are incorporated into Mask R-CNN to detect text lines in historical documents. The DRPN can dynamically generate horizontal or rotational proposals to enhance the robustness of the model for multi-oriented text lines and alleviate the multi-scale problem in historical documents. The proposed IAH integrates a multi-dimensional attention mechanism that can better learn the features of dense historical document text lines while improving detection accuracy and reducing the model parameters via an iterative mechanism. Our HisDoc R-CNN achieves state-of-the-art performance on various historical document benchmarks including CHDAC (the IACC competition ( http://iacc.pazhoulab-huangpu.com/shows/108/1.html ) dataset), MTHv2, and ICDAR 2019 HDRC CHINESE, thereby demonstrating the robustness of our method. Furthermore, we present special tricks for historical document scenarios, which may provide useful insights for practical applications.
BaDLAD: A Large Multi-Domain Bengali Document Layout Analysis Dataset

While strides have been made in deep learning based Bengali Optical Character Recognition (OCR) in the past decade, absence of large Document Layout Analysis (DLA) datasets has hindered the application of OCR in document transcription, e.g., transcribing historical documents and newspapers. Moreover, rule-based DLA systems that are currently being employed in practice are not robust to domain variations and out-of-distribution layouts. To this end, we present the first multi-domain large Bengali Document Layout Analysis Dataset: BaDLAD. This dataset contains 33, 695 human annotated document samples from six domains - i) books and magazines ii) public domain govt. documents iii) liberation war documents iv) new newspapers v) historical newspapers and vi) property deeds; with 710K polygon annotations for four unit types: text-box, paragraph, image, and table. Through preliminary experiments benchmarking the performance of existing state-of-the-art deep learning architectures for English DLA, we demonstrate the efficacy of our dataset in training deep learning based Bengali document digitization models.
Detecting Text on Historical Maps by Selecting Best Candidates of Deep Neural Networks Output

The final and perhaps the most crucial step in Object Detection is the selection of the best candidates out of all the proposed regions a framework outputs. Typically, Non-Maximum Suppression approaches (NMS) are employed to tackle this problem. The standard NMS relies exclusively on the confidence scores, as it selects the bounding box with the highest score within a cluster of boxes determined by a relatively high Intersection over Union (IoU) between each other, and then suppresses the remaining ones. On the other hand, algorithms like Confluence determine clusters of bounding boxes according to the proximity between them and select as best the box that is closer to the other ones within each cluster. In this work, we combine these methods by creating clusters of high confidence scores according to their IoU and then we calculate the sums of the Manhattan distances between the vertices of each box and all the others, in order to finally select the one with the minimum overall distance. Our results are compared with the standard NMS and the Locality-Aware NMS (LANMS), an algorithm that is widely used in Object Detection and merges the boxes row by row. The research field that this work explores is the text detection on historical maps and the proposed approach results to average precision that is 2.14–2.94% higher for evaluation IoU in range 0.50 to 0.95 with step 0.05 than the two other methods.
CCpdf: Building a High Quality Corpus for Visually Rich Documents from Web Crawl Data

In recent years, the field of document understanding has progressed a lot. A significant part of this progress has been possible thanks to the use of language models pretrained on large amounts of documents. However, pretraining corpora used in the domain of document understanding are single domain, monolingual, or nonpublic. Our goal in this paper is to propose an efficient pipeline for creating a big-scale, diverse, multilingual corpus of PDF files from all over the Internet using Common Crawl, as PDF files are the most canonical types of documents as considered in document understanding. We analyzed extensively all of the steps of the pipeline and proposed a solution which is a trade-off between data quality and processing time. We also share a CCpdf corpus in a form or an index of PDF files along with a script for downloading them, which produces a collection useful for language model pretraining. The dataset and tools published with this paper offer researchers the opportunity to develop even better multilingual language models.
Decoupling Visual-Semantic Features Learning with Dual Masked Autoencoder for Self-Supervised Scene Text Recognition

Self-supervised text recognition has attracted more and more attention since it provides an effective way to utilize unlabeled real text images. Nowadays, Masked Image Modeling (MIM) shows superiority in visual representation learning, and several works introduce it into text recognition. In this paper, we take a further step and design a method for text-recognition-friendly self-supervised feature learning. Specifically, we propose to decouple visual and semantic feature learning with different masking strategies. For the visual features, intra-window random masking is proposed where the reconstruction is applied on a local image region with random masking, which prevents the model from the help of much context information. In the meanwhile, semantic feature learning is based on a window random masking, which removes more visual clues and boosts the sequence modeling of the model. Based on this idea, we first propose a siamese network that aligns dual features with each other, then we explore the dual distillation with a co-teacher framework. Our proposed method shows the effectiveness of self-supervised scene text recognition with state-of-the-art performances on most benchmarks.
ICDAR 2023 Competition on Robust Layout Segmentation in Corporate Documents

Transforming documents into machine-processable representations is a challenging task due to their complex structures and variability in formats. Recovering the layout structure and content from PDF files or scanned material has remained a key problem for decades. ICDAR has a long tradition in hosting competitions to benchmark the state-of-the-art and encourage the development of novel solutions to document layout understanding. In this report, we present the results of our ICDAR 2023 Competition on Robust Layout Segmentation in Corporate Documents, which posed the challenge to accurately segment the page layout in a broad range of document styles and domains, including corporate reports, technical literature and patents. To raise the bar over previous competitions, we engineered a hard competition dataset and proposed the recent DocLayNet dataset for training. We recorded 45 team registrations and received official submissions from 21 teams. In the presented solutions, we recognize interesting combinations of recent computer vision models, data augmentation strategies and ensemble methods to achieve remarkable accuracy in the task we posed. A clear trend towards adoption of vision-transformer based methods is evident. The results demonstrate substantial progress towards achieving robust and highly generalizing methods for document layout understanding.
How to Choose Pretrained Handwriting Recognition Models for Single Writer Fine-Tuning

Recent advancements in Deep Learning-based Handwritten Text Recognition (HTR) have led to models with remarkable performance on both modern and historical manuscripts in large benchmark datasets. Nonetheless, those models struggle to obtain the same performance when applied to manuscripts with peculiar characteristics, such as language, paper support, ink, and author handwriting. This issue is very relevant for valuable but small collections of documents preserved in historical archives, for which obtaining sufficient annotated training data is costly or, in some cases, unfeasible. To overcome this challenge, a possible solution is to pretrain HTR models on large datasets and then fine-tune them on small single-author collections. In this paper, we take into account large, real benchmark datasets and synthetic ones obtained with a styled Handwritten Text Generation model. Through extensive experimental analysis, also considering the amount of fine-tuning lines, we give a quantitative indication of the most relevant characteristics of such data for obtaining an HTR model able to effectively transcribe manuscripts in small collections with as little as five real fine-tuning lines.
A Study on Reproducibility and Replicability of Table Structure Recognition Methods

Concerns about reproducibility in artificial intelligence (AI) have emerged, as researchers have reported unsuccessful attempts to directly reproduce published findings in the field. Replicability, the ability to affirm a finding using the same procedures on new data, has not been well studied. In this paper, we examine both reproducibility and replicability of a corpus of 16 papers on table structure recognition (TSR), an AI task aimed at identifying cell locations of tables in digital documents. We attempt to reproduce published results using codes and datasets provided by the original authors. We then examine replicability using a dataset similar to the original as well as a new dataset, GenTSR, consisting of 386 annotated tables extracted from scientific papers. Out of 16 papers studied, we reproduce results consistent with the original in only four. Two of the four papers are identified as replicable using the similar dataset under certain IoU values. No paper is identified as replicable using the new dataset. We offer observations on the causes of irreproducibility and irreplicability. All code and data are available on Codeocean at https://codeocean.com/capsule/6680116/tree .
RealCQA: Scientific Chart Question Answering as a Test-Bed for First-Order Logic

We present a comprehensive study of chart visual question-answering(QA) task, to address the challenges faced in comprehending and extracting data from chart visualizations within documents. Despite efforts to tackle this problem using synthetic charts, solutions are limited by the shortage of annotated real-world data. To fill this gap, we introduce a benchmark and dataset for chart visual QA on real-world charts, offering a systematic analysis of the task and a novel taxonomy for template-based chart question creation. Our contribution includes the introduction of a new answer type, ‘list’, with both ranked and unranked variations. Our study is conducted on a real-world chart dataset from scientific literature, showcasing higher visual complexity compared to other works. Our focus is on template-based QA and how it can serve as a standard for evaluating the first-order logic capabilities of models. The results of our experiments, conducted on a real-world out-of-distribution dataset, provide a robust evaluation of large-scale pre-trained models and advance the field of chart visual QA and formal logic verification for neural networks in general. Our code and dataset is publicly available ( https://github.com/cse-ai-lab/RealCQA ).
MIDV-Holo: A Dataset for ID Document Hologram Detection in a Video Stream

One of the most important tasks related to identity document analysis is to verify its authenticity. A powerful family of security features of ID documents is optically variable devices, such as holograms. The development of computer vision methods for the automatic validation of holograms is complicated due to the lack of public datasets. In this paper, we present MIDV-Holo – a public dataset of synthetically created passports and ID cards, captured on video under various conditions. The dataset features 300 video clips of documents with custom holographic security elements, as well as 400 video clips of presentation attacks of different types. The paper describes the process of dataset creation, available annotation, and a baseline algorithm for the holograms detection method based on an analysis of pixel-wise chromaticity statistics accumulated in a video stream, with an evaluation on the presented dataset. Authors believe that, being the first public dataset in the domain of hologram-based identity document authentication, MIDV-Holo will provide researchers in the field of identity verification with a valuable resource for testing new methods of video-based ID document analysis.
Optimizing the Performance of Text Classification Models by Improving the Isotropy of the Embeddings Using a Joint Loss Function

Recent studies show that the spatial distribution of the sentence representations generated from pre-trained language models is highly anisotropic. This results in a degradation in the performance of the models on the downstream task. Most methods improve the isotropy of the sentence embeddings by refining the corresponding contextual word representations, then deriving the sentence embeddings from these refined representations. In this study, we propose to improve the quality of the sentence embeddings extracted from the [CLS] token of the pre-trained language models by improving the isotropy of the embeddings. We add one feed-forward layer between the model and the downstream task layers, and we train it using a novel joint loss function. The proposed approach results in embeddings with better isotropy, that generalize better on the downstream task. Experimental results on 3 GLUE datasets with classification as the downstream task show that our proposed method is on par with the state-of-the-art, as it achieves performance gains of around 2–3% on the downstream tasks compared to the baseline.
ICDAR 2023 Competition on Hierarchical Text Detection and Recognition

We organize a competition on hierarchical text detection and recognition. The competition is aimed to promote research into deep learning models and systems that can jointly perform text detection and recognition and geometric layout analysis. We present details of the proposed competition organization, including tasks, datasets, evaluations, and schedule. During the competition period (from January 2nd 2023 to April 1st 2023), at least 50 submissions from more than 20 teams were made in the 2 proposed tasks. Considering the number of teams and submissions, we conclude that the HierText competition has been successfully held. In this report, we will also present the competition results and insights from them.
Multimodal Scoring Model for Handwritten Chinese Essay

Essay writing plays a critical role in Chinese language skill teaching. With the smart education becomes a hot topic, the demand for automatic essay scoring (AES) has been emerging among teachers and students. Existing works frequently ignore the impact of the visual modal during the scoring process, such as writing quality in terms of neatness or legibility. This paper addresses the problem with a visual-textual integrating perspective and proposes a deep learning based multi-modal AES. Specifically, implicit alignment algorithm is presented to cohere the distinct visual modal and text modal. Methods are tested on a large-scale dataset consisting of over 4000 essays including HSK publicly available samples. The results show that multi-modal AES reduce the MAE of scoring from 1.13 to 1.06, and the implicit alignment algorithm reduces it further to 1.01.
SelfDocSeg: A Self-supervised Vision-Based Approach Towards Document Segmentation

Document layout analysis is a known problem to the documents research community and has been vastly explored yielding a multitude of solutions ranging from text mining, and recognition to graph-based representation, visual feature extraction, etc. However, most of the existing works have ignored the crucial fact regarding the scarcity of labeled data. With growing internet connectivity to personal life, an enormous amount of documents had been available in the public domain and thus making data annotation a tedious task. We address this challenge using self-supervision and unlike, the few existing self-supervised document segmentation approaches which use text mining and textual labels, we use a complete vision-based approach in pre-training without any ground-truth label or its derivative. Instead, we generate pseudo-layouts from the document images to pre-train an image encoder to learn the document object representation and localization in a self-supervised framework before fine-tuning it with an object detection model. We show that our pipeline sets a new benchmark in this context and performs at par with the existing methods and the supervised counterparts, if not outperforms. The code is made publicly available at: github.com/MaitySubhajit/SelfDocSeg .
LayoutGCN: A Lightweight Architecture for Visually Rich Document Understanding

Visually rich document understanding techniques have numerous practical application scenarios in the real world. However, existing researches tend to focus on large-scale models, especially pre-trained ones, training which is resource-consuming. Additionally, these models are unfeasible for low-resource situations, like edge applications. This paper proposes the LayoutGCN, a novel, lightweight architecture, to classify, extract, and structuralize information from visually rich documents. It treats a document as a graph of text blocks and employs convolution neural networks to encode all features from different modalities. Rich representations for text blocks containing textual, layout, and visual information are generated by a graph convolution network whose adjacency matrix is carefully designed to fully use the relative position information between text blocks. Extensive experiments on five benchmarks show the applicability of LayoutGCN for various downstream tasks and its comparable performance to existing large-scale models.
I-WAS: A Data Augmentation Method with GPT-2 for Simile Detection

Simile detection is a valuable task for many natural language processing (NLP)-based applications, particularly in the field of literature. However, existing research on simile detection often relies on corpora that are limited in size and do not adequately represent the full range of simile forms. To address this issue, we propose a simile data augmentation method based on Word replacement And Sentence completion using the GPT-2 language model. Our iterative process called I-WAS, is designed to improve the quality of the augmented sentences. To better evaluate the performance of our method in real-world applications, we have compiled a corpus containing a more diverse set of simile forms for experimentation. Our experimental results demonstrate the effectiveness of our proposed data augmentation method for simile detection.
Linear Object Detection in Document Images Using Multiple Object Tracking

Linear objects convey substantial information about document structure, but are challenging to detect accurately because of degradation (curved, erased) or decoration (doubled, dashed). Many approaches can recover some vector representation, but only one closed-source technique introduced in 1994, based on Kalman filters (a particular case of Multiple Object Tracking algorithm), can perform a pixel-accurate instance segmentation of linear objects and enable to selectively remove them from the original image. We aim at re-popularizing this approach and propose: 1. a framework for accurate instance segmentation of linear objects in document images using Multiple Object Tracking (MOT); 2. document image datasets and metrics which enable both vector- and pixel-based evaluation of linear object detection; 3. performance measures of MOT approaches against modern segment detectors; 4. performance measures of various tracking strategies, exhibiting alternatives to the original Kalman filters approach; and 5. an open-source implementation of a detector which can discriminate instances of curved, erased, dashed, intersecting and/or overlapping linear objects.
CED: Catalog Extraction from Documents

Sentence-by-sentence information extraction from long documents is an exhausting and error-prone task. As the indicator of document skeleton, catalogs naturally chunk documents into segments and provide informative cascade semantics, which can help to reduce the search space. Despite their usefulness, catalogs are hard to be extracted without the assist from external knowledge. For documents that adhere to a specific template, regular expressions are practical to extract catalogs. However, handcrafted heuristics are not applicable when processing documents from different sources with diverse formats. To address this problem, we build a large manually annotated corpus, which is the first dataset for the Catalog Extraction from Documents (CED) task. Based on this corpus, we propose a transition-based framework for parsing documents into catalog trees. The experimental results demonstrate that our proposed method outperforms baseline systems and shows a good ability to transfer. We believe the CED task could fill the gap between raw text segments and information extraction tasks on extremely long documents. Data and code are available at https://github.com/Spico197/CatalogExtraction
UTRNet: High-Resolution Urdu Text Recognition in Printed Documents

In this paper, we propose a novel approach to address the challenges of printed Urdu text recognition using high-resolution, multi-scale semantic feature extraction. Our proposed UTRNet architecture, a hybrid CNN-RNN model, demonstrates state-of-the-art performance on benchmark datasets. To address the limitations of previous works, which struggle to generalize to the intricacies of the Urdu script and the lack of sufficient annotated real-world data, we have introduced the UTRSet-Real, a large-scale annotated real-world dataset comprising over 11,000 lines and UTRSet-Synth, a synthetic dataset with 20,000 lines closely resembling real-world and made corrections to the ground truth of the existing IIITH dataset, making it a more reliable resource for future research. We also provide UrduDoc, a benchmark dataset for Urdu text line detection in scanned documents. Additionally, we have developed an online tool for end-to-end Urdu OCR from printed documents by integrating UTRNet with a text detection model. Our work not only addresses the current limitations of Urdu OCR but also paves the way for future research in this area and facilitates the continued advancement of Urdu OCR technology. The project page with source code, datasets, annotations, trained models, and online tool is available at abdur75648.github.io/UTRNet .