README.md

---
license: apache-2.0
---

# Model Card for eynollah-main-regions-ensembled

<!-- Provide a quick summary of what the model is/does. [Optional] -->
This model is part of a suite of 13 models. The suite introduces an end-to-end pipeline to extract layout, text lines and reading order for historic documents, where the output can be used as an input for OCR engines.

Questions and comments about the models can be directed to Vahid Rezanezhad at Vahid.Rezanezhad@sbb.spk-berlin.de.

#  Table of Contents

- [Model Card for eynollah-main-regions-ensembled](#model-card-for-eynollah-main-regions-ensembled)
- [Table of Contents](#table-of-contents)
- [Model Details](#model-details)
  - [Model Description](#model-description)
- [Uses](#uses)
  - [Direct Use](#direct-use)
  - [Downstream Use](#downstream-use)
  - [Out-of-Scope Use](#out-of-scope-use)
- [Bias, Risks, and Limitations](#bias-risks-and-limitations)
  - [Recommendations](#recommendations)
- [Training Details](#training-details)
  - [Training Data](#training-data)
  - [Training Procedure](#training-procedure)
    - [Preprocessing](#preprocessing)
    - [Speeds, Sizes, Times](#speeds-sizes-times)
- [Evaluation](#evaluation)
  - [Testing Data, Factors and Metrics](#testing-data-factors-and-metrics)
    - [Testing Data](#testing-data)
    - [Metrics](#metrics)
- [Model Examination](#model-examination)
- [Environmental Impact](#environmental-impact)
- [Technical Specifications](#technical-specifications)
  - [Model Architecture and Objective](#model-architecture-and-objective)
  - [Software](#software)
- [Citation](#citation)
- [More Information](#more-information)
- [Model Card Authors](#model-card-authors)
- [Model Card Contact](#model-card-contact)
- [How to Get Started with the Model](#how-to-get-started-with-the-model)


# Model Details

## Model Description

<!-- Provide a longer summary of what this model is/does. -->

This suite of 13 models presents a document layout analysis (DLA) system for historical documents implemented by pixel-wise segmentation using convolutional neural networks. In addition, heuristic methods are applied to detect marginals and to determine the reading order of text regions.

The detection and classification of multiple classes of layout elements such as headings, images, tables etc. as part of DLA is required in order to extract and process them in subsequent steps. Altogether, the combination of image detection, classification and segmentation on the wide variety that can be found in over 400 years of printed cultural heritage makes this a very challenging task. Deep learning models are complemented with heuristics for the detection of text lines, marginals, and reading order. Furthermore, an optional image enhancement step was added in case of documents that either have insufficient pixel density and/or require scaling. Also, a column classifier for the analysis of multi-column documents was added. With these additions, DLA performance was improved, and a high accuracy in the prediction of the reading order is accomplished.

Two Arabic/Persian terms form the name of the model suite: عين الله, which can be transcribed as "ain'allah" or "eynollah"; it translates into English as "God's Eye" -- it sees (nearly) everything on the document image. 



- **Developed by:** [Vahid Rezanezhad](Vahid.Rezanezhad@sbb.spk-berlin.de)
- **Shared by:** [Staatsbibliothek zu Berlin / Berlin State Library](https://huggingface.co/SBB)
- **Model type:** Neural Network
- **Language(s) (NLP):** Irrelevant; works on all languages
- **License:** apache-2.0
- **Resources for more information:** 
    - The GitHub Repo can be found [here](https://github.com/qurator-spk/eynollah)
    - Associated Paper: [Document Layout Analysis with Deep Learning and Heuristics](https://doi.org/10.1145/3604951.3605513)

# Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

The intended use of the suite is performing document layout analysis (DLA) on image data. The system returns the results in [PAGE-XML format](https://github.com/PRImA-Research-Lab/PAGE-XML).

## Direct Use

<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
<!-- If the user enters content, print that. If not, but they enter a task in the list, use that. If neither, say "more info needed." -->

The system performs document layout analysis in a series of steps. First, the image is cropped and the number of columns determined. Then the pixels-per-inch (ppi) rate is detected, and when ppi is below 300, the image is re-scaled and enhanced. Now the main region types (text regions, images, separators, background) are detected as the early layout. Marginals are detected with a heuristic method, then -- optionally -- headers (or headings or floatings) and drop capitals. Next, textline segmentation is performed for text regions, and for each text region the slope of deskewing is calculated. Heuristics are used to determine bounding boxes (or contours in the case of curved lines) of textlines in each region after deskewing. After that, the reading order of text regions is detected based on separators, headers (or headings or floatings) and the coordinates of columns. Finally, all results are written to a PAGE-XML file.

Four models of the suite are focused on the implementation of early document layout detection techniques aimed at extracting essential attributes such as background, text regions, images, and separators from document images. To enhance the performance and adaptability of the models across diverse scenarios, a comprehensive training approach involving various augmentation procedures has been employed. By systematically experimenting with different augmentation techniques, efforts are made to achieve optimal results, ensuring that the models exhibit robustness and accuracy in effectively detecting the distinct components of documents in a wide range of real-world scenarios. 

**Within the suite, the model *eynollah-main-regions-ensembled/2021-04-25/saved_model.pb* is used for the task of early layout detection**.

The robustness of this model is attained through an ensembling technique that combines the weights from various epochs. By employing this approach, the model achieves a high level of resilience and stability, effectively leveraging the strengths of multiple epochs to enhance its overall performance and deliver consistent and reliable results.

## Downstream Use

<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
<!-- If the user enters content, print that. If not, but they enter a task in the list, use that. If neither, say "more info needed." -->

The intended use of this suite of 13 models is conceived of as a system. Comparable systems are [dhSegment](https://doi.org/10.1109/ICFHR-2018.2018.00011) and [P2PaLA](https://github.com/lquirosd/P2PaLA). However -- and as always with a system -- , every component can potentially used on its own. Each model might therefore be used or trained for other downstream purposes.


## Out-of-Scope Use

<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
<!-- If the user enters content, print that. If not, but they enter a task in the list, use that. If neither, say "more info needed." -->

This model suite does **NOT** perform any Optical Character Recognition (OCR), it is an image-to-PAGE-XML system only.

# Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

The pre-processing of digitised historical and contemporary texts is a task contributing to knowledge creation, both by developing models facilitating the necessary pre-processing steps of document layout analysis and ultimately by enabling better discoverability of information in the processed works. Since the content of the document images is not touched, ethical challenges cannot be identified. The endeavor of developing the model was not undertaken for profit. Though a commercial product based on this model may be developed in the future, it will always remain openly accessible without any commercial interest. The aim of the development of this model was to improve document layout analysis, an endeavour that is not for profit. As a technical limitation, it has to be noted that there is a lot of performance to gain for historical text by adding more historical Ground Truth data. 

## Recommendations

<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->

The application of machine learning models to convert a document image into PAGE-XML is a process which can still be improved. The suite of 13 models performs pixel-wise segmentation, which is done in patches; it therefore lacks a global understanding of documents and makes the document layout analysis system unable to detect some document subcategories like page numbers. The end-to-end system with different stages uses the output of each task for the next step. Therefore a poor prediction in early steps may cause a poor final document information extraction. Patch-wise segmentation can cause problems to segment pixels between text blocks, large scale drop capitals, headers and tables, since in each patch the model sees only a part of the document element and not all of it. Therefore, there is a lot to gain by improving the existing model suite.

# Training Details

## Training Data

<!-- This should link to a Data Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

For model training, Ground Truth in PAGE-XML format was sourced mainly from three datasets. The [IMPACT dataset of historical document images](https://doi.org/10.1145/2501115.2501130) contains a representative sample of historical books and newspapers from Europe’s major libraries. Due to restrictions, only the open data from the National Library of the Netherlands (KB) and the Poznan Supercomputing and Networking Center (PSNC) were used. The [Europeana Newspapers Project (ENP) image and ground truth dataset of historical newspapers](https://doi.org/10.1109/ICDAR.2015.7333898) contains images representative of the newspaper digitisation projects of 12 national and major European libraries. The [OCR-D dataset](https://doi.org/10.1145/3322905.3322916) of German prints between 1500 and 1900 is based on a selection from the holdings of the “German Text Archive” (DTA), the Digitized Collections of the Berlin State Library and the Wolfenbüttel Digital Library.

## Training Procedure

All models, except for the column classifier, are designed for pixelwise segmentation. The training of these models follows a patchwise approach, wherein the documents are divided into smaller patches and fed into the models during training. To train these segmentation models, annotated labels are required. Each sub-element in the document is assigned a unique value for identification.

If you consider examining the training process, take a look at the repository which contains the source code for training an encoder model for document image segmentation [on GitHub](https://github.com/qurator-spk/sbb_pixelwise_segmentation).


### Preprocessing

In order to use this suite of models for historical documents, no preprocessing is needed for the input images.

### Speeds, Sizes, Times

The duration of training per epoch varies, typically lasting between 2 to 5 hours, depending on the specific use case, the datasets used, and the extent of applied data augmentation.
Our ResNet-50-U-Net model has 38.15M parameters where only parameters of the decoder part are fully trained (14.71M parameters).
In the case of the column classifier, we used a ResNet-50 with two dense connected layers at the top. The column classifier model has 25.6M parameters, where only parameters of dense layers are fully trained (2.16M parameters).

### Training hyperparameters

Within the context of a constant model architecture, our hyperparameters encompass diverse augmentations, each characterized by its unique set of parameters. In addition to these, the model training process involves other hyperparameters, including the choice of the loss function, the number of batches utilized, the patch size of input images, and the number of epochs.

### Training results

Training results are reported in [this paper](https://doi.org/10.1145/3604951.3605513).

# Evaluation

Given the inadequacy of the prevailing segmentation metric in achieving optimal outcomes for document segmentation, particularly with respect to isolating regions as effectively as desired, we proceeded to evaluate our model using smaller batches of the three above-named datasets used for training. In pursuit of improved results, we employed an ensemble learning approach by aggregating the best epoch weights.

## Testing Data, Factors and Metrics

### Testing Data

Three new datasets were used for evaluation to ensure an unbiased comparison. [The PRImA Layout Analysis Dataset](https://www.primaresearch.org/datasets/Layout_Analysis) contains 478 pages of realistic documents, reflecting various challenges in layout analysis.
[The German-Brazilian Newspapers (GBN) Dataset](https://web.inf.ufpr.br/vri/databases/gbn/) comprises 152 pages from historical newspapers. We used only 61 pages to keep the complexity of the documents similar. Finally, the (unreleased) Vlaamse Erfgoedbibliotheken (VEB) dataset comprises ground truth for 75 pages from historical Belgian newspapers, split across three sets.

### Metrics

In addition to utilizing conventional performance metrics such as mean Intersection over Union (mIoU), precision, recall, and F1-score, we have incorporated the [Prima layout evaluation](https://www.primaresearch.org/tools/PerformanceEvaluation) metrics, namely Merge, Split, Miss, and the overall success rate.

# Model Examination

Examination results are reported in [this paper](https://doi.org/10.1145/3604951.3605513).

# Environmental Impact

<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->

Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).

- **Hardware Type:** GeForce RTX 2070
- **Hours used:** 2 to 5 hours per epoch
- **Cloud Provider:** No cloud.
- **Compute Region:** Germany.
- **Carbon Emitted:** More information needed.

# Technical Specifications

## Model Architecture and Objective

See [publication](https://doi.org/10.1145/3604951.3605513).

### Software

See the code published on [GitHub](https://github.com/qurator-spk/eynollah).

# Citation

<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->



**BibTeX:**

```bibtex
@inproceedings{10.1145/3604951.3605513,
author = {Rezanezhad, Vahid and Baierer, Konstantin and Gerber, Mike and Labusch, Kai and Neudecker, Clemens},
title = {Document Layout Analysis with Deep Learning and Heuristics},
year = {2023},
isbn = {9798400708411},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3604951.3605513},
doi = {10.1145/3604951.3605513},
abstract = {The automated yet highly accurate layout analysis (segmentation) of historical document images remains a key challenge for the improvement of Optical Character Recognition (OCR) results. But historical documents exhibit a wide array of features that disturb layout analysis, such as multiple columns, drop capitals and illustrations, skewed or curved text lines, noise, annotations, etc. We present a document layout analysis (DLA) system for historical documents implemented by pixel-wise segmentation using convolutional neural networks. In addition, heuristic methods are applied to detect marginals and to determine the reading order of text regions. Our system can detect more layout classes (e.g. initials, marginals) and achieves higher accuracy than competitive approaches. We describe the algorithm, the different models and how they were trained and discuss our results in comparison to the state-of-the-art on the basis of three historical document datasets.},
booktitle = {Proceedings of the 7th International Workshop on Historical Document Imaging and Processing},
pages = {73–78},
numpages = {6},
keywords = {Document layout analysis, Reading order detection, Segmentation},
location = {San Jose, CA, USA},
series = {HIP '23}
}
```

**APA:**

(Rezanezhad et al., 2023)

# More Information

SHA256 and MD5 hashes for the model *eynollah-main-regions-ensembled/2021-04-25/saved_model.pb*:

SHA256: 983b2fe2d76891acba3affe50ae45e712f9daa16b54b359022561d54b8cedd08
MD5: 70a33a786219a4eac4848301d812ecd9



# Model Card Authors

<!-- This section provides another layer of transparency and accountability. Whose views is this model card representing? How many voices were included in its construction? Etc. -->

[Vahid Rezanezhad](vahid.rezanezhad@sbb.spk-berlin.de), [Clemens Neudecker](clemens.neudecker@sbb.spk-berlin.de) and [Jörg Lehmann](joerg.lehmann@sbb.spk-berlin.de)

# Model Card Contact

Questions and comments about the model can be directed to [Vahid Rezanezhad](Vahid.Rezanezhad@sbb.spk-berlin.de), questions and comments about the model card can be directed to [Jörg Lehmann](joerg.lehmann@sbb.spk-berlin.de).

# How to Get Started with the Model

How to get started with this model is explained in the ReadMe file of the GitHub repository [over here](https://github.com/qurator-spk/eynollah). 
   
&nbsp;
    
Model Card as of August 17th, 2023