Pretrained multilingual Party model
Creators: name: Benjamin Kiessling
Publisher: name: Zenodo
Year: 2025
Subjects: automatic-text-recognition
multilingual
multiscriptal
multimodal

description: Llama Party

Party is page-wise recognition of text-y. It is a replacement for
conventional text recognizers in automatic text recognition pipelines that
utilize either bounding box or baseline+bounding polygon segmentation methods
for layout analysis.

Llama party is a full-page generative text recognizer that has been pretrained
on a large corpus of multilingual historical, contemporary, and born-digital
document page images, both handwritten and machine-printed.

Architecture

The recognizer is a deep fusion multimodal model consisting of a Swin vision
encoder and a tiny Llama (100M parameters) decoder trained with octet
tokenization. The network is prompted with the line positions through
positional embeddings added to the encoder hidden state. 

During training the encoder weights were initialized with a ImageNet-22k
pretrained Swin-base from pytorch-image-models, the decoder weights came from a
custom Llama 3.2 pretrained on a
subset of OSCAR 2301
tokenized with a ByT5-style octet tokenizer.

The pre-initialized model was then pre-trained on a collection of public and
private training historical document page datasets augmented with born-digital
data crafted from PubLayNet.

Uses

Llama party is a recognition foundation model primarily targeted at automatic
text recognition for the humanities. While it produces fairly accurate output
on an impressive range of material it is intended to be fine-tuned on some
target dataset to ensure compliance with desired transcription guidelines.

Transcription guidelines, Normalization, and Transformations

No attempts have been made to normalize the datasets or to only use data
adhering to common transcription guidelines. While some subsets of the corpus
are internally consistent, only a very small subset of the languages in the
training data only contain datasets from a single source.

Bias, Risks, and Limitations

The training corpus is heavily skewed towards a couple of languages (Chinese,
English, French, German, and Portuguese) and frequently incorporates datasets
of esoteric material transcribed for specific purposes. Especially
machine-printed and born-digital material lack diversity, so error rates will
most likely vary considerably across languages and document type.

Some additional limitations are to be expected:



Some of the training data are Latin transliterations of Yiddish and Ottoman
  Turkish. Arabic and Hebrew script recognition is likely to require
  fine-tuning.

Some transcriptions resolved abbreviations while others did not. Inconsistent
  output is to be expected, in particular for European manuscripts in Latin
  script.

As the model predicts 8-bit UTF-8 code units directly the lack of consistent
  Unicode normalization can cause slightly different code point streams during
  prediction.


How to Get Started with the Model

Install the party package from github and follow the instructions.

Training Details

Training Data

The model has been pretrained on the vast majority of publicly available ATR
datasets, in addition to a decent number of restricted datasets. For English
exclusively we converted the PubLayNet dataset for layout analysis on
born-digital documents into an ATR dataset with
PDFMiner and some basic
baseline heuristic based on the line bounding box.

|Language|Pages|Lines|Datasets|
|---|---|---|---|
|Arabic|   |   |RASAM 1TariMaOpenITI Arabic MS DataOpenITI Arabic Print Data|
|Catalan|   |   |FONDUE-CA-PRINT-20|
|Chinese|   |   |1 large private dataset|
|Corsican|   |   |HN2021-OCR-Poesie-Corse|
|Czech|   |   |Padeřov-Bible-handwriting-ground-truth|
|Dutch|   |   |ATR_TrainingSet_NLF_Newseye_GT_SV_M2+4 private manuscript datasetsVOC dataset|
|English|   |   |FONDUE-EN-PRINT-20PubLayNetUniversity of Denver CollectionsJoseph Hooker HTRCCCC MS 41|
|Finnish|   |   |NewsEye/READ OCR Finnish Newspapers|
|French|   |   |NewsEye READ AS French NewspapersBoccaceFabliauxLiberCremma MedievalDecameronFRFONDUE-FR-MSS-18FONDUE-FR-MSS-19FONDUE-FR-PRINT-16FONDUE-FR-PRINT-17FONDUE-FR-PRINT-20Données imprimés gothiques du 16e siècleDonnées HTR incunables du 15e siècleDonnées HTR manuscrits du 15e siècle"Tables Décennales" French Civil RegistryDonnées imprimés du 16e siècleDonnées imprimés du 17e siècleDonnées imprimés du 18e siècleIncunable français du 15e siècleHTRomanceHTR-SETAF-Jean-MichelHTR-SETAF-LesFaictzJCHHTR-SETAF-Pierre-de-VingleLa Correspondance Jacques Doucet - René JeanOCR17+Tapus CorpusTIMEUS CorpusRecensement Valaisan3 private handwritten and print datasets|
|Georgian|   |   |1 private dataset|
|German|   |   |Charlottenburger AmtsschrifttumDACH GTDigiTue GTFibelnFONDUE-DE-MSS-18FoNDUE_Wolfflin_FotosammlungHKB GTGround truth for Neue Zürcher Zeitung black letterReichsanzeiger GTStABS Ratsbücher O10NewsEye / READ OCR Austrian NewspapersWeisthuemer3 private manuscript datasets|
|Greek|   |   |EPARCHOSHTR CPgr23Handwritten Paleographic Greek Text RecognitionΧΦ114XΦ79ΧΦ5310 small private manuscript datasets|
|Hebrew|   |   |Tikkoun SofrimBiblIA|
|Italian|   |   |episearch-htrFONDUE-IT-PRINT-20HTRomance Italian1 private print dataset|
|Japanese|   |   |mm-ocr-dataset-v1|
|Latin|   |   |Caroline MinusculeCREMMA-Medieval-LATHTRomance LatinDIVA-HisDBEutychesFONDUE-LA-MSS-MAFONDUE-LA-PRINT-16Lateinische GedichteWien ÖNB Cod 21602 private manuscript datasets|
|Multilingual|   |   |FONDUE-MLT-ART[FONDUE-MLT-CAT](https://github.com/FoNDUE-HTR/FONDUE-MLT-CAT)[FONDUE-MLT-PRINT-TEST](https://github.com/FoNDUE-HTR/FONDUE-MLT-PRINT-TEST)gt_structure_text|
|Ottoman Turkish|   |   |OpenITI Arabic MS DataOpenITI Arabic Print Data1 private manuscript dataset|
|Farsi|   |   |OpenITI Arabic MS DataOpenITI Arabic Print Data|
|Portuguese|   |   |Portuguese Handwriting 16th-19th c.|
|Russian|  |  |  |1 private manuscript dataset|
|Spanish|   |   |FONDUE-ES-PRINT-19FoNDUE-Spanish-chapbooks-DatasetHTR AraucaniaHTRomance Spa3 private manuscript datasets|
|Syriac|   |   |2 private print and manuscript datasets|
|Urdu|   |   |OpenITI Arabic MS DataOpenITI Arabic Print Data|
|Yiddish|   |   |1 private print datasets|

Training Procedure and Hyperparameters



Training regime:: 6 * A40 GPU, BF16 precision, Mars-AdamW optimizer with caution, batch size: 32, gradient accumulation: 4, effective batch size: 768, 5 epochs with 5000 iteration warmup and cosine decay, max LR 5e-4, min LR 5e-6 at end of epoch 5, weight decay 1e-5, gradient clipping 1.0, augmentation, random sampling of bbox and curve batches


Evaluation

Testing Data, Factors & Metrics

Testing Data



{{ testing_data | default("[More Information Needed]", true)}}

Factors



{{ testing_factors | default("[More Information Needed]", true)}}

Metrics

CER:
WER: 

Summary

{{ results_summary | default("", true) }}

Citation [optional]

BibTeX:

{{ citation_bibtex | default("[More Information Needed]", true)}}