# Using LLMs for post-OCR correction on historical French texts: A case study using synthetic data

Mikhail Biriuchinskii, Motasem Alrahabi, Glenn Roe

<figure><img src="image_1.png" /></figure>

## To cite this version:

Mikhail Biriuchinskii, Motasem Alrahabi, Glenn Roe. Using LLMs for post-OCR correction on historical French texts: A case study using synthetic data. DH2025 CFP - Digital Humanities Conference 2025, Jul 2025, Lisbonne, Portugal. hal-04987584

HAL Id: hal-04987584

https://hal.science/hal-04987584v1

Submitted on 12 Mar 2025

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>
# Using LLMs for post-OCR correction on historical French texts:
A case study using synthetic data

Mikhail Biriuchinskii, Motasem Alrahabi, Glenn Roe

L'Observatoire des textes, des idées et des corpus (ObTIC), Sorbonne University
prenom.nom@sorbonne-universite.fr

## Abstract

In the wake of generative AI, new methods have arisen to ensure that Optical Character Recognition (OCR) remains coherent and usable in the creation of better text corpora across a variety of domains and historical periods. This short paper investigates one such effort: the ability of large language models (LLMs) to amend poor transcriptions of 19th-century French monographs. We evaluate the state-of-the-art Llama-3.2-Instruct model, known for its advanced instruction-following capabilities and adaptability to domain-specific fine-tuning, on both real and synthetic data derived from the ICDAR 2017 corpus, enhanced using the 'ScrambledText' module to introduce realistic OCR errors. Although advanced in instruction-following, the results reveal challenges in generalization, with fine-tuned models increasing CER errors, highlighting limitations in LLMs for handling character-level OCR corrections.

**Keywords:** OCR, large language model, synthetic data generation, historical text, digital humanities.

## Introduction

Automated text recognition systems have recently achieved remarkable accuracy, with character error rates (CER) as low as 0.01% (Reul et al., 2018). However, transcription quality remains a persistent challenge for degraded or complex layouts, particularly in historical documents. These often feature ornate fonts, physical degradation, and unconventional page layouts, making OCR particularly error-prone.

As part of the Valentin Haüy Digital Library project<fnref n="1" />, our goal is to establish a pipeline for the digitization and OCR of texts, as well as their correction and enriched digital editing. In this paper, we focus on the issue of post-OCR correction.

## State of the art

Three main approaches to post-OCR correction are prominent: lexical methods, machine learning techniques, and language models (Petkovic et al., 2022). Lexical approaches are declining, but machine learning and pre-trained models show promise.

Pre-trained LLMs have garnered interest. Thomas et al. (2024) showed that fine-tuned Llama-2 models reduced CER by 50% in the British newspaper dataset BLN600. Boros et al. (2024) reported mixed results with pre-trained models evaluated with the Post-Correction Improvement Score (PCIS). They noted quality degradation across models and prompts but slight improvements with sentence-level tokenization and zero-shot prompts with explicit instructions for the GPT-4 model.

Jonathan Bourne (2024) emphasized diverse training data for OCR correction. Analyzing OCR error distributions with a character-level Markov corruption process, he developed the 'ScrambledText' Python library to simulate OCR errors. Bourne found that augmenting datasets with synthetic errors improved performance on real-world data.

<footnote n="1">¹ https://obtic.sorbonne-universite.fr/projet/bvh/</footnote>
Our study builds on these insights to explore the application of advanced LLMs for post-OCR correction, focusing on the unique challenges presented by French historical texts.

OCR evaluation metrics vary, sometimes incorporating semantic or morphosyntactic features based on evaluation needs (Roux et al., 2022). This study used WER, CER, and PCIS (Boros et al., 2024). PCIS (-1 to 1) reflects improvement (positive) or degradation (negative).

## Dataset & Preprocessing

The dataset<fnref n="2" /> used is from the ICDAR 2017 competition corpus (Chiron et al., 2017a; 2017b), containing historical OCR transcriptions aligned with ground truth (GT). Texts were filtered by publication year (19th century), type (monograph), and language (French).

After character-level preprocessing<fnref n="3" />, OCR and GT texts were tokenized with pySBD (Sadvilkar & Neumann, NLPOSS 2020), ensuring identical phrase splits. A similarity threshold of 85% was applied via the 'FuzzyWuzzy' module to verify sentence pairing.

Table 1 summarizes the dataset, detailing sizes of training, development, and test sets with CER and WER medians. A filter limited zero CER cases to 15%, ensuring models learned to

preserve accurate OCR outputs. Consequently, 89% of cases showed 0-0.03 CER between OCR and GT pairs.

<table><caption>Table 1: Summary of the dataset's characteristics</caption><thead><tr><th></th><th>Train</th><th>Dev</th><th>Test</th></tr></thead><tbody><tr><th>Phrases</th><td>2.63k</td><td>336</td><td>301</td></tr><tr><th>Characters</th><td>434k</td><td>48k</td><td>46k</td></tr><tr><th>CER</th><td>0.0126</td><td>0.0147</td><td>0.0139</td></tr><tr><th>WER</th><td>0.0552</td><td>0.0632</td><td>0.0621</td></tr></tbody></table>

To increase CER diversity, the 'ScrambledText' Python module introduced realistic OCR errors based on observed error probabilities. This created a 'Sentence_OCR_corrupted' version, raising CER in 51% of sentences to 0.07 - 0.11.

## Case Study

We used the Llama-3.2-3B-Instruct model for OCR correction with a resource-constrained setup using Low-Rank Adaptation (LoRA) for efficient fine-tuning. Experiments were conducted on two instruction datasets: "natural" with authentic OCR errors and "synthetic" augmented with simulated errors from the 'ScrambledText' library.

<table><caption>Table 2: Prompt translation from French</caption><thead><tr><th>Prompt template</th></tr></thead><tbody><tr><td>You are performing a task to correct OCR-processed text.<br>1. Correct OCR errors (missing letters, accents) in the text provided from the Gallica digital library.<br>2. Preserve the 19th-century style and language, avoiding modernization.<br>3. Do not alter line break markers (e.g., seu-lement, oc-tobre).<br>4. Return the corrected text only between the tags 'RESULT'</td></tr></tbody></table>

This approach evaluated the impact of different training data on model performance.

<footnote n="2">²https://huggingface.co/datasets/m-biriuchinskii/ICDAR2017-filtered-1800-1900-6. See also https://sites.google.com/view/icdar2017-postcorrectionocr and https://tc11.cvc.uab.es/datasets/Post-OCR%202017_1.</footnote>

<footnote n="3">³ Symbols such as "@" mark alignment points, while "#" represents uncertainties in the dataset. Since no images of the original documents were included, we first preprocessed the GT by replacing "#" symbols with the corresponding OCR characters to maintain input coherence. This ensures that LLMs can identify and learn patterns in corrupted text, leaving untouched the corrupted text difficult to reconstruct.</footnote>
Task-specific details were included in prompts to improve contextual understanding, and formatting constraints required wrapping corrected text in XML-style tags.

Using LoRA, we adapted the model efficiently without updating all parameters, reducing computational demands. Fine-tuning employed 4-bit quantization for memory efficiency with Hugging Face TRL's SFTTrainer, evaluating every 30 steps and applying early stopping after 7 steps without improvement.

3 models were prepared for this study:

- **Base model** trained on “natural” errors of the original dataset;

- **Synth model** trained on phrases with “synthetic” errors;

- **Mixed model** trained on 80% “natural” and 20% “synthetic” errors.

## Results

All three final models showed negative PCIS scores, suggesting that the fine-tuning process did not improve the OCR output:

<figure><img src="image_3.png" /><figcaption>Figure 1: Average of post-correction improvement score (PCIS) per model and dataset evaluation type</figcaption></figure>

Figure 2 shows that the best-performing model, which was fine-tuned on and tested with real data, did not achieve significant improvements in terms of CER and WER, indicating that the LLMs were unsuccessful in reducing these error rates for this dataset.

These results highlight the inherent challenges of generalization that arise when using LLMs to correct OCR output on historical documents. Anecdotally, LLMs – especially when used through the chatbot interfaces they power – are capable of OCR text correction at a fine-grained level, but even then can be prone to overcorrection and hallucinations. At the scale of large corpora, LLMs still fall short of surpassing the state-of-the-art in post-OCR correction, particularly for historical texts.

## Perspectives & limitations

This study illustrates the difficulties of achieving positive results in real-world conditions, where resource constraints often prevent the full potential of advanced techniques from being realized. Although this result does not have an immediate impact on our project (since the expected digital edition at the end of the project will still require manual correction after the automatic process), we aim to focus in the near future on optimizing training strategies and exploring alternative approaches. This includes developing a custom dataset<fnref n="4" />, integrating more sophisticated error simulations, or leveraging greater computational resources.

## Data availability

The code and data resources used in this work are available at:
https://github.com/obtic-sorbonne/ocr correcti on.

<footnote n="4">⁴ Our project partners provide large volumes of OCR-processed and manually corrected data. This data will be valuable for training after proper alignment. The dataset will be available by the time the paper is presented.</footnote>
<figure><img src="image_4.png" /><figcaption>Figure 2: Comparison of Average Model Character Error Rates and Word Error Rates</figcaption></figure>

## References

Alan Thomas, Robert Gaizuskas, and Haiping Lu. 2024. Leveraging LLMs for Post-OCR Correction of Historical Newspapers. In Proceedings of the Third Workshop on Language Technologies for Historical and Ancient Languages (LT4HALA) @ LREC-COLING-2024, pages 116–121, Torino, Italia. ELRA and ICCL.

Emanuela Boros, Maud Ehrmann, Matteo Romanello, Sven Najem-Meyer, and Frédéric Kaplan. 2024. Post-Correction of Historical Text Transcripts with Large Language Models: An Exploratory Study. In Proceedings of the 8th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (LaTeCH-CLfL 2024), pages 133–159, St. Julians, Malta. Association for Computational Linguistics.

Guan, S., & Greene, D. (2024). Advancing Post-OCR Correction: A Comparative Study of Synthetic Data. ArXiv, abs/2408.02253.

Guillaume Chiron, Antoine Doucet, Mickaël Coustaty, and Jean-Philippe Moreux. 2017a. ICDAR2017 Competition on Post-OCR Text Correction. In 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), volume 01, pages 1423–1428.

Guillaume Chiron, Antoine Doucet, Mickaël Coustaty, Muriel Visani, and Jean-Philippe Moreux. 2017b. Impact of OCR errors on the use of digital libraries: Towards a better access to information.

In Proceedings of the 17th ACM/IEEE Joint Conference on Digital Libraries, pages 249–252, Toronto, ON, Canada. IEEE, IEEE Press.

Jonathan Bourne. 2024. Scrambled text: training Language Models to correct OCR errors using synthetic data. arXiv preprint arXiv:2409.19735. Ljudmila Petkovic, Motasem Alrahabi, Glenn Roe. Impact de la correction automatique de l'OCR/HTR sur la reconnaissance d'entités nommées dans un corpus bruité. JIS - Journal of Information Sciences, 2022, 21 (2), pp.42-57. (10.34874/IMIST.PRSM/jis-v21i2.36599) (hal-04063970).

Nipun Sadvilkar and Mark Neumann. 2020. PySBD: Pragmatic Sentence Boundary Disambiguation. In Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS), pages 110–114, Online. Association for Computational Linguistics.

Reul, C., Springmann, U., Wick, C., & Puppe, F. 2018. State of the Art Optical Character Recognition of 19th Century Fraktur Scripts using Open Source Engines. ArXiv, abs/1810.03436.

Thibault Bañeras Roux, Mickael Rouvier, Jane Wottawa, Richard Dufour. Mesures linguistiques automatiques pour l'évaluation des systèmes de Reconnaissance Automatique de la Parole. 29ème Conférence sur le Traitement Automatique des Langues Naturelles (TALN), Jun 2022, Avignon, France. (hal-03688029v2).