RoWeR: RoBERTa Word Error Rate Estimator for OCRed Texts

---
language:
- en
- fr
- de
- es
- it
- nl
- pt
library_name: transformers
base_model: FacebookAI/xlm-roberta-base
tags:
- ocr
- quality-estimation
- word-error-rate
- xlm-roberta
- regression
- historical-documents
- multilingual
---

RoWeR is a reference-free quality estimation model for OCRed text. It predicts the Word Error Rate (WER) of OCR output using only the OCR text itself, without requiring a manually corrected reference transcription.

RoWeR is based on XLM-RoBERTa and was adapted for WER regression using LoRA. The models were trained and evaluated on historical OCR data in seven languages:

• Portuguese
• German
• French
• Italian
• English
• Dutch
• Spanish

The main use case is OCR quality estimation at scale: deciding which OCR outputs are good enough for downstream processing, which should be sent to post-OCR correction, and which are too degraded to use reliably.

Important: output scaling

RoWeR was trained to focus on WER values in the 0.0–0.2 range. During training, this interval was linearly scaled to 0–1, with WER values above 0.2 clipped at the upper boundary.

Therefore, the model's raw regression output should be converted back to WER approximately as:

estimated_wer = clip(model_output, 0, 1) * 0.2

For example:

Model output	Estimated WER
0.10	0.02 — 2%
0.25	0.05 — 5%
0.50	0.10 — 10%
1.00	0.20 — 20% or higher

The model is intended primarily for quality estimation and triage in the 0–20% WER range. It should not be used to distinguish precisely between highly degraded texts above that range.

Available models

The repository contains both language-specific and multilingual RoWeR models.

Multilingual models

Folder	Description
RoWer-multi-lingual-balanced	Multilingual model trained with balanced language sampling
RoWer-multi-lingual-unbalanced	Multilingual model trained using the natural training-data distribution

For general use, we recommend starting with a multilingual model. In the paper, multilingual training substantially reduced the cross-language calibration problems observed with some language-specific models, while the balanced and unbalanced variants performed similarly overall.

Language-specific models

Folder	Language	Training corpus
RoWeR-RETAS_english	English	RETAS
RoWeR-RETAS_french	French	RETAS
RoWeR-RETAS_german	German	RETAS
RoWeR-RETAS_spanish	Spanish	RETAS
RoWeR-dbnl_dutch	Dutch	DBNL
RoWeR-pocr_english	English	POCR
RoWeR-pocr_french	French	POCR
RoWeR-pocr_german	German	POCR
RoWeR-pocr_italian	Italian	POCR
RoWeR-porto_portuguese	Portuguese	PORTO

Where multiple models exist for the same language, the variants were trained on different OCR corpora and therefore represent different domains and OCR-noise distributions.

Repository structure

Each model variant contains a LoRA checkpoint and a merged model:

<Model folder>/
├── checkpoint-final/
└── final_merged_model/

For the simplest inference setup, use final_merged_model.

The merged model folders contain the trained XLM-RoBERTa regression model. The tokenizer should be loaded from the original base model:

FacebookAI/xlm-roberta-base

Installation

pip install torch transformers

Quick start

The following example uses the balanced multilingual model.

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

REPO_ID = "TomasOsorio/RoweR"
MODEL_SUBFOLDER = "RoWer-multi-lingual-balanced/final_merged_model"
BASE_MODEL = "FacebookAI/xlm-roberta-base"

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)

model = AutoModelForSequenceClassification.from_pretrained(
    REPO_ID,
    subfolder=MODEL_SUBFOLDER,
)
model.to(device)
model.eval()

ocr_text = """
Th1s is an examp1e of OCRed text with recogn1tion errors.
"""

inputs = tokenizer(
    ocr_text,
    return_tensors="pt",
    truncation=True,
    max_length=256,
)
inputs = {key: value.to(device) for key, value in inputs.items()}

with torch.no_grad():
    output = model(**inputs).logits.squeeze().item()

# RoWeR was trained with WER [0, 0.2] scaled to [0, 1].
normalized_score = max(0.0, min(1.0, output))
estimated_wer = normalized_score * 0.2

print(f"Estimated WER: {estimated_wer:.4f}")
print(f"Estimated word error percentage: {estimated_wer * 100:.2f}%")

Training data

RoWeR was developed using four OCR corpora:

• PORTO — historical Portuguese OCR data
• DBNL — historical Dutch texts from the Digital Library of Dutch Literature
• RETAS — English, French, German, and Spanish OCR data
• POCR — large-scale English, French, German, and Italian OCR data

The training pipeline aligned OCR outputs with reference transcriptions, computed segment-level WER, and applied quality filters to remove unreliable or anomalous samples.

Model architecture

RoWeR uses:

• FacebookAI/xlm-roberta-base
• a sequence-level regression head
• LoRA-based parameter-efficient fine-tuning

At inference time, the model takes OCR text as input and outputs a single regression value representing estimated OCR degradation.

RoWeR does not require:

• the source document image
• OCR-engine confidence scores
• a second OCR system
• a manually corrected reference transcription

Intended uses

RoWeR is intended for tasks such as:

• large-scale OCR quality assessment
• document triage before downstream NLP
• deciding whether post-OCR correction is likely to be necessary
• filtering low-quality OCR text
• prioritising documents for manual review
• monitoring OCR quality across large digital collections

RoWeR estimates OCR quality; it does not correct OCR errors.

Citation

If you use RoWeR in your work, please cite:

@InProceedings{10.1007/978-3-032-36033-5_1,
author="Os{\'o}rio, Tom{\'a}s Freitas and Cardoso, Henrique Lopes",
title="RoWeR: RoBERTa Word Error Rate Estimator for OCRed Texts",
booktitle="Document Analysis and Recognition -- ICDAR 2026",
year="2027",
publisher="Springer Nature Switzerland",
address="Cham",
pages="3--18",
}