Zone detection model for (17th–18th c.) manuscripts, trained on Leibniz's manuscripts using SegmOnto ontology.
This model was developed by Denisa-Florina BUMBA as part of the ERC Philiumm project (Grant ADG No. 101020985), at the Laboratoire SPHERE, Université Paris Cité – CNRS, France.
Fine-tuned RF-DETR-Seg (Nano) for instance segmentation and zone detection on scanned pages of Leibniz's manuscripts. The model detects zones such as main text, margin text, numbering, diagrams, digitization artifacts, and several classes of mathematical formulae using the SegmOnto controlled vocabulary, in order to support automatic transcription in the digital humanities. It is trained specifically on Leibniz's manuscripts, but may be applied to other historical manuscripts with similar layout.
Dataset
Available at : rfdetr-segmentation-leibniz-dataset
Classes
Class names follow the SegmOnto controlled vocabulary for manuscript layout annotation, which is widely used across the HTR/OCR and the digital humanities community (Kraken, eScriptorium, HTR-United).
| Class | Description |
|---|---|
MainZone |
Main body text region |
MarginTextZone |
Marginal additions |
NumberingZone |
Page numbering |
DigitizationArtefactZone |
Digitization artefacts |
GraphicZone-figure |
Diagrams, other graphic zones |
GraphicZone-formula |
Covers all mathematical formulae |
GraphicZone-formula-complex |
Complex/multi-line formulae |
GraphicZone-formula-inline |
Inline formulae within text |
GraphicZone-formula-strikethrough |
Struck-through/deleted formulae |
Training Data
| Split | Images | Percentage |
|---|---|---|
| Train | 1,026 | 94% |
| Validation | 40 | 4% |
| Test | 20 | 2% |
| Total | 1,086 | 100% |
Preprocessing:
- Auto-orient applied
- Resized (stretch) to 728×728
Augmentation (applied to the training set only, 3 augmented outputs generated per source image):
- Horizontal and vertical flip
- Rotation: ±3°
- Shear: ±5° horizontal, ±5° vertical
- Hue: ±15°
- Brightness: ±15%
- Noise: up to 1.49% of pixels
Training Parameters
| Parameter | Value |
|---|---|
| Model | RF-DETR Seg Nano |
| Epochs | 80 |
| Batch size | 2 |
| Gradient accumulation steps | 8 |
| Learning rate | 1e-4 |
| Encoder learning rate | 1.5e-4 |
| LR scheduler | Cosine |
| Warmup epochs | 3 |
| Weight decay | 1e-4 |
| Resolution | 720 px (closest valid value to the 728×728 preprocessing target) |
| EMA | Enabled |
| Early stopping | Disabled |
| Checkpoint interval | Every 5 epochs |
| Seed | 42 |
The published weights are from epoch 43, the best raw checkpoint (segm_mAP_50:95 = 0.415), and from the best EMA checkpoint, at epoch 49 (segm_mAP_50:95 = 0.410).
Performance Metrics
| Checkpoint | Epoch | segm mAP@50:95 |
|---|---|---|
checkpoint_best_regular.pth |
43/80 | 0.4154 |
checkpoint_best_ema.pth |
49/80 | 0.4102 |
checkpoint_best_regular.pth (epoch 43) — Overall metrics
| Metric | Value |
|---|---|
| mAP@50 | 0.612 |
| mAP@50:95 | 0.438 |
| mAP@75 | 0.460 |
| mAR | 0.530 |
| Precision | 0.684 |
| Recall | 0.639 |
| F1 | 0.618 |
| Segm mAP@50 | 0.608 |
| Segm mAP@50:95 | 0.415 |
checkpoint_best_regular.pth (epoch 43) — Per-class AP (detection)
| Class | AP |
|---|---|
| DigitizationArtefactZone | 0.936 |
| MainZone | 0.890 |
| MarginTextZone | 0.575 |
| NumberingZone | 0.494 |
| GraphicZone-figure | 0.484 |
| GraphicZone-formula-complex | 0.271 |
| GraphicZone-formula-inline | 0.111 |
| GraphicZone-formula | 0.106 |
| GraphicZone-formula-strikethrough | 0.073 |
checkpoint_best_ema.pth (epoch 49) — Overall metrics
| Metric | Value |
|---|---|
| EMA mAP@50 | 0.612 |
| EMA mAP@50:95 | 0.437 |
| EMA mAR | 0.541 |
| EMA Segm mAP@50 | 0.603 |
| EMA Segm mAP@50:95 | 0.410 |
Per-class AP is not available for checkpoint_best_ema.pth. Only per-class metrics were logged for the regular checkpoint during training.
Note on class imbalance: the model performs well on structural zones (main text, margin text, graphic zones, numbering) but is substantially weaker on the smaller formula classes (-inline, -strikethrough).
How to Get Started with the Model
from rfdetr import RFDETRSegNano
import cv2
from rfdetr.assets.coco_classes import COCO_CLASSES
model = RFDETRSegNano(pretrain_weights="checkpoint_best_regular.pth")
image = cv2.imread("image.jpg")
detections = model.predict(image, threshold=0.4)
labels = [f"{COCO_CLASSES[class_id]}" for class_id in detections.class_id]
annotated_image = sv.MaskAnnotator().annotate(detections.metadata["source_image"], detections)
annotated_image = sv.LabelAnnotator().annotate(annotated_image, detections, labels)
cv2.imwrite("image_annotated.jpg", annotated_image)
Replace "image.jpg" with the path to your manuscript image.
The threshold can be changed, but we noticed that we have fair predictions with a value of 0,4. Under this value we start having overlapping regions and over this value we loose some correct predictions with low confidence score.
eScriptorium Pipeline
A preprocessing pipeline for preparing model outputs for import into eScriptorium is available here:
Pipeline RF-DETR-Seg to XML PAGE
It converts this model's zone predictions into the XML-Page format compatible with eScriptorium.
Limitations
- Fine-tuned on manuscripts from a single author (16th–early 17th c.). Generalization to other authors or periods has not been tested.
- Performance depends on image quality.
GraphicZone-formulais a generic label for formulae that weren't sorted into one of the three more specific classes (-complex,-inline,-strikethrough). Since annotations came from different practices and different annotators, not all formulae were manually reclassified. This is a known issue we plan to fix in a future version of the dataset.
Funding
This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme, Advanced Grant ADG No. 101020985.