CaliDoc: WildReceipt

LayoutLMv3-Large fine-tuned for key information extraction on WildReceipt, plus the CaliDoc calibration artifacts fitted for this exact checkpoint. Paper: CaliDoc: Calibrated Field-Level Risk and Annotation Auditing for Document Information Extraction, WACV 2027 (Evaluations and Datasets Track). Code: https://github.com/MHHamdan/CaliDoc · Supplement: PDF · Collection: CaliDoc — WACV 2027

Token-level reliability is not complete-field reliability. CaliDoc calibrates the object a downstream system consumes: the complete field.

Model Description

Extractor. LayoutLMv3ForTokenClassification (microsoft/layoutlmv3-large), 49 BIO labels, trained on receipts photographed in the wild (WildReceipt), 24 key/value field types on OCR boxes (Others mapped to O, Ignore boxes dropped). This is a dataset-specific checkpoint, and CaliDoc does not provide a universal model. Each dataset has its own repository in the collection.
Checkpoint. This repository contains the seed-42 canonical checkpoint, the one the paper uses for every ablation, SpanCRC and audit result. Headline WACV results report mean ± standard deviation over three training seeds (42, 123, 456). Those aggregates are not metrics of this single checkpoint.
Calibration (post hoc, no retraining). Temperature scaling followed by field-conditional SpanCRC, a conformal risk control over max-token span scores. The neural extractor and the calibration are separate: the weights are a standard Transformers model, and the thresholds in calibration/ turn its probabilities into calibrated label sets.

Intended Use

Research on uncertainty quantification, conformal prediction and annotation auditing for document information extraction.
Reproducing and extending the WACV 2027 results.
Prototyping human-in-the-loop review routing on documents from the same distribution as WildReceipt.

Out-of-Scope Use

Commercial use: the weights inherit CC BY-NC-SA 4.0 from LayoutLMv3.
Documents from other domains, layouts, languages or OCR engines. The calibration guarantee assumes exchangeability with the calibration documents and does not transfer to out-of-domain inputs.
Autonomous decisions in consequential workflows (finance, legal, medical, identity). CaliDoc is meant to decide what a human should check, not to replace that check.
Treating calibrated-set status as a per-field probability of correctness.

How CaliDoc Works

Input: a document image, plus OCR words and word boxes on LayoutLMv3's 0–1000 grid. OCR is not included: the paper uses the dataset-provided OCR.
Extractor: LayoutLMv3 token classification. Each word is scored by its first sub-word, and probabilities are softmax(logits / T) with T = 1.1144.
Calibration: the calibrated set of word i is C(x_i) = { y : 1 − p(y|x_i) ≤ λ_type(y) }. A gold field is fully covered exactly when its worst word clears the threshold, which makes this a max-token complete-field score.
Reliability: taking the document as the exchangeable unit, SpanCRC bounds the expected per-document fraction of missed gold fields by α = 0.1. The bound holds on fresh documents exchangeable with the calibration documents that contain at least one gold field.
Audit (corpus level): at the pooled threshold, a boundary-escape test asks whether threshold escapes concentrate at field edges, which is a symptom of alignment and annotation errors. It runs over a calibration corpus with Holm control and a document-blocked bootstrap. It is a dataset diagnostic, not a per-document error detector, and it is not part of single-document inference.

WildReceipt is CaliDoc's easiest case: most field types have enough support for their own threshold, and calibrated sets stay close to singletons (paper: average set size 0.93, span coverage 0.913). Most WildReceipt fields are one or two words long, so the boundary-escape audit has almost no power here. In the paper only 2 of 22 field types were eligible for the test, and a clean audit result on short fields means untested, not clean.

Quick Start

pip install "calidoc @ git+https://github.com/MHHamdan/CaliDoc"

Plain Transformers (the extractor only):

from transformers import AutoModelForTokenClassification, AutoProcessor

model_id = "MHamdan/calidoc-layoutlmv3-large-wildreceipt"
processor = AutoProcessor.from_pretrained(model_id, apply_ocr=False)
model = AutoModelForTokenClassification.from_pretrained(model_id)
print(model.config.id2label)

CaliDoc calibrated extraction, given an image and an OCR JSON {"words": [...], "boxes": [[x0, y0, x1, y1], ...], "box_format": "pixel" | "normalized_1000"}:

import json
from PIL import Image
from calidoc.inference import CaliDocPredictor, normalize_boxes

predictor = CaliDocPredictor.from_pretrained("MHamdan/calidoc-layoutlmv3-large-wildreceipt")
image = Image.open("document.png").convert("RGB")
ocr = json.load(open("ocr.json"))
boxes = ocr["boxes"] if ocr["box_format"] == "normalized_1000" else normalize_boxes(ocr["boxes"], *image.size)
result = predictor.predict(image, ocr["words"], boxes)
for field in result["fields"]:
    print(field["type"], "|", field["text"], "|", field["status"], field["set_sizes"])
print(result["summary"]["route"])

or from the command line:

python -m calidoc.predict --model MHamdan/calidoc-layoutlmv3-large-wildreceipt --image document.png --ocr ocr.json --out prediction.json

This model's training data is not cleared for redistribution, so no example image is shipped. The GitHub repository has a CORD example with the same input format, and assets/examples/wildreceipt/ holds this model's output on a selected test document with the text removed.

Each predicted field gets one of these statuses:

status	meaning	suggested action
`covered`	every word's calibrated set is exactly the predicted label	pass downstream
`ambiguous_review`	the predicted label is in every set, but some set also admits other labels	validate / review
`conflict_review`	some word's set is non-empty but excludes the predicted label	review
`empty_abstain`	some word's set is empty	abstain / review

unassigned_field_candidates lists words predicted O whose set still admits a field label, which are possible missed fields. These statuses are properties of calibrated prediction sets, not confidence scores.

Calibration Artifacts

file	content
`calidoc_config.json`	α = 0.1, m0 = 25, T, pooled λ, supported/fallback field types, released-checkpoint test metrics
`calibration/temperature.json`	T = 1.114367, fitted by NLL on the `val_es` early-stopping split
`calibration/pooled_threshold.json`	pooled SpanCRC λ = 0.147488 and the token-marginal q̂ for comparison
`calibration/field_thresholds.json`	λ per field type with its calibration support
`calibration/split_manifest.json`	calibration document IDs (identifiers only, no data)

The thresholds were fitted on calibration split seed 42 (the paper's first calibration unit) and match the committed paper artifacts exactly. Field types with fewer than m0 = 25 calibration spans fall back to the pooled λ, so they inherit only the pooled document-level bound and get no per-type guarantee.

field type	λ	calibration spans	threshold
`Date_key`	0.1264	69	own threshold
`Date_value`	0.0214	313	own threshold
`Prod_item_key`	0.4349	73	own threshold
`Prod_item_value`	0.0894	1617	own threshold
`Prod_price_key`	0.9405	73	own threshold
`Prod_price_value`	0.0491	1500	own threshold
`Prod_quantity_key`	0.1326	57	own threshold
`Prod_quantity_value`	0.0648	1037	own threshold
`Store_addr_key`	0.1475	0	pooled fallback
`Store_addr_value`	0.1056	493	own threshold
`Store_name_key`	0.1475	0	pooled fallback
`Store_name_value`	0.2226	415	own threshold
`Subtotal_key`	0.2125	241	own threshold
`Subtotal_value`	0.3262	260	own threshold
`Tax_key`	0.1924	266	own threshold
`Tax_value`	0.2451	264	own threshold
`Tel_key`	0.0349	99	own threshold
`Tel_value`	0.0517	210	own threshold
`Time_key`	0.2381	43	own threshold
`Time_value`	0.0418	292	own threshold
`Tips_key`	0.9077	25	own threshold
`Tips_value`	0.9903	25	own threshold
`Total_key`	0.6190	338	own threshold
`Total_value`	0.6070	405	own threshold

Evaluation

Paper: three training seeds (headline, mean ± s.d. over seeds 42/123/456, full test split):

metric	value
entity F1	94.38 ± 0.27

Paper: CaliDoc on this seed-42 checkpoint (90% target, α = 0.10, mean over three calibration splits; the paper's SpanCRC table):

	token-marginal threshold	field-conditional SpanCRC
realized field-miss risk (the bounded quantity)	0.093	0.100
full-span coverage (empirical)	0.918	0.913
average set size	0.93	0.93

This released checkpoint and calibration (computed from the release artifacts):

metric	value
test entity F1 / precision / recall	94.04 / 92.80 / 95.31 (472 test documents)
test ECE (after temperature)	1.83%
SpanCRC on the evaluation side of the released calibration split: field-miss risk / span coverage / set size	0.097 / 0.912 / 0.94 (3601 gold spans)
same split, token-marginal threshold: token / span coverage	0.889 / 0.910

The released weights reproduce the paper pipeline's cached probabilities to within 3e-5 and the recorded test F1 exactly (see docs/release/HF_MODEL_ARTIFACT_AUDIT.md in the GitHub repository).

Limitations

Dataset and domain specific. One checkpoint per benchmark. Results do not transfer to other document types, and the guarantee assumes exchangeability with the calibration documents.
English-script benchmarks only (CORD receipts are Indonesian with mostly Latin-script menu text). One backbone family (LayoutLMv3).
OCR dependence. Inputs are OCR words and boxes, and the paper uses dataset-provided OCR. A different OCR engine changes tokenization and boxes and voids the calibration.
Max-score conservatism. The worst word decides a field, so long fields (FUNSD answers, SROIE addresses) need larger thresholds and larger sets.
Support-limited field conditioning. Types below m0 = 25 calibration spans fall back to the pooled threshold and receive no per-type guarantee (most of CORD).
Short-span audit blind spot. The boundary-escape test cannot see spans of two words or fewer, so a clean result on short fields means untested.
Audit independence assumption. The randomization null assumes independence across spans, which the method states rather than derives. Findings are claimed only when a document-blocked bootstrap agrees.
The guarantee is in expectation, over calibration draws and fresh documents. A single document, or a single calibration split, can exceed α.
Not a replacement for human review in consequential workflows.

License

Model weights: CC BY-NC-SA 4.0, inherited from microsoft/layoutlmv3-large (CC BY-NC-SA 4.0). This repository fine-tunes that model for token classification and adds calibration files. Non-commercial use only, with attribution, under the same license.
CaliDoc source code: Apache-2.0, in the GitHub repository.
Training data: WildReceipt's authors state no license (the MMOCR dataset zoo lists it as N/A); obtain it from the official MMOCR instructions. Lineage used: the OpenMMLab WildReceipt release, obtained from two byte-identical Kaggle mirrors because the canonical OpenMMLab host was unreachable (official 1,267/472 split; 100 training receipts held out for early stopping and temperature). Cite Sun et al., Spatial Dual-Modality Graph Reasoning for Key Information Extraction, arXiv:2103.14470, 2021. No dataset images or annotations are included in this repository.

Citation

WACV 2027 proceedings metadata (pages, DOI) will be added once published.

@inproceedings{hamdan2027calidoc,
  author    = {Hamdan, Mohammed and Cheriet, Mohamed},
  title     = {{CaliDoc}: Calibrated Field-Level Risk and Annotation Auditing for Document Information Extraction},
  booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
  year      = {2027}
}

@inproceedings{huang2022layoutlmv3,
  author    = {Huang, Yupan and Lv, Tengchao and Cui, Lei and Lu, Yutong and Wei, Furu},
  title     = {{LayoutLMv3}: Pre-training for Document {AI} with Unified Text and Image Masking},
  booktitle = {Proceedings of the 30th ACM International Conference on Multimedia},
  year      = {2022}
}