doc-fraud-ocr-validate

OCR + field-validation pipeline for document fraud detection: TrOCR-based printed-text field extraction evaluated on real scanned documents, plus a pure-Python rule layer that flags structurally invalid document fields (number format, date logic, ICAO MRZ check digits).

Results

Rule layer (evaluate_validators.py, 2000 seeded cases per rule, seed 13)

Labels are fixed at generation time (by construction), never by the code under test, so a shared bug cannot hide.

Rule	Precision	Recall	F1
ssn	1.0	1.0	1.0
date_past	1.0	1.0	1.0
name	1.0	1.0	1.0
mrz_line2	1.0	1.0	1.0

The MRZ validator is additionally verified against the ICAO Doc 9303 Part 4 specimen line L898902C36UTO7408122F1204159ZE184226B<<<<<10 (composite check digit covers lower-line positions 1-10, 14-20, 22-43; the sex character is excluded) and the worked example L898902C3 -> 6.

OCR extraction (evaluate_ocr.py)

microsoft/trocr-base-printed on 1000 lines sampled (seed 7) from the test split of priyank-m/SROIE_2019_text_recognition (real scanned receipts):

Metric	Value
CER, raw	0.0079
CER, normalized (case/punctuation stripped)	0.0033
Exact match, raw	94.6%
Exact match, normalized	98.6%

Worst-case errors are single-character confusions (digits, N/M), not catastrophic failures. Run on a T4 in ~4 min of pure inference (1000 lines, batch 16). Live metrics: https://huggingface.co/spaces/Offlin33er/doc-fraud-ocr-trackio

Usage

pip install torch transformers==5.17.0 datasets==5.0.1 tokenizers huggingface_hub pillow

# rule layer (CPU, no deps beyond stdlib)
python evaluate_validators.py --n 5000

# OCR evaluation (GPU recommended)
python evaluate_ocr.py --n 1000 --batch-size 16

validators.py is dependency-free; validate_document(fields) returns a list of failed checks for an extracted-field dict, e.g.:

from validators import validate_document
validate_document({
    "document_number": "123-45-6789",
    "name": "ANNA MARIA ERIKSSON",
    "date_of_birth": "1974-08-12",
    "issue_date": "2023-04-15",
    "expiry_date": "2022-04-14",
})
# -> ["expiry_before_issue"]

Notes and limitations

• SSN checks are structural only (public SSA randomization rules: area 001-899 excluding 666, group 01-99, serial 0001-9999). A structurally valid number is not an issued number.
• TrOCR is a line-level recognizer: feeding it full pages requires a line segmentation step upstream.
• The MRZ synthetic evaluation is label-by-construction; correctness of the check-digit formula itself is anchored to the ICAO worked example above.
• trocr_compat.py is a compatibility shim: transformers 5.x cannot auto-convert TrOCR's legacy RoBERTa tokenizer files, so the module builds an equivalent tokenizer.json into the HF cache on first use (idempotent).

Attribution

• TrOCR: Li et al., "TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models" (arXiv:2109.10282) - model microsoft/trocr-base-printed.
• SROIE: Huang et al., ICDAR 2019 competition dataset, mirrored as priyank-m/SROIE_2019_text_recognition.
• MRZ check digits: ICAO Doc 9303 Part 4 (TD3 specification and specimen).