byosm-doctype

Classify a business document (PDF or scan) into one of 16 types to route it to the right team: invoice, letter, memo, form, email, resume, ...

A single-purpose System 1 model built with BYOSM: fine-tuned from sentence-transformers/all-MiniLM-L6-v2 + microsoft/dit-base-finetuned-rvlcdip for this task only, with a calibrated decision policy, shipped as a 109.55M-parameter ONNX bundle that runs on any CPU with onnxruntime.

task	document / classify
labels	advertisement, budget, email, file_folder, form, handwritten, invoice, letter, memo, news_article, presentation, questionnaire, resume, scientific_publication, scientific_report, specification
parameters	109,546,385 (shipped as int8 ONNX, 113 MB)
backbone	text hf:sentence-transformers/all-MiniLM-L6-v2 + visual hf:microsoft/dit-base-finetuned-rvlcdip, route fused · embedding dim 256
objective	accuracy, primary suite rvl
decision policy	threshold, threshold – (calibrated on rvl's calib split)

Labels

• advertisement: promotional material: ads, product promotions, flyers, posters
• budget: budget tables, financial plans, cost breakdowns and spending summaries
• email: printed e-mail messages (From/To/Subject headers)
• file_folder: file folder covers, tabs and labels
• form: fill-in forms with fields, boxes and checkboxes
• handwritten: documents that are mostly handwritten
• invoice: bills and invoices: amounts due, line items, totals
• letter: formal correspondence: letterhead, salutation, signature
• memo: internal memoranda (MEMO / TO / FROM / RE headers)
• news_article: newspaper or magazine articles and clippings
• presentation: slides and presentation pages
• questionnaire: surveys and questionnaires with questions to answer
• resume: resumes / CVs and biographical sketches
• scientific_publication: published scientific papers and journal pages
• scientific_report: scientific / technical reports, lab reports
• specification: product or technical specifications, spec sheets

Comparison

Every model below, this one and the baselines, was scored by the same code on the same frozen test splits, with its decision policy calibrated on the same calib splits. Bold = this model.

model	params	builtin accuracy	rvl accuracy (primary)	builtin ROC-AUC	rvl ROC-AUC
v2-ditft (this model)	109.55M	–	0.9481	–	–
hf:microsoft/dit-base-finetuned-rvlcdip (baseline)	85.82M	0.9271	0.9400	–	–
v1 (BYOSM experiment)	109.55M	–	0.9184	–	–
r2 (BYOSM experiment)	109.55M	–	0.9050	–	–
fused-dit (BYOSM experiment)	109.55M	0.8792	0.8838	–	–
fused (BYOSM experiment)	45.70M	0.8063	0.8009	–	–

Test evaluation (this model)

suite	dataset	test items	accuracy	95% CI	precision	recall	ROC-AUC	avg precision	decision threshold
rvl	chainyo/rvl-cdip	3200	0.9481	0.941-0.956	–	–	–	–	–

Usage

pip install huggingface_hub numpy onnxruntime tokenizers pillow pypdfium2 rapidocr_onnxruntime

from huggingface_hub import snapshot_download
import sys
path = snapshot_download("bsye/byosm-doctype")
sys.path.insert(0, path)
from byosm_rt import load

m = load(path)
print(m.predict("invoice.pdf"))       # PDF or page image; scans are OCR'd
emb = m.encode(...)                # the learned representation (JEPA-style), for probes / retrieval / other heads

Local HTTP server

python byosm_rt.py serve . --port 8800
# GET /v1/models · POST /v1/doctype/predict · POST /v1/doctype/encode

How it was built

1. Spec: labels, definitions, objective and data plan in spec.yaml (included in this repo).
2. Data (34,935 training / 3,881 dev examples):

• chainyo/rvl-cdip: role labeled

3. Supervised training: layer-wise LR decay on pretrained encoders (floor 0.5), EMA of the weights, mixed precision, the benchmark's builtin test split folded into training; checkpoint chosen on dev (best epoch 2).
4. Evaluation: frozen eval suites; decision policy calibrated on each suite's calib split; baselines scored identically; promotion decided on calib, test only reported.
5. Improvement lineage (each step proposed by byosm improve from the previous board):

• spec.yaml: initial spec
• fused ← manual (from spec.yaml): document.route=fused
• fused-dit ← more-data (from fused): rvl: 0.809, 0.139 behind hf:microsoft/dit-base-finetuned-rvlcdip (0.948), with 2400 training examples; dev was still rising at the end (0.61 -> 0.84), consistent with being data-limited. chainyo/rvl-cdip (val) has more rows per label: 200 -> 800.
• fused-dit ← domain-encoder (from fused): rvl: 0.139 behind hf:microsoft/dit-base-finetuned-rvlcdip. hf:facebook/dinov2-small was pretrained on a different domain; hf:microsoft/dit-base is self-supervised (masked image modelling) on 42M unlabelled document scans; never saw RVL-CDIP labels.
• fused-dit ← train-longer (from fused): best dev epoch was 7 of 8 (curve 0.77, 0.81, 0.84, 0.83): still converging.
• r2 ← manual (from fused-dit): train.holdout_test=auto
• v1 ← more-data (from r2): rvl: 0.906, 0.042 behind hf:microsoft/dit-base-finetuned-rvlcdip (0.948), with 11520 training examples; dev plateaued at 0.90, so more of the same data may help less than new data or a new recipe. chainyo/rvl-cdip (val) has 2426 rows for its rarest label: 800 -> 2426 per label (all of it).
• v2-ditft ← manual (from v1): document.image_backbone=hf:microsoft/dit-base-finetuned-rvlcdip document.image_pool=pooler

Notes

What this model is. A two-view document classifier: the page image goes through the visual encoder of microsoft/dit-base-finetuned-rvlcdip (read through its own pooler, exactly as its classifier did), the page text (PDF text layer, or RapidOCR for scans) goes through MiniLM, and a learned gate fuses the two. It was then trained on 34,935 RVL-CDIP documents (the val split plus a builtin held-out split).

How to read the comparison with DiT-base. On 3,200 documents from RVL-CDIP's original test split this model scores 94.8% vs DiT-base's 94.0% (+0.8 pt, paired-bootstrap 95% CI +0.3 to +1.3, P(better) = 1.00; it fixes 50 of DiT's errors and introduces 24). The gain comes from reading the text, which DiT does not. This is "DiT-base + the text view beats DiT-base", not a model that reached DiT's accuracy from less data: it starts from DiT's checkpoint, which was fine-tuned on RVL-CDIP's 320k-document train split. That split is disjoint from the test documents used here. On the 800-document calib split the two tie (93.5% vs 93.6%).

Label noise sets the ceiling. On 69 of the 3,200 test documents (2.2%) this model, DiT-base and an independent BYOSM model all agree on the same label, and it is not the given one. In a manual look at a random sample, about half were clearly mislabelled (for example an email printout labelled letter) and others were genuinely ambiguous (scientific_report vs presentation). Published estimates put RVL-CDIP's label error rate at several percent, so accuracy far above ~96-97% on this benchmark would mean fitting the noise.

Confidence-gated use. If you need very high precision, act only on confident predictions and route the rest to a person. With a confidence threshold fitted on calib, a DiT + BYOSM ensemble answered 59.6% of test documents at 99.6% accuracy. Measure this on your own documents before relying on it.

Domain. RVL-CDIP is 1990s scanned tobacco-industry correspondence at low resolution. Modern born-digital PDFs are out of distribution; add an eval suite of your own documents to see how it does there.

Limitations

• Evaluated on rvl (3200 test items). Small suites have wide confidence intervals.

Reproduce

git clone https://github.com/bsye/BYOSM && cd BYOSM && pip install -e .
# put spec.yaml from this repo at specialists/doctype/spec.yaml, then:
byosm build doctype && byosm board doctype && byosm export doctype