mailroom-modernbert-classifier

Hierarchical document classifier for the LLM-Mailroom intake pipeline: a fine-tuned ModernBERT-base encoder with a doc_type head plus one subclass head per document class. It is the deterministic pre-check in the BERT-coupled intake overhaul (mailroom-issues #85).

Architecture

• Backbone: answerdotai/ModernBERT-base — 22 layers, 768 hidden, 8,192-token context, bf16.
• Heads: one head per class — doc_type (6 classes) plus 5 subclass heads (contract, corporate_record, correspondence, insurance_claim, merger_agreement). MLP heads with dropout 0.1.
• Windowing: token-level, 8,192 tokens with 512-token overlap. doc_type by plurality vote over windows; subclass by plurality over windows whose doc_type vote is the winning class.
• Calibration: per-head temperature scaling (temperatures.json).

Files

file	purpose
model.safetensors	ModernBERT backbone weights (bf16, ~298 MB)
heads.pt	hierarchical head state dicts
labels.json	head vocabularies (labels / label2id / id2label / weights)
temperatures.json	per-head calibration temperatures
train_counts.json	per-(doc_type, subclass) authentic train-row counts (support gate)
config.json, tokenizer.json, tokenizer_config.json	backbone config + tokenizer
summary.json	full run summary (hyperparameters, per-epoch metrics, selection, test metrics)

Labels

doc_type (6): contract, merger_agreement, corporate_record, correspondence, insurance_claim, unknown (inference-only abstention — not a trained class)

contract (24): agency, co_branding, collaboration, consulting, development, distributor, endorsement, franchise, hosting, ip, joint_venture, license, maintenance, manufacturing, marketing, other, outsourcing, promotion, reseller, service, sponsorship, strategic_alliance, supply, transportation

corporate_record (10): articles_of_incorporation, board_resolution, bylaws, charter_amendment, indenture, officer_certificate, other, powers_of_attorney, rights_instrument, subsidiary_list

correspondence (7): demand, email, letter, meeting_request, memo, notice, press_release

insurance_claim (6): auto, carrier, inpatient, outpatient, pde, property

merger_agreement (5): all_cash, all_stock, mixed_cash_stock, mixed_cash_stock_election, other

Results

Trained 2 epochs on Lucius-Morningstar/mailroom-modernbert-training @ 5b72a345cd3c057b736bea4910fdbef6509ad1c3 — 4,497 train / 489 validation windows; 323 held-out test documents (never used for training, calibration, or threshold tuning).

Validation

epoch	val_loss	doc_acc	macro-F1 (observed)	ECE (calibrated)
1	1.0041	0.8859	0.8524	0.0201
2	0.9121	0.9195	0.9051	0.0205

Selected epoch: 2 — best observed doc_type macro-F1 subject to calibrated ECE ≤ 0.05 (gate_met: true).

Held-out test (323 docs)

metric	value
doc_type accuracy	0.9319 (301/323)
subclass accuracy (given correct doc_type)	0.5449 (164/301)

Per-head (epoch 2, validation)

head	window acc	macro-F1 (observed)	ECE (calibrated)
doc_type	0.9141	0.9051	0.0205
insurance_claim	0.8889	0.8085	0.0500
corporate_record	0.5405	0.2218	0.1037
merger_agreement	0.4656	0.1988	0.0350
correspondence	0.5161	0.0980	0.1050
contract	0.1630	0.0899	0.0630

Usage

The backbone is a standard ModernBertModel; the hierarchical heads are a custom bundle. Load it with the mailroom-ml inference layer:

from mailroom_ml.inference import load_bundle, classify_document

bundle = load_bundle("Lucius-Morningstar/mailroom-modernbert-classifier")
result = classify_document(
    title="Notice of Default",
    text=document_text,
    bundle=bundle,
)
# result["doc_type"], result["subclass"], result["confidence"], result["route"]

route == "fast_path" means the calibrated gate passed and the LLM sorter may be skipped (skip mode + allowlist only); otherwise route the document to the LLM sorter with the BERT triage as an advisory prior.

Provenance

• Training run: 20260920-173810 — 2 epochs, ~3.78 h on a Modal L4.
• Hyperparameters: batch 4, grad-accum 8, lr 2e-5, seed 42, λ_dt 0.65, label smoothing 0.05, sqrt-inverse class weights (cap 10), MLP heads (dropout 0.1), weight decay 0.01, subclass support floor 12.
• Dataset revision: 5b72a345cd3c057b736bea4910fdbef6509ad1c3.
• Calibration temperatures: doc_type 0.484, contract 0.705, corporate_record 0.596, correspondence 0.960, insurance_claim 0.122, merger_agreement 0.679.

Limitations

• Subclass heads are weak for contract / correspondence / corporate_record / merger_agreement (macro-F1 0.09–0.22). Use the doc_type head for routing; route subclass-ambiguous documents to the LLM sorter (the Tier-1 prior-scoped lane, mailroom-issues #108). Do not trust a skip-mode subclass for these classes yet.
• Trained on a curated legal-document corpus; not a substitute for legal review.
• unknown is an inference-only abstention label, not a trained class.
• Long documents are windowed (8,192 tokens, 512 overlap); the model never truncates silently — oversize documents fall back to the LLM path.

License

Apache-2.0 (inherits answerdotai/ModernBERT-base).