typed-extraction-bge-small

An open baseline for typed-extraction: extract a named set of typed fields (text, number, date, time, datetime, duration, list) from a document, or answer null when the document does not say.

It is an adaptive-classifier on a frozen BAAI/bge-small-en-v1.5 encoder, used as a candidate ranker:

1. Candidate spans are generated from the document: regex patterns for numbers, money, dates and times, spaCy entities and noun chunks, label-value lines, and n-grams.
2. For each field the 40 most relevant candidates are kept (sentence BM25 against the field's name and instructions).
3. One shared classifier reads each candidate in context with the field and answers is this the answer?. The best candidate becomes the typed value. Below a threshold tuned on held-out training cases (0.35; list items 0.35), the answer is null, and p_null is one minus the best score.

It is one classifier for every field and family, so it can be asked about fields it has never seen. It was trained on 184,438 candidate examples from the dataset's train split; the three held-out families never appear in training.

Results (test split, 1,690 cases, 17,733 fields)

metric	value
headline (exact, mean over families)	0.291
exact on fields with a value	0.143
hallucination (answered an absent field) ↓	0.103
miss (left a present field empty) ↓	0.600
p_null AUROC	0.788
exact at 90% coverage	0.377
ECE ↓	0.303
exact on computed answers	0.008
CPU time per case	1.7 s

See the leaderboard for other models.

family	fields	exact	exact, fields with a value	candidate ceiling	hallucination
bank_statement	608	0.375	0.321	0.616	0.000
claim_email	617	0.331	0.189	0.564	0.018
cuad	4,182	0.697	0.000	0.081	0.007
drop	1,684	0.011	0.011	0.299	–
extractbench	2,115	0.140	0.001	0.159	0.483
invoice	681	0.325	0.218	0.678	0.103
itinerary	608	0.342	0.250	0.811	0.072
job_posting	609	0.420	0.383	0.782	0.296
lease	770	0.487	0.404	0.638	0.000
medical_appointment	760	0.166	0.043	0.600	0.495
meeting_notes	659	0.126	0.062	0.722	0.550
real_estate_listing	961	0.166	0.039	0.520	0.237
shipping_notice	607	0.260	0.198	0.557	0.000
squad2	1,908	0.461	0.000	0.745	0.027
wikipedia	964	0.059	0.020	0.684	0.071

Candidate ceiling is, among fields that have a value, the share whose answer is among the 40 candidates at all (0.489 overall). No classifier over these candidates can beat it on those fields, so it caps the "fields with a value" column, not the headline (which also counts correct nulls).

Limits

• It cannot compute. It returns text found in the document, parsed into the field's type, so totals, counts and day spans are out of reach (0.008 exact on computed answers).
• It is cautious. It leaves 60% of present fields empty. In return it seldom invents a value.
• The candidate step is the bottleneck. The classifier can only be as good as the ceiling above.
• English only. Long documents are slow on CPU, because every candidate is a separate classifier input.

Usage

pip install "adaptive-classifier>=0.3.1" torch transformers spacy dateparser numpy huggingface_hub
python -m spacy download en_core_web_sm
huggingface-cli download adaptive-classifier/typed-extraction-bge-small --local-dir typed-extraction-bge-small

import sys; sys.path.insert(0, "typed-extraction-bge-small")
from predict import Extractor

x = Extractor("typed-extraction-bge-small")
x.predict("Invoice INV-2207. Bill to: Harbor Supplies Ltd. Total due: $1,240.00 by 14 March 2026.",
          {"total": {"type": "number", "instructions": "The total amount due."}, "due_date": {"type": "date", "instructions": "When payment is due."}, "po_number": {"type": "text", "instructions": "The purchase order number."}})

Each answer has value, p_null, confidence and, for a scalar, the span it came from. This model's actual answer to that call (values and p_null):

"total": {"value": null, "p_null": 0.74}
"due_date": {"value": null, "p_null": 0.7}
"po_number": {"value": null, "p_null": 0.75}

That is typical of this baseline: it is cautious, and when it does answer it can pick the wrong span of the right type. results.json holds the full scores.

Licence

CC BY-SA 4.0, like the dataset: the classifier stores its training examples (examples.json), which come from share-alike sources (SQuAD 2.0, DROP, Wikipedia). The code files are included under the same terms.