chenjigaram/kid-extract-135m-lora

Model card — kid-extract

LoRA adapter for HuggingFaceTB/SmolLM2-135M-Instruct that extracts a fixed JSON schema from European fund disclosure documents (PRIIPs KID and UCITS KIID).

Intended use

Converting the text of a fund disclosure document into validated JSON: risk indicator, charges, recommended holding period, investment objective, performance scenarios and fund identifiers. Built for batch processing where cost and latency per document matter.

Runs on CPU. No GPU is required for training or inference.

Out of scope

• Investment advice, suitability assessment, or any decision affecting a retail investor
• Documents outside the PRIIPs and UCITS disclosure families
• Languages other than English, German, French and Dutch
• Any use where an unverified extraction feeds a regulatory filing or a client-facing figure

Output is machine-readable, not authoritative. Validate before relying on it.

Training data

Synthetic documents generated by the kid-extract repository, grounded optionally in real fund attributes from the Morningstar European funds dataset. See the dataset card for how the corpus is built and what it does not capture. No real disclosure document was used for training, and nothing proprietary or confidential was involved.

Training procedure

Base model	HuggingFaceTB/SmolLM2-135M-Instruct
Method	LoRA (PEFT) via TRL SFTTrainer
Trainable parameters	4,884,480 of 139,399,488 (3.5%)
Rank / alpha / dropout	16 / 32 / 0.05
Target modules	q_proj k_proj v_proj o_proj gate_proj up_proj down_proj
Max sequence length	1408
Learning rate / schedule	0.0002 / cosine
Batch / accumulation	1 / 4
Training examples / epochs	1500 / 1
Optimiser steps	375
Final train / eval loss	0.0478 / 0.0057
Wall clock	5.7 hours on 4 CPU threads
Hardware	Intel i5-8365U, 4 cores, CPU only

Loss is computed on the JSON completion alone. The prompt contains a field specification that is byte-identical in every example, so training the model to reproduce it would spend capacity on nothing.

An earlier run at 38 optimiser steps was measurably undertrained: it produced unbalanced JSON and runaway generation, and raising the token cap from 400 to 640 did not help. Gradient accumulation was halved from 8 to 4 to buy ten times the weight updates for the same wall clock. Anyone shortening this run should expect schema validity to degrade first.

Evaluation

Scored on test_unseen_layout, which combines four layouts held out of training with the reserved label vocabulary — wordings for every field that appear in no training document. This is deliberately the harder of the two test splits.

Metrics: per-field precision, recall and F1 over 23 fields; null accuracy; schema validity; hallucination rate measured against the source text; exact match; median latency per document.

Reference point

A hand-written multilingual regular expression extractor, given the training vocabulary only:

Split	Micro F1	Macro F1	Exact match
test_seen	0.978	0.976	0.633
test_unseen_layout	0.645	0.516	0.000

The collapse on unseen wordings is the gap this adapter is trained to close.

Results

All systems scored on the same 50 documents by identical code.

Unseen layouts and unseen label wordings:

System	Micro F1	Macro F1	Exact	Schema valid	Hallucination	Latency
fine-tuned 135M	0.858	0.830	0.02	0.88	0.005	21.3 s
rules	0.643	0.514	0.00	1.00	0.000	0.001 s
few-shot 135M	0.169	0.155	0.00	0.50	0.491	40.0 s
zero-shot 135M	0.000	0.000	0.00	0.00	0.000	28.1 s

Known layouts and wordings:

System	Micro F1	Macro F1	Exact	Schema valid	Hallucination	Latency
fine-tuned 135M	0.988	0.988	0.82	1.00	0.006	21.1 s
rules	0.983	0.982	0.72	1.00	0.000	0.001 s

The same base model goes from producing no valid JSON at all to 0.858 micro F1 on documents whose layout and vocabulary it has never seen. Few-shot prompting reaches only 0.50 schema validity and invents 49% of the values it emits.

Rules remain better on fields recoverable by shape rather than by name — ISIN, fund name, the scenario table, the risk scale — and run 20,000 times faster. The model wins on every label-dependent field, where an unfamiliar wording sends a regular expression to zero. A hybrid would beat both.

Still weak: transaction_costs_pct at 0.078 and domicile at 0.350 on unseen wordings, schema validity dropping to 0.88, and exact match over all 23 fields at once of 0.02. Per-field accuracy is not whole-record accuracy; validate fields individually rather than trusting a record.

Limitations

• Trained on generated documents. Performance on real provider PDFs is unmeasured until the hand-labelled real test set exists, and should be assumed lower.
• A 135M model has little capacity for reasoning about a value that is implied rather than stated. Fields requiring inference are out of reach by design; the prompt instructs the model to copy, not calculate.
• Long documents are truncated at 1408 tokens.
• Text must already be extracted from the PDF. No OCR or layout analysis is included.

Licence

MIT. Base model is Apache 2.0.