arjhinety/schemaforge-v2-distilled-minicpm5-1b

SchemaForge V2 — Distilled MiniCPM5-1B (V2 final, iteration 15)

This is the V2 release checkpoint. It closes out the V2 phase of the SchemaForge project: the pipeline (teacher-validation gate, hard-example generation, confidence calibration, hybrid routing, failure-category analysis, and a now-deterministic/reproducible training pipeline) is considered methodologically complete as of this checkpoint. Follow-on work (V3) targets the model's own remaining weaknesses (see Known limitations) under a fresh whitepaper. See SchemaForge V2 whitepaper and logs/V2_TRAINING_FAILURES.md for the complete 15-iteration research history, including every negative result along the way.

What this is

A student checkpoint (openbmb/MiniCPM5-1B, ~1.04B params) sequence-level knowledge-distilled from a google/gemma-4-31B teacher for JSON/structured extraction across 12 document schemas (invoice, receipt, resume, contract, support ticket, medical note, insurance claim, CRM record, email, conversation, form, knowledge-graph triple). Part of the SchemaForge V2 project: an automatic distillation pipeline with teacher-output validation, hard-example generation, and field-level evaluation, not just a checkpoint — and the pipeline itself is the project's primary contribution. This specific checkpoint's headline result is not a raw-model number: it's that routing between this model and a tuned deterministic pre-pass (by field ownership) beats either system alone on every metric simultaneously — see Results below.

Why this checkpoint, specifically

15 training iterations were run during V2 development (full history in logs/V2_TRAINING_FAILURES.md). The single highest field-F1 measurement across all of them (0.6858, iteration 5/10) was produced by a checkpoint whose weights were later lost to a since- fixed process bug (a checkpoint got overwritten without a backup before that safeguard existed — documented, not hidden, in iteration 12's postmortem). This checkpoint (iteration 15) is the most recent, and was produced by the fully bug-fixed pipeline: a fixed training random seed, an automatic checkpoint-backup step, and — critically — deterministic (greedy) teacher-label generation, none of which were true for the iteration-5 checkpoint. A controlled re-run (iteration 15) measured this fix's actual impact: field F1 moved by only 0.0003 versus its immediate predecessor, meaning this checkpoint's score (0.6827) is a stable, reproducible measurement, not a lucky or unlucky draw — the property that matters most for a checkpoint meant to be published and cited.

Training data

• 2700 hard-example records generated by schemaforge/hardexamples/generate.py across the 9 training schemas (3 schemas — insurance_claim, conversation, kg_triple — are held out from training entirely, evaluated only for generalization).
• Teacher outputs gated through a 4-step validation pipeline (JSON parse, Pydantic schema validation, source-support/ontology-derivation check, no-over-assertion check) before admission. 1625/2700 (60.2%) of teacher outputs passed the gate.
• 3 epochs, sequence-level cross-entropy distillation (no cross-tokenizer logit KL — found invalid in an earlier project phase and dropped, see docs/PROJECT_CHARTER.md §7.1).
• Teacher generation uses greedy decoding (temperature=0.0) for reproducibility — see "Why this checkpoint" above.

Results

On a 72-record held-out eval set spanning all 12 schemas (including the 3 held-out ones):

system	field precision	field recall	field F1	hallucination rate	schema validity
deterministic rules alone	0.9185	0.1729	0.2911	0.0000	1.0000
this model alone (residual-field prompt)	0.4900	0.4784	0.4841	0.0914	0.8194
hybrid (rules → this model)	0.7174	0.6513	0.6827	0.0092	0.8333

The hybrid system beats both individual systems on every metric simultaneously — this is the project's central architectural claim, and it holds here as it has in every hybrid configuration tested. missing_field (omission) is the dominant failure category across every measured configuration in this project (roughly 55-62% of all failures) — the clearest remaining target for future work.

Known limitations

• Model-alone field F1 (0.4841) is close to, and on some measurements below, the un-distilled base MiniCPM5-1B's zero-shot score — the distillation's value in this pipeline is realized through the hybrid architecture, not as a standalone extractor. Use it hybrid, with the deterministic pre-pass, not alone.
• Confidence output is poorly calibrated without post-hoc temperature scaling (raw mean-token confidence has been measured badly overconfident on this project's checkpoints); a fitted temperature scaler helps but was not re-fit specifically for this checkpoint.
• No hybrid escalation policy to a frontier LLM is implemented — the confidence signal has been characterized (coverage-at-risk curves exist) but no production routing threshold is set.
• missing_field remains the dominant failure mode; a targeted fix attempted in iteration 14 (isolated corruption-operator training) did not help and is documented as a negative result.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("<this-repo>")
model = AutoModelForCausalLM.from_pretrained("<this-repo>", trust_remote_code=True)

prompt = (
    "Extract the following fields as JSON from the text below. Schema: invoice.\n"
    "Fields to extract: line_items[].description, vendor_name.\n"
    "Text:\nINVOICE #INV-1001. Vendor: Acme Supply Co. ...\n"
    "JSON Output:"
)
inputs = tokenizer(prompt, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

For production use, run this model behind the deterministic pre-pass (schemaforge/deterministic/) and merge with schemaforge/hybrid/pipeline.py's merge_prediction — using the model alone forgoes the architecture's actual advantage. The prompt format (schema name + explicit residual field list, generated from the schema's semantic_fields minus whatever the deterministic pass already resolved) must match training; see src/08_hybrid_eval.py's build_prompt in the source repository for the exact template.

Training/compute

AMD Instinct MI300X (192GB), ROCm 7.0.2, PyTorch 2.11.0.dev+rocm7.0. GPU access provided by the AMD AI Developer Program.

Checkpoint identity

• model.safetensors sha256: c13f7f6c21c35d2ed9159acc044a9f88abcd9fdfe9d0248ee625236b70c8470d
• Training run: loop-iter3-20260810T095747Z (see experiments/loop-iter3-20260810T095747Z/ in the source repository for full machine-readable provenance)
• Git commit: 9c941ea12f299b66c7bfd04603f6beea0ce04c9b

Source repository

See the SchemaForge GitHub repository for the full pipeline (schema registry, deterministic pre-pass, hard-example generator, teacher-validation gate, evaluation harness, calibration module, failure-category classifier, hybrid routing), the V2/V3 whitepapers, all experiment manifests, and evidence graphs.