Dellboy/toppdblx-residual-parser

TopPDBLX residual parser — LoRA adapters

LoRA adapters for SmolLM2-360M-Instruct, trained to read the crystallisation conditions in the Protein Data Bank that a deterministic rule parser cannot.

Part of TopPDBLX: every exptl_crystal_grow.pdbx_details string in the PDB, turned into typed components with role, concentration and unit.

What problem this solves

A rule parser reads ~93% of the archive. The remaining 52,000 records are written as prose, across line breaks, or in notations no regex reaches. This model reads those.

Measured against 96 hand-labelled records (a crystallographer, one condition at a time):

Source	Precision	Recall	F1	F0.5
Rule parser alone	100.0%	67.7%	80.7	91.3
Rules + this model	99.6%	87.4%	93.1	96.9

Precision: of the reagents claimed, how many are really there. Recall: of the reagents really there, how many were found. F1: their harmonic mean. F0.5: the same, weighting precision twice as heavily — the right summary here, because a missed reagent is recoverable by re-reading the deposition and an invented one is not.

+20.1 points of recall, 59 reagents the rules never find, for 0.4 points of precision. Applied over the corpus it contributed 163,353 components and took the classified share from 59.5% to 77.1%.

Rounds

Each subdirectory is one training round. All are rank-16 LoRA on 16 layers, prompt-masked, trained with MLX-LM on an M1 Max. Identification and grounding are measured on a frozen 2,000-record benchmark so rounds are comparable to each other; earlier rounds were scored against a live residual that shrank as curation improved and are marked accordingly.

Round	What changed	Identification	Grounding	Notes
round01	Bootstrap distillation from rule output, lexicon 0.1.0	87.0%	not measured	live residual, not comparable
round02	not_a_component class, confidence gate fixed	89.7%	not measured	live residual
round03	Cosine schedule, dropout, class rebalanced	88.4%	not measured	live residual
round04	Retrained on the 502-reagent lexicon	—	—	abandoned: trained on 36% duplicate rows
round05	Deduplicated training set, 95,818 distinct pairs	87.58%	93.41%	first frozen-benchmark round
round06	Full epoch, rank 16, 6,856 empty-answer examples	90.52% at iter 2,000	94.36%	the one to use — the final adapter, see below
round07	32B-teacher labels, 92.6% precise, 0.23 epochs	—	—	regressed: precision 99.6% → 92.3%
round08	Same idea, labels 97.6% precise, 1.06 epochs	—	—	regressed: precision 94.5%. Closes the line of work

Which checkpoint to use, and a correction

Use round06/adapters.safetensors, the final adapter.

An earlier version of this card said to use round06/promoted_checkpoint_2000 instead, on the strength of a checkpoint sweep against a frozen 2,000-record benchmark. Measured against hand-labelled truth, that was wrong:

Round 06 adapter	Precision	Recall	F1	False positives
final, 6,000 iterations	99.6%	91.5%	95.4	1
checkpoint 2,000	95.7%	91.2%	93.4	12

Twelve extra false positives against one, p = 0.0034. The checkpoint is kept in the repo because the disagreement is the interesting part, not because it should be used.

Why the sweep chose wrongly. It ranked checkpoints on identification — whether an emitted reagent name exists in the curated lexicon. That metric cannot see a name which is real, present in the text, and simply not what the depositor meant, so a checkpoint that invents plausible chemistry scores well on it. Only labelled truth separates the two. The frozen benchmark remains useful for comparing rounds; it is not sufficient for choosing between them.

The sweep is still worth reading for what it shows about distillation:

Iteration	Fidelity to rules	Residual identification
500	80.60%	86.80%
2,000	89.60%	90.52%
6,000	93.60%	88.99%

Fidelity climbs monotonically while identification turns over: past a point the model spends its capacity imitating the rule parser — learning what is already in code — rather than reading the residual. Validation loss moved 0.003 across the whole span, which is why it is not the stopping signal here.

Rounds 07 and 08, and a correction to how they were judged

Both were attempts to get past the ceiling above by training on labels from a local Qwen2.5-32B teacher rather than from the rule parser. Both scored worse than round 06 — and the metric that said so was biased against them.

Five of the 96 hand-labelled records carried a reagent name the curated lexicon could not resolve, so their truth set was incomplete and a model naming that reagent correctly was scored as a false positive. The penalty falls hardest on whichever model says the most. After resolving those labels (lexicon 0.6.0) and excluding the records still incomplete:

	Precision	Recall	F1	F0.5
round 06 final	99.6%	87.4%	93.1	96.9
round 07 — labels 92.6% precise, 0.23 epochs	93.6%	89.1%	91.3	92.6
round 08 — labels 97.6% precise, 1.06 epochs	95.3%	89.1%	92.1	94.0

On F0.5, which weights precision twice as heavily and is the right summary for a released dataset, round 06 leads by 2.9 points rather than F1's 1.0.

Round 06 remains the model to use: it leads on F1 and holds a 4.3-point precision advantage, and for a released scientific dataset missing chemistry is recoverable where invented chemistry is not. But the two teacher rounds do gain real recall, and the earlier version of this card called the approach closed on the strength of the biased numbers.

What the failures actually are. None of round 08's false positives names a reagent absent from the source text. Every one is a reagent genuinely present but not part of the crystallisation condition — a protein storage buffer, a soak, a cryoprotection step. The models are misattributing rather than inventing, which is a question about roles rather than chemistry.

They are also systematic: 11 of round 08's 14 false positives are the same reagents as round 07's, across independently filtered labels and four times the training.

Two teachers, measured

Correlated single-model error is what independent-model agreement removes and heuristic filters cannot, so a teacher-only find was kept only when a second, architecturally different 32B (Gemma-4-31b) named the same reagent unprompted:

	Precision	Recall	F1	F0.5
rules + this model (shipped)	99.6%	87.4%	93.1	96.9
rules + Gemma alone	96.3%	87.8%	91.8	94.4
+ every Qwen find	92.2%	92.9%	92.5	92.4
+ only where both agree	96.1%	91.8%	93.9	95.2

Agreement keeps 13 of the 16 correct finds while cutting the wrong ones from 22 to 10 — the best F1 measured in this project, and still behind the shipped model on precision and on F0.5. It is useful as an inference-time ensemble, not as a training signal: the surviving additions are 13 correct to 10 wrong, and rounds 07 and 08 established that this student absorbs label noise rather than averaging it out.

Using it

from mlx_lm import load, generate

model, tokenizer = load(
    "mlx-community/SmolLM2-360M-Instruct",
    adapter_path="round06",          # the final adapter, not promoted_checkpoint_2000
)

SYSTEM = (
    "You convert a PDB crystallisation condition string into JSON. "
    "Return only a JSON array. Each element has: role (precipitant, salt, buffer, additive, "
    "cryo, not_a_component or unknown), name (the canonical reagent, or null when the text "
    "names no reagent), amount (a number or null) and "
    "unit (percent_w_v, percent_v_v, molar, millimolar, mg_ml or null). "
    "Use not_a_component for text that names no reagent at all: method notes, screen "
    "references, or an unnamed protein, inhibitor or compound."
)

prompt = tokenizer.apply_chat_template(
    [{"role": "system", "content": SYSTEM},
     {"role": "user", "content": "0.1M HEPES pH 8.0, 10% PEG 8000, 10% 1-BUTANOL"}],
    add_generation_prompt=True,
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=512))

The emitted name is a canonical id from TopPDBLX's curated lexicon (542 reagents, 1,306 spellings). A name outside that lexicon should be treated as a hallucination and dropped — that guard is part of the pipeline, not the model.

Limitations

• It cannot discover chemistry the lexicon does not contain. It emits names it saw in training, so a genuinely novel reagent looks identical to a model error.
• Recall is the weaker half. 99.6% precision against 91.5% recall: it is far likelier to miss a reagent than to invent one. Every metric used before the hand-labelled set was precision-shaped and blind to that.
• Trained on one corpus. Depositor conventions in the PDB; no claim beyond them.
• Concentrations are not covered by the 96-record evaluation, which measures reagent identity only.

Licence

MIT for the adapters. The base model carries its own licence.