MolPerceive-4B: A 4B Model That Reads a Molecule Instead of Guessing at It
MolPerceive-4B is a LoRA adaptation of google/gemma-3-4b-it that answers structural questions
about a molecule from its SMILES string: atom and bond counts, ring systems, shortest paths through
the graph, per-atom hydrogen counts, molecular formula, and whether the structure is chemically
valid at all. Built for the Adaption AutoScientist Challenge, Part 2 (Science); the training
corpus was co-optimized with Adaptive Data (Adaption Labs).
Every label in the corpus is computed by RDKit and verified against an independently written parser, never authored by a language model and never judged by one.
Read this first: what kind of number follows
Adapters are usually scored by win rate, the share of rows where a language model judge preferred the tuned output. Every number on this card is exact-match accuracy: the share of rows where the model produced the correct answer, checked against a label computed by executable code.
The two are not interchangeable, and this project has the receipt. A sibling entry in this portfolio trained its identical corpus on Adaption's own AutoScientist. The platform reported a 95.48% win rate. Our scorer, run on that same adapter, measured 62.0% exact-match accuracy. A win rate is a preference; an accuracy is a fact.
| what is measured | exact match against RDKit-computed labels |
| a language model judged something | never, not the labels, not the scoring |
| held-out rows scored | 1,300, across four independent slices |
| rows the base model wins | 0 of 1,300 |
| significance | p = 5.2e-113, exact McNemar, novel scaffolds |
| external validation | 200 real deposited structures from wwPDB, a public database |
| cost to recompute every number | one laptop, no GPU, no API key |
TL;DR
On its own held-out distribution the base model scores 0.0% and MolPerceive-4B scores 92.0%. Across 1,300 held-out molecules spanning four independent tests, the base model did not win a single row.
The result that matters most is not that number. It is that performance on real molecules from an external public database is statistically indistinguishable from performance on unseen scaffolds from our own generator. A model that had memorized our synthetic idiom would collapse on the external slice. It does not move.
Headline results
Base and tuned generated in one process under identical greedy decoding, scored by the released
score_molperceive.py, which a reviewer can run on a laptop with no GPU.
| test | what it proves | n | base | MolPerceive-4B |
|---|---|---|---|---|
| held-out, familiar scaffolds | own distribution, the competition's own yardstick | 300 | 0.0% | 92.0% |
| held-out, novel scaffolds | ring systems and substituents absent from all 90,000 training rows | 500 | 0.0% | 74.8% |
| real molecules, wwPDB CCD | not our generator at all: real deposited structures from a public database | 200 | 0.0% | 73.5% |
| hard shard, 4 to 9 rings | built to be the hardest thing we own | 300 | 0.0% | 90.0% |
The base model never wins a single row. Every tie is both models failing; the base beat the tuned model zero times out of 1,300. Forced-choice win rate is therefore 100% wherever it is defined, so the tie-allowed figure above is the honest headline and the one we quote.
Significance, stated rather than assumed
Every slice is a paired design, so significance is an exact McNemar test on the rows where exactly one model was right. Rows both models get right, or both get wrong, carry no information about which is better and are conditioned out. The null is that the adapter is no better than the base, under which each discordant row is a fair coin.
| slice | only tuned right | only base right | exact two-sided p |
|---|---|---|---|
| held-out, familiar | 276 | 0 | 1.6e-83 |
| held-out, novel scaffolds | 374 | 0 | 5.2e-113 |
| real molecules, wwPDB CCD | 147 | 0 | 1.1e-44 |
| hard shard | 270 | 0 | 1.1e-81 |
The base does not win a single row on any of the four slices. That is what makes the
middle column zero everywhere, and it is why the p-values are what they are. Recompute them
with the released significance.py, which reads the released generation files and uses
exact integer binomial coefficients rather than a normal approximation.
Asking the same question differently
Both slices below are released, so a reviewer can run them rather than take this on trust. Neither shares wording with the training rows.
| test | what changes | n | base | MolPerceive-4B |
|---|---|---|---|---|
mp_para |
the request is reworded entirely: "which element sits at index 0", "its empirical composition", phrased as a database validation task rather than a chemistry question | 200 | 0.0% | 95.0% |
mp_random |
more fields per request, asked in randomized order, so a memorized answer template cannot line up | 200 | 0.0% | 72.0% |
Paraphrase costs nothing: 95.0% is above the 92.0% headline, so the model is reading the request rather than pattern-matching familiar phrasing. Randomizing the field set and its order costs real accuracy, 72.0%, and that is the honest shape of the limitation: the perception is robust, the harder job is holding many simultaneous requested fields in one answer. Both are reported because both slices ship.
The gains come from the data, not from the choice of base
To rule out the possibility that these deltas depend on a conveniently weak base, the same corpus was used to train a second, unrelated 4B model, Qwen3.5-4B, on a seeded 30,000 row subsample:
| base model | held-out familiar | held-out novel scaffolds |
|---|---|---|
google/gemma-3-4b-it |
0.0% to 92.0% | 0.0% to 74.8% |
Qwen/Qwen3.5-4B |
0.0% to 92.7% | 0.0% to 76.0% |
Both bases score exactly zero before training on both slices, so the task is hard for 4B models generally rather than for one chosen model, and both land within about a point of each other after training on the same data. The capability is in the corpus.
Version history, stated plainly
v1 was trained on 90,000 rows for 2 epochs. Diagnosis of its failures produced a 12,000 row hard shard targeting two specific defects, and v2 is a continuation from the v1 adapter on that shard plus an 18,000 row seeded replay of the original corpus, not a fresh run. Both sets of numbers are published so the effect of the continuation is visible rather than absorbed:
| slice | v1 | v2 |
|---|---|---|
mp_held_seen |
93.7% | 92.0% |
mp_held_novel |
74.6% | 74.8% |
mp_real |
74.0% | 73.5% |
mp_hard, 4 to 9 rings |
59.0% | 90.0% |
The forecast for v2 was pre-registered before the retrain ran, in
docs/results/molperceive_v2_plan_and_forecast.md, because a calibration claim is only checkable
if the number predates the result.
The generalization result, and why it is the important one
Our generator produced the training rows, so a good score on held-out rows from the same generator is necessarily a weaker claim than it looks. The wwPDB Chemical Component Dictionary slice exists to close that gap: 200 real, deposited chemical structures written by other people, for other purposes, years before this project.
Performance there sits within sampling error of performance on unseen scaffolds from our own generator. Status and field selection hold at 99.5% on the real molecules, so on structures from an entirely different source the model still reliably knows what was asked and whether the input is valid. The entire remaining loss sits in the values, which is where per-structure perception lives.
What actually changed in the model's behaviour
| base | MolPerceive-4B | |
|---|---|---|
| parseable JSON on real molecules | 66.0% | 100% |
| answered a chemically impossible structure anyway (familiar) | 100% (36/36) | 3.3% |
| answered a chemically impossible structure anyway (novel) | 100% (60/60) | 13.0% |
| falsely rejected a valid structure | 0.0% | 0.9% familiar, 0.0% novel |
The base model answers every chemically impossible structure it is shown, all 96 of them, confidently and in the requested format. Learning to refuse an impossible input is a real capability and it is one of the things this corpus teaches.
An independent blind audit, not just a scorer
A separate auditor, blind to the scorer's verdicts and to these results, re-judged a seeded sample using its own RDKit calls: 34 of 40 correct (85.0%), and the scorer agreed with all six of its failure calls, row for row. Two instruments, one blind to the other, converging on identical verdicts.
The audit also tested memorization rather than assuming: 34 of 40 tuned derivations are character-identical to gold, which looks damning, but 0 of 300 held-out molecules share an InChIKey or a prompt with any of the 90,000 training rows, and per-atom implicit hydrogen counts match RDKit exactly on 35 of 36 walks including a 27-atom chain. The verdict was genuine per-structure perception rendered through a deterministic template.
It flagged its own sampling bias unprompted: it had audited the familiar-scaffold slice, which is the flattering one by construction.
Scope
Organic main-group chemistry as written in SMILES: C, H, N, O, S, P, F, Cl, Br, I, B and Si, at 5 to 40 heavy atoms. Not stereochemistry, not organometallics, not 3D conformation.
Training details
| base | google/gemma-3-4b-it |
| method | LoRA r32 alpha64 on the language-model linears, completion-only masking |
| corpus | 90,000 rows (v1) then a 30,000 row continuation (12,000 hard shard, 18,000 seeded replay) |
| epochs | 2, then 1 for the continuation |
| max_len | 1024, measured with the real tokenizer rather than assumed |
| hardware | one A100-40GB, Spot |
max_len is measured, not chosen. Under completion-only masking a truncated completion is a
silently poisoned label, so the p99 token length is measured with the actual tokenizer before every
run on this project.
Evaluation protocol
Base and tuned are generated in one process under identical greedy decoding (do_sample=False),
so the comparison is not confounded by sampling. Output length is compared base against tuned, and a
tuned median below 0.6x the base median fails the run regardless of accuracy, which is a guard
against buying a win rate with terseness. Every gate was proven able to fail by injecting
deliberately corrupted rows: 200 of 200 injected defective rows are rejected.
Usage
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
BASE = "google/gemma-3-4b-it"
tok = AutoTokenizer.from_pretrained(BASE)
# AutoModelForCausalLM resolves gemma-3-4b-it to its multimodal wrapper, which is correct.
# Do NOT load it through a text-only class: the decoder is then randomly initialized and
# the model emits whitespace, with no error raised.
model = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype="bfloat16",
attn_implementation="eager", device_map="cuda")
model = PeftModel.from_pretrained(model, "Jainamshahhh/molperceive-4b").eval()
Try it, and everything that backs it
Live side by side demo: https://huggingface.co/spaces/Jainamshahhh/molperceive-demo Enter your own input and watch the base model and MolPerceive-4B answer it under identical greedy decoding. The GPU backend scales to zero, so a cold first request takes about a minute.
Released on both platforms, with the scorer, every eval slice, the per row verdicts and
significance.py alongside, so every number on this page can be recomputed rather than
trusted.
Reproducibility and license
The released dataset, the scorer, and the eval slices are all published. A reviewer holding those files can recompute every number on this card on a laptop with no GPU. Apache-2.0, matching the base model. Built with Adaptive Data by Adaption Labs.
- Downloads last month
- -