Supermix v80

A 15.3M-parameter research model that solves physics, chemistry and arithmetic problems by writing out its working. Every training example was checked by an exact symbolic solver before it was trained on.

A body of mass 47 kg has an acceleration of 6 m/s^2. What is the force?
  → force = mass x acceleration, 40 x 6 = 240, 7 x 6 = 42, 240 + 42 = 282,
    the force is 282 newtons, total 282
  → CORRECT (282.0)

It is not a chat model, and the limitations section says plainly what it cannot do.

Scores

0.575 overall on 630 freshly generated novel problems across 21 task types, 0 unparseable.

task	v80	v79	task	v80	v79
multiplication	1.00	0.93	molarity ⚛	0.73	0.47
wave_speed ⚛	0.87	0.00	two_step	0.73	0.33
electrical_power ⚛	0.87	0.07	arithmetic	0.63	0.33
word_problem	0.87	0.53	percent	0.60	0.67
sequence	0.83	0.47	division	0.60	0.50
work ⚛	0.80	0.07	acceleration ⚛	0.50	0.33
momentum ⚛	0.80	0.10	power ⚛	0.33	0.10
voltage ⚛	0.80	0.13	algebra_one_step	0.30	0.20
force ⚛	0.77	0.03	average	0.03	0.03

⚛ = science task. science 0.539 · arithmetic 0.622

Three tasks score 0.00 — arithmetic_series, combination, kinetic_energy. The cause is known and is described below.

Do not compare this with Supermix v74's 0.894

Kai9987kai/supermix-v74 scores 0.894, and that number is not comparable to this one. v74 was measured on 10 arithmetic tasks; v80 is measured on 21, including twelve science tasks v74 cannot attempt at all. Different benchmarks.

What this model is for

It is a research artifact for one question: can a small model be made to solve problems verifiably, rather than fluently? Three things follow from that.

Every training row was checked by an exact solver. The corpus is generated by working a problem step by step and then handing a canonical form of the same question to a deterministic Fraction/Decimal solver. Disagreement drops the row. 415,370 science rows shipped this way, combined with 496,108 arithmetic and dialogue rows.

The working is the point. The model does not emit an answer; it emits a derivation ending in one. That is what makes a wrong answer visible instead of plausible.

The answers are checkable at inference. src/answer_check.py re-derives the result from the question for 21 question shapes and returns None when it cannot. None means not checked, never correct.

The finding this model exists to demonstrate

v79 — the same architecture, same corpus size, same step count — scored 0.252. The only meaningful change was writing out the multiplication:

v79 corpus:  force = mass x acceleration, 167 x 11 = 1837, ...
v80 corpus:  force = mass x acceleration, 100 x 11 = 1100, 60 x 11 = 660,
                                          7 x 11 = 77, ...

Measured on v79, accuracy on force by operand size:

operands	correct
single-digit	12/12
two-digit	9/12
three-digit	1/12

The model had learned the physics perfectly — it picked the right formula every time — and could not do the arithmetic in one jump. The six tasks that use multiplication went from 0.00–0.13 to 0.77–0.87 from that change alone, and arithmetic improved too without the arithmetic corpus being touched.

Loss did not show this. v79 posted dev loss 0.1286 and scored 0.252; an earlier model posted worse loss and scored far better. Selection runs on a periodic accuracy probe for this reason.

Known failures, and why

The three tasks at 0.00 share one cause — the same single-step arithmetic problem, in generators that were missed when the others were fixed:

kinetic_energy    : 25 x 25 = 625, 30 x 625 = 18750   two undecomposed products
arithmetic_series : 5 x 228 = 1140, 1140 / 2          undecomposed
combination       : 7 choose 2 = 21                   no working at all

combination asked the model to memorise the entire nCk table. All three are fixed in the corpus generator shipped here (src/build_omni_corpus.py), but this checkpoint was trained before those fixes and still shows them.

average (0.03) is a long-standing weakness inherited from the v74 corpus.

Usage

python example_usage.py "A body of mass 47 kg has an acceleration of 6 m/s^2. What is the force?"

import sys; sys.path.insert(0, "src")
import answer_check
from train_mimomix_talk import load_talk_checkpoint, generate_reply

model, tokenizer, _ = load_talk_checkpoint("supermix_v80.pt")
model.eval()

question = "A wave has frequency 30 Hz and wavelength 7 m. What is its speed?"
reply = generate_reply(model, tokenizer, question, max_new_tokens=72)["reply"]

verdict = answer_check.check(question, reply)   # None => not checkable

Requires torch. CPU is fine — a reply takes well under a second.

Questions can be typed naturally; the corpus carries four to five phrasings per task. src/prompt_normaliser.py additionally rewrites arithmetic questions into the trained form and reports what it sent.

Architecture

Hybrid sliding-window/global attention, sparse mixture-of-experts feed forward, recursive thinking core, and a multi-token-prediction head for self-speculative decoding.

	v74	v80
parameters	8,575,977	15,269,685
active per token	2,810,973	3,912,997
routed experts	32	48
hidden size	192	256
attention heads	6	8
training rows	471,347	866,748
context	128	128
vocabulary	8,417	8,551
dev loss	0.0651	0.0699

Digit-level tokenisation is what makes arithmetic representable at all; a word-level vocabulary cannot spell an unseen number.

Trained 18,000 steps in a single leg, selected on a periodic accuracy probe (0.44 at the final probe), with the checkpoint chosen on that probe rather than on loss.

Limitations

Not a chat model. It answers "hello" fluently, and conversational replies are reproduced verbatim from training data. Treat conversation as a demonstration, not a capability.
Operand range is bounded. Multiplicative tasks were trained on two-digit by one-digit products. Larger operands degrade.
Three tasks score 0.00, as described above.
No world knowledge. It was trained on generated problems and a small templated dialogue set. It knows twelve formulas, not physics.
15.3M parameters. A research artifact, not an assistant.

License

Downloads last month: -; Downloads are not tracked for this model. How to track