Supermix v74

A newer model is available. Kai9987kai/supermix-v80 adds twelve physics and chemistry tasks and scores 0.575 across 21 task types. Its number is not comparable to the 0.894 below, which was measured on 10 arithmetic tasks only — they are different benchmarks. v74 remains the stronger model on those 10 arithmetic tasks.

An 8.6M-parameter research model that solves arithmetic word problems by writing out its working. It is not a general chat model, and the sections below say plainly what it cannot do.

what is 47 times 6
  asked as : What is 47 x 6?  (multiplication)
  reply    : 40 x 6 = 240, 7 x 6 = 42, total 282
  check    : CORRECT (282.0)

What it scores

0.894 (447/500) on a held-out problem-solving benchmark of ten task types, against 0.756 for the previous version, z=5.74. Problems are generated fresh at evaluation time, so these are novel problems rather than recalled ones. 0 replies were unparseable.

task	v74	previous (v73)
division	1.00	— not present
multiplication	1.00	— not present
sequence	0.98	— not present
two_step	0.98	— not present
word_problem	0.96	0.99
algebra_one_step	0.89	0.94
arithmetic	0.89	0.99
percent	0.75	0.70
average	0.59	0.16

Read that headline honestly

The 0.894 is flattered by the four new task types, which this model finds easy. Compared like-for-like on the five tasks both versions were measured on, it is 0.818 vs 0.756 (z=1.99) — over the significance line by a hair, and driven almost entirely by average improving from 0.16 to 0.59.

One task genuinely regressed: arithmetic fell 0.99 → 0.89 (z=−2.81). Adding six task types to the training mix cost accuracy on the simplest one.

Raw numbers are in results/problem_solving_n500.json.

Prompt format matters — a lot

The benchmark score measures this model on its own training distribution. It is not a measure of how it handles natural language, and the gap is total: typed the way a person writes, the first five arithmetic questions tried against it were all wrong.

prompt	reply
`What is 47 x 6?`	`40 x 6 = 240, 7 x 6 = 42, total 282`	correct
`What is 47 times 6?`	`400 x 6 = 200, 7 x 6 = 42, total 242`	wrong
`What is 47 * 6?`	`40 and 0 = 400, 7 x 6 = 42, total 442`	wrong
`47 x 6`	`subtract 6 from both sides, 44 - 6 = 38, total 38`	wrong

Measured, varying one feature at a time:

the operator token matters — x works, times and * do not
a lead-in phrase must be present — bare 47 x 6 is parsed as algebra
capitalisation and trailing ? do not matter

src/prompt_normaliser.py rewrites a naturally-typed question into the trained form, which takes it to 7/7 on the same kind of questions. It is presentation, not capability: it computes nothing, never alters a number, and questions the model gets wrong in the trained format stay wrong.

The trained format, one example per task:

task	prompt
addition	`Please help with this. 721 + 513`
subtraction	`Solve this basic math problem: 832 - 630`
multiplication	`What is 25 x 7?`
division	`Quick question: 70 / 5`
percent	`What is 50% of 1175?`
two_step	`What is 50% of 698, then add 28?`
average	`Find the average (mean) of these numbers: 34, 35, 47, 49`
sequence	`What comes next in the sequence: 7, 17, 27, 37?`
algebra_one_step	`Solve for x: x + 0 = -12`
word_problem	`A student has 68 cookies. They get 32 more and then give away 60. How many cookies do they have now?`

It recites on conversation

The model produces fluent-looking conversational replies:

"Check the traceback first, then we can isolate the failing function."

Every one of them is reproduced verbatim from its training data. The verbatim rate on the held-out dialogue probes is 1.0. This is a measured property of the training corpus, 19.8% of whose dialogue portion is a single repeated fragment.

Do not use this as a chat model. Its conversational ability is recall.

Usage

python example_usage.py "what is 47 times 6"

import sys; sys.path.insert(0, "src")
import prompt_normaliser, answer_check
from train_mimomix_talk import load_talk_checkpoint, generate_reply

model, tokenizer, _ = load_talk_checkpoint("supermix_v74.pt")
model.eval()

asked = prompt_normaliser.normalise("what is 128 divided by 8")
reply = generate_reply(model, tokenizer, asked.prompt, max_new_tokens=64)["reply"]

verdict = answer_check.check(asked.prompt, reply)   # None => not checkable

Requires torch. CPU is fine — the model is 8.6M parameters and a reply takes well under a second.

Always check the answer. src/answer_check.py re-derives the result from the question for nine question shapes and returns None when it cannot verify one. A None means not checked, never correct.

Architecture

Hybrid sliding-window/global attention with a sparse mixture-of-experts feed forward and a recursive thinking core, plus a multi-token-prediction head used for self-speculative decoding.


parameters	8,575,977 total / 2,810,973 active per token
experts	32 routed, top-2
vocabulary	8,417 word types, digit-level number tokens
context	128 tokens, turn-aligned packing
training	18,000 steps, OneCycle, selected on accuracy (probe 0.89)
dev loss	0.0651

Digit-level tokenisation is what makes arithmetic representable at all — a word-level vocabulary cannot spell an unseen number.

Generalisation

Sentences were withheld wholesale from training, not split by row, so the held-out tiers measure composition rather than recall of a template:

tier	loss	ppl
tier1 — seen response	0.0884	1.0925
tier2 — unseen response	0.1499	1.1617
tier3 — unseen sentence	0.0969	1.1018

Perplexity ratio tier3/tier1 is 1.008x. Full report in results/generalisation_results.json.

Training notes

The run segfaulted at step 11,500 of 18,000 after 9.2 hours and was resumed mid-curve — the OneCycle schedule was restored at 11,500 and ran to exactly 18,000, rather than re-warming on a fresh curve.

That detail matters for interpreting the model: at the point of the crash it looked much worse than it is. Three shared tasks appeared to have regressed badly (arithmetic 0.55, algebra_one_step 0.61, word_problem 0.73), and two of the three were simply unfinished, recovering to 0.89 and 0.96 during the anneal.

Limitations

Not a chat model. Conversational replies are verbatim recall.
Format-brittle. Use the trained prompt format, or the included normaliser.
No knowledge. It was trained on generated arithmetic and a small templated dialogue set. It knows no facts about the world.
percent and average are weak (0.75 and 0.59). It reliably produces correct intermediate working and then fumbles the final combine — 15% of 240 decomposes to 24 + 12 and answers 26.0.
8.6M parameters. This is a research artifact for studying whether small models can be made to solve problems verifiably, not a useful assistant.

License

Downloads last month: -; Downloads are not tracked for this model. How to track