code/CLAUDE.md

Tiny cube-solving model — operating manual

Context for working in training/. The root CLAUDE.md is the benchmark's manual; this one is only about the model. README.md here is the command reference; this file is the reasoning, the traps, and the state of the work.

What this is for

Not a cube solver. packages/solver already solves any cube perfectly in ~23ms via Kociemba, and nothing here competes with that. The model is a control for the benchmark's central claim: frontier models score 0–80% on a task with a published exact algorithm from 2008, and the obvious objection is that the task may just be hostile to neural computation. A tiny model tests that objection directly.

Framed as a capability claim ("we built a cube solver") it is worthless. Framed as a learnability claim it is the control the leaderboard was missing. Keep that distinction in any writeup.

Where it got to (2026-09-09)

42.0M parameters, ~1 hour of training, ~$2 of GPU. Scored through bench.ts on the canonical scrambles at n=200 per depth:

depth	3	6	10	15	20+
solve rate	100%	69%	1.5%	0%	0%

At depth 6 that is statistically indistinguishable from gpt-5.6-sol (73%, n=11) — say matches, not beats. Full write-up, including what the result does not support, in docs/findings/2026-09-09-tiny-model-reach-wall.md.

The finding is the reach wall. The falloff is a cliff at a particular solution length, not gradual decay, and it did not move across 15,000 steps of training — the model improves within its ~8-move reach while the reach itself stays put.

Model shape

Stock LlamaConfig, trained from scratch — the architecture, not Llama weights. Nothing is downloaded. Pretraining has nothing to transfer: the vocabulary is 28 tokens, the task is one fixed function rather than language, and training data is unlimited and verified. A standard architecture is used only so checkpoints load with plain transformers and push to the Hub without custom code.

• hidden 512, 10 layers, 8 heads, seq 83 → 41.97M params, 168MB fp32
• Embeddings are 0.03% of parameters (14k, from a 28-token vocabulary), against 20–40% in a normal LLM. Essentially every parameter computes rather than stores.

Things that will bite you

Evaluate by solve rate, and break it down by length

A cube has enormously many valid solutions and Kociemba emits one, so a model producing a different valid solve scores badly on token match and perfectly on what matters. Never use loss or token accuracy as the metric.

And never use a single aggregate over the holdout. It is ~85% fully-mixed states, so a model with an 8-move reach cannot exceed ~15% on it however well it trains. On the first run that aggregate sat at 11–12% from step 13000 to 40000 and was read as a plateau three times, while depth 6 was in fact going 52% → 72%. solve_rate() now buckets by solution length for exactly this reason.

A saturated metric must never choose checkpoints

Same run, sharper consequence: checkpoints were saved only on improvement, so with the metric pinned the run kept a step-18000 model and discarded steps 18000–40000. Fixed in 5779f7e — latest weights always written to --out, best-scoring to --out-best. If you change the save logic, keep that property.

Install kociemba from git, never PyPI

The 1.2.1 sdist dates from 2017 and on most machines installs "successfully" while producing no C extension, silently falling back to a pure-Python implementation ~50x slower. Nothing errors; the only symptom is that generation crawls while a rented box bills by the hour.

pip install --no-build-isolation "git+https://github.com/muodov/kociemba.git"

gen_data.py refuses to run below native speed and vast_bootstrap.sh gates on it. Two diagnoses that look right and are not, recorded so they are not re-derived: missing libffi-dev (a real error, but not the cause) and python 3.10 vs 3.11 (also not the cause).

gen_data.cjs is the escape hatch — cubejs, pure JavaScript, no compiler, and it is the solver the harness scores with. About a third the throughput; always works.

Verified decoding needs a high temperature

Decoding several candidates and replaying each through the engine is free and exact, but the obvious temperature makes it useless. Measured over greedy failures, 16 samples each:

temp	distinct candidates	failures rescued
0.7	2.8	0/5
1.0	4.0	1/5
1.5	11.0	2/5

The model is confidently wrong, not uncertain, so mild sampling redraws the same answer. Default is 1.5. Keep attempts=1 (bare greedy) as the model's own score and report anything above it separately — it is a system score.

Generation is CPU-bound; the GPU is idle for it

Each pair needs a Kociemba solve, which is IDA* tree search — branchy control flow and random pruning-table lookups, the opposite of what a GPU does. The lever is fewer solver calls, not more GPU. --augment 8 takes several states from each solution path for an ~8x cut (measured 7.8x).

Augmentation also improves coverage rather than harming it, contrary to the first assumption here: uniformly sampled states are ~95% distance 18–20, whereas sampling along solution paths spans distance 1–20 evenly, matching a benchmark whose buckets start at depth 3. Keep the holdout unaugmented and uniform so evaluation stays against the hard case.

nproc lies on a rented box

A machine advertising 384 cores gave 73 under its cgroup quota. Sizing the pool by the host count puts five processes on every CPU. Aggregate throughput is unchanged (the quota binds either way) but every ETA derived from it is wrong by that factor — which is how a 20-minute job was predicted for a 100-minute one. --workers 0 (the default) reads the real quota and prints both numbers.

vast.ai

• Check the GPU before generating data. torch cu124 has no Blackwell (sm_120) kernels; torch.cuda.is_available() returns true and it fails only when a kernel runs. That was discovered after 50 minutes of generation on a box that could not train. Install cu128 and run the smoke test (real model, one forward+backward) first.
• Size the disk deliberately. dph_total includes storage: a 100GB disk took a $0.112/hr box to $0.240/hr. 6M pairs is ~1GB; 20–30GB is plenty.
• This repo is private, so a rented box cannot clone it. A curl to the GitHub API from inside a session misleadingly returns 200 because the outbound proxy injects credentials. Code ships via the model's own Hub repo instead.
• /api/v0/instances/ is deprecated in favour of /api/v1/, but log retrieval works only on the v0 request_logs path.
• Archive the instance log before destroying the box. The first run's raw training log was lost this way; only a reconstruction survives.
• Avoid boxes behind a national firewall. A machine in CN could not reach Hugging Face at all, and snapshot_download does not raise on that -- it returns the empty local dir, so the fetch looks successful until a later step cannot find a file, and the run exits 0 having done nothing. Such a box also cannot push checkpoints, so it is unusable for this workflow regardless. The onstart script now checks for vast_bootstrap.sh after the download and aborts loudly if it is missing.

Not a like-for-like comparison, and say so

serve.py parses the 54 facelets out of the prompt and feeds the model only those. The LLMs must read an ASCII net out of prose. That is aligned with intent — the whole v1→v2 prompt change existed to remove net-parsing as a confound — but it is not identical conditions, and any writeup has to state it.

Overfitting, by contrast, is not a plausible worry and has been checked: 4.3e19 states against 6M training pairs (~1e-13 of the space), and 200k states from the training seed versus 200k from the holdout seed gave 400k distinct states with zero overlap. eval_canonical.py additionally scores against generateScrambleSet — a different code path from the training sampler — so a model fitted to its sampler would show a gap there.

The depth hypothesis is dead (2026-09-10)

The ~8-move reach wall looked like a depth limit: one step of sequential computation per layer, 10 layers, ~8-move reach. It is not. Two arms at matched parameters -- 720x10 (83.0M) and 512x20 (83.9M), identical data, batch, schedule and steps -- scored the same at all nine matched checkpoints, finished at exactly 10.9%, and produced identical per-depth results: 100% at depth 3, 64% at depth 6, zero from depth 10 up, down to the same single 15+ holdout solve each.

The practical conclusion inverts the hypothesis. Prefer width. Depth cost 3.3x the wall-clock (0.9 it/s against 3.0 -- layers run sequentially and do not parallelise the way width does) and 36% more memory at identical batch (33.1GB against 24.4GB, twice the layers holding activations for the backward pass), for no gain whatsoever.

See docs/findings/2026-09-10-depth-does-not-buy-reach.md.

Next experiments, in priority order

1. Chunked inference. Feed the true state back every ~8 moves. With depth eliminated, this is the intervention most likely to work: it sidesteps the compounding-error arithmetic entirely and plays directly to a model that is already 100% at depth 3. It makes the task interactive rather than one-shot, so it is a different benchmark mode, not a config flag, and its numbers must not sit in the same table.
2. Train on the model's own rollouts (DAgger-style). The model imitates one Kociemba solution per state and is never taught to recover from its own drift, which is exactly what a long solve requires. This is textbook exposure bias, and it is the remaining explanation for the wall now that architecture is ruled out.
3. Re-run the discarded steps. The benchmarked model is step 18000 of 40000; with the metric and checkpointing now fixed, resuming would show whether depth 6 kept climbing past 72%. ~20 minutes, ~$0.50.
4. RL for solution length. Reward solved ? (C - len) : 0. The verifier is already exact and free. Closest to the "train a reasoning model" question.