code/README.md

Tiny cube-solving model

A small, non-reasoning model trained from scratch to solve a 3x3x3 from a single facelet state, scored by the same harness as every LLM on the board.

It is not a cube solver. packages/solver already solves any cube perfectly in ~23ms via Kociemba. This model is the control for the benchmark's central claim: frontier models score 0-80% on a task with a solved algorithm from 2008, and the obvious objection is that the task may simply be hard for neural computation. A tiny model settles that.

This file is the command reference. CLAUDE.md in this directory is the operating manual -- why the metric must be bucketed by solution length, why checkpoints must not be chosen by it, the decode temperature that makes verified sampling work, and the environment traps. Read that before changing anything here.

Result so far (42.0M params, ~1 hour, ~$2): 100% at depth 3, 69% at depth 6 (matching gpt-5.6-sol's 73%), 1.5% at depth 10, zero beyond -- a hard reach wall at about eight moves. Full write-up in docs/findings/2026-09-09-tiny-model-reach-wall.md.

Setup

python3 -m venv venv
venv/bin/pip install "setuptools<60" wheel
venv/bin/pip install --no-build-isolation "git+https://github.com/muodov/kociemba.git"
venv/bin/pip install torch transformers huggingface_hub fastapi uvicorn

Install kociemba from git, not PyPI. The 1.2.1 sdist on PyPI dates from 2017 and its build metadata is too old: on most machines it installs "successfully" while producing no C extension, silently falling back to a pure-Python implementation ~50x slower. Nothing errors -- the only symptom is that generation crawls while a rented box bills by the hour. gen_data.py therefore refuses to run below native speed, and vast_bootstrap.sh gates on it too.

Two diagnoses that look right and are not, recorded so they are not re-derived: missing libffi-dev (a real error, but not the cause) and python 3.10 vs 3.11 (also not the cause).

Generate data

venv/bin/python gen_data.py --count 10000000 --workers $(nproc) --out data/train.jsonl
venv/bin/python verify_data.py data/train.jsonl        # must report 0 FAILED

verify_data.py replays every pair through the engine. Never train on a dataset that has not been through it: a mislabelled pair is silent, and only surfaces much later as a benchmark number that looks like a modelling problem.

No compiler? Use the JavaScript generator instead.

npm install cubejs
node gen_data.cjs --count 10000000 --workers $(nproc) --out data/train.jsonl

gen_data.cjs has no compiled dependency, uses the solver the harness itself scores with, and samples the group uniformly via Cube.random(). About a third the throughput (~15 pairs/s/core against ~45), but it always works.

Why there is no depth curriculum

Every reachable state is at most 20 moves from solved, so a 200-move walk and a 50-move walk sample the same fully-mixed distribution. The model never sees "depth" -- it sees a state -- which is why depth-50 and depth-100 benchmark buckets need nothing special. Near-solved states are the genuinely rare tail, so --shallow-frac mixes them in deliberately.

Train

venv/bin/python train.py --data data/train.jsonl --val-data data/val.jsonl \
  --hidden 512 --layers 10 --batch-size 512 --max-steps 60000 \
  --out checkpoints/cube --hub-repo <user>/tiny-cube-solver

Watch SOLVE RATE, not loss. A cube has enormously many valid solutions and Kociemba emits one, so a model producing a different valid solve scores badly on token match and perfectly on what matters. Training decodes greedily and replays the moves through the engine, exactly as the harness does.

Generate the holdout with its own seed and --augment 0. Evaluating against the augmented mix inflates the number, since augmented states sit partway along a solution path and are nearer to solved.

--resume <dir-or-hub-id> continues from a checkpoint (weights only, so the LR schedule restarts). Checkpoints push to the Hub as training runs, which is what makes a reclaimed vast instance cost minutes rather than the whole run.

Evaluate

venv/bin/python eval_canonical.py --checkpoint checkpoints/cube
venv/bin/python eval_canonical.py --distribution-only     # no model needed
./run_benchmark.sh checkpoints/cube 200                   # through the real harness

eval_canonical.py scores the model on generateScrambleSet states -- the harness's own generator, a different code path from the training sampler. A gap between that and the in-training holdout would mean the model fitted its sampler rather than the task.

run_benchmark.sh is the real thing: it serves the checkpoint and runs bench.ts against it, so the model faces the canonical index-addressed scrambles and lands in results/ as an ordinary row. It uses n=200 per depth rather than the usual 10 -- that convention exists only because API calls cost money, and local inference is free.

Renting a box (vast.ai)

vast_bootstrap.sh runs the whole pipeline. Notes worth keeping:

• Size the disk deliberately. dph_total includes storage: a 100GB disk took a $0.112/hr box to $0.240/hr. 10M pairs is ~1GB; 20-30GB is plenty.
• Cores matter more than the GPU here. The model is small; generation is the wall-clock bottleneck.
• This repo is private, so a rented box cannot clone it. A curl to the GitHub API from inside a session misleadingly returns 200 because the outbound proxy injects credentials. Code ships via the model's own Hub repo instead, reusing the token already needed for checkpoints.
• /api/v0/instances/ is deprecated in favour of /api/v1/, but log retrieval works only on the v0 request_logs path.