code/README.md

# Tiny cube-solving model

A small, non-reasoning model trained from scratch to solve a 3x3x3 from a single
facelet state, scored by the same harness as every LLM on the board.

**It is not a cube solver.** `packages/solver` already solves any cube perfectly
in ~23ms via Kociemba. This model is the *control* for the benchmark's central
claim: frontier models score 0-80% on a task with a solved algorithm from 2008,
and the obvious objection is that the task may simply be hard for neural
computation. A tiny model settles that.

**This file is the command reference. [`CLAUDE.md`](CLAUDE.md) in this directory
is the operating manual** -- why the metric must be bucketed by solution length,
why checkpoints must not be chosen by it, the decode temperature that makes
verified sampling work, and the environment traps. Read that before changing
anything here.

Result so far (42.0M params, ~1 hour, ~$2): 100% at depth 3, 69% at depth 6
(matching `gpt-5.6-sol`'s 73%), 1.5% at depth 10, zero beyond -- a hard reach
wall at about eight moves. Full write-up in
[`docs/findings/2026-09-09-tiny-model-reach-wall.md`](../docs/findings/2026-09-09-tiny-model-reach-wall.md).

## Setup

```bash
python3 -m venv venv
venv/bin/pip install "setuptools<60" wheel
venv/bin/pip install --no-build-isolation "git+https://github.com/muodov/kociemba.git"
venv/bin/pip install torch transformers huggingface_hub fastapi uvicorn
```

**Install kociemba from git, not PyPI.** The 1.2.1 sdist on PyPI dates from 2017
and its build metadata is too old: on most machines it installs "successfully"
while producing no C extension, silently falling back to a pure-Python
implementation ~50x slower. Nothing errors -- the only symptom is that generation
crawls while a rented box bills by the hour. `gen_data.py` therefore refuses to
run below native speed, and `vast_bootstrap.sh` gates on it too.

Two diagnoses that look right and are not, recorded so they are not re-derived:
missing `libffi-dev` (a real error, but not the cause) and python 3.10 vs 3.11
(also not the cause).

## Generate data

```bash
venv/bin/python gen_data.py --count 10000000 --workers $(nproc) --out data/train.jsonl
venv/bin/python verify_data.py data/train.jsonl        # must report 0 FAILED
```

`verify_data.py` replays every pair through the engine. Never train on a dataset
that has not been through it: a mislabelled pair is silent, and only surfaces much
later as a benchmark number that looks like a modelling problem.

**No compiler? Use the JavaScript generator instead.**

```bash
npm install cubejs
node gen_data.cjs --count 10000000 --workers $(nproc) --out data/train.jsonl
```

`gen_data.cjs` has no compiled dependency, uses the solver the harness itself
scores with, and samples the group uniformly via `Cube.random()`. About a third
the throughput (~15 pairs/s/core against ~45), but it always works.

### Why there is no depth curriculum

Every reachable state is at most 20 moves from solved, so a 200-move walk and a
50-move walk sample the same fully-mixed distribution. The model never sees
"depth" -- it sees a state -- which is why depth-50 and depth-100 benchmark
buckets need nothing special. Near-solved states are the genuinely rare tail, so
`--shallow-frac` mixes them in deliberately.

## Train

```bash
venv/bin/python train.py --data data/train.jsonl --val-data data/val.jsonl \
  --hidden 512 --layers 10 --batch-size 512 --max-steps 60000 \
  --out checkpoints/cube --hub-repo <user>/tiny-cube-solver
```

Watch **`SOLVE RATE`**, not loss. A cube has enormously many valid solutions and
Kociemba emits one, so a model producing a *different* valid solve scores badly
on token match and perfectly on what matters. Training decodes greedily and
replays the moves through the engine, exactly as the harness does.

Generate the holdout with its own seed and `--augment 0`. Evaluating against the
augmented mix inflates the number, since augmented states sit partway along a
solution path and are nearer to solved.

`--resume <dir-or-hub-id>` continues from a checkpoint (weights only, so the LR
schedule restarts). Checkpoints push to the Hub as training runs, which is what
makes a reclaimed vast instance cost minutes rather than the whole run.

## Evaluate

```bash
venv/bin/python eval_canonical.py --checkpoint checkpoints/cube
venv/bin/python eval_canonical.py --distribution-only     # no model needed
./run_benchmark.sh checkpoints/cube 200                   # through the real harness
```

`eval_canonical.py` scores the model on `generateScrambleSet` states -- the
harness's own generator, a different code path from the training sampler. A gap
between that and the in-training holdout would mean the model fitted its sampler
rather than the task.

`run_benchmark.sh` is the real thing: it serves the checkpoint and runs `bench.ts`
against it, so the model faces the canonical index-addressed scrambles and lands
in `results/` as an ordinary row. It uses `n=200` per depth rather than the usual
10 -- that convention exists only because API calls cost money, and local
inference is free.

## Renting a box (vast.ai)

`vast_bootstrap.sh` runs the whole pipeline. Notes worth keeping:

- **Size the disk deliberately.** `dph_total` includes storage: a 100GB disk took
  a $0.112/hr box to $0.240/hr. 10M pairs is ~1GB; 20-30GB is plenty.
- **Cores matter more than the GPU here.** The model is small; generation is the
  wall-clock bottleneck.
- **This repo is private**, so a rented box cannot clone it. A curl to the GitHub
  API from inside a session misleadingly returns 200 because the outbound proxy
  injects credentials. Code ships via the model's own Hub repo instead, reusing
  the token already needed for checkpoints.
- `/api/v0/instances/` is deprecated in favour of `/api/v1/`, but log retrieval
  works *only* on the v0 `request_logs` path.