code/CLAUDE.md

# Tiny cube-solving model — operating manual

Context for working in `training/`. The root `CLAUDE.md` is the benchmark's
manual; this one is only about the model. `README.md` here is the command
reference; this file is the reasoning, the traps, and the state of the work.

## What this is for

**Not a cube solver.** `packages/solver` already solves any cube perfectly in
~23ms via Kociemba, and nothing here competes with that. The model is a
**control** for the benchmark's central claim: frontier models score 0–80% on a
task with a published exact algorithm from 2008, and the obvious objection is
that the task may just be hostile to neural computation. A tiny model tests that
objection directly.

Framed as a capability claim ("we built a cube solver") it is worthless. Framed
as a learnability claim it is the control the leaderboard was missing. Keep that
distinction in any writeup.

## Where it got to (2026-09-09)

42.0M parameters, ~1 hour of training, ~$2 of GPU. Scored through `bench.ts` on
the canonical scrambles at n=200 per depth:

| depth | 3 | 6 | 10 | 15 | 20+ |
|---|---|---|---|---|---|
| solve rate | **100%** | **69%** | 1.5% | 0% | 0% |

At depth 6 that is statistically indistinguishable from `gpt-5.6-sol` (73%,
n=11) — say **matches**, not beats. Full write-up, including what the result
does *not* support, in
[`docs/findings/2026-09-09-tiny-model-reach-wall.md`](../docs/findings/2026-09-09-tiny-model-reach-wall.md).

**The finding is the reach wall.** The falloff is a cliff at a particular
solution length, not gradual decay, and it did not move across 15,000 steps of
training — the model improves *within* its ~8-move reach while the reach itself
stays put.

## Model shape

Stock `LlamaConfig`, trained from scratch — the architecture, not Llama weights.
Nothing is downloaded. Pretraining has nothing to transfer: the vocabulary is 28
tokens, the task is one fixed function rather than language, and training data is
unlimited and verified. A standard architecture is used only so checkpoints load
with plain `transformers` and push to the Hub without custom code.

- hidden 512, 10 layers, 8 heads, seq 83 → **41.97M params, 168MB fp32**
- **Embeddings are 0.03% of parameters** (14k, from a 28-token vocabulary), against
  20–40% in a normal LLM. Essentially every parameter computes rather than stores.

## Things that will bite you

### Evaluate by solve rate, and break it down by length

A cube has enormously many valid solutions and Kociemba emits one, so a model
producing a *different* valid solve scores badly on token match and perfectly on
what matters. Never use loss or token accuracy as the metric.

**And never use a single aggregate over the holdout.** It is ~85% fully-mixed
states, so a model with an 8-move reach cannot exceed ~15% on it however well it
trains. On the first run that aggregate sat at 11–12% from step 13000 to 40000
and was read as a plateau three times, while depth 6 was in fact going 52% → 72%.
`solve_rate()` now buckets by solution length for exactly this reason.

### A saturated metric must never choose checkpoints

Same run, sharper consequence: checkpoints were saved only on improvement, so
with the metric pinned the run kept a **step-18000 model and discarded steps
18000–40000**. Fixed in `5779f7e` — latest weights always written to `--out`,
best-scoring to `--out-best`. If you change the save logic, keep that property.

### Install kociemba from git, never PyPI

The 1.2.1 sdist dates from 2017 and on most machines installs "successfully"
while producing **no C extension**, silently falling back to a pure-Python
implementation ~50x slower. Nothing errors; the only symptom is that generation
crawls while a rented box bills by the hour.

```bash
pip install --no-build-isolation "git+https://github.com/muodov/kociemba.git"
```

`gen_data.py` refuses to run below native speed and `vast_bootstrap.sh` gates on
it. Two diagnoses that look right and are not, recorded so they are not
re-derived: **missing `libffi-dev`** (a real error, but not the cause) and
**python 3.10 vs 3.11** (also not the cause).

`gen_data.cjs` is the escape hatch — cubejs, pure JavaScript, no compiler, and it
is the solver the harness scores with. About a third the throughput; always works.

### Verified decoding needs a high temperature

Decoding several candidates and replaying each through the engine is free and
exact, but the obvious temperature makes it useless. Measured over greedy
failures, 16 samples each:

| temp | distinct candidates | failures rescued |
|---|---|---|
| 0.7 | 2.8 | **0/5** |
| 1.0 | 4.0 | 1/5 |
| 1.5 | 11.0 | 2/5 |

The model is *confidently wrong*, not uncertain, so mild sampling redraws the
same answer. Default is 1.5. Keep `attempts=1` (bare greedy) as the model's own
score and report anything above it separately — it is a **system** score.

### Generation is CPU-bound; the GPU is idle for it

Each pair needs a Kociemba solve, which is IDA* tree search — branchy control
flow and random pruning-table lookups, the opposite of what a GPU does. The lever
is fewer solver calls, not more GPU. `--augment 8` takes several states from each
solution path for an ~8x cut (measured 7.8x).

Augmentation also *improves* coverage rather than harming it, contrary to the
first assumption here: uniformly sampled states are ~95% distance 18–20, whereas
sampling along solution paths spans distance 1–20 evenly, matching a benchmark
whose buckets start at depth 3. **Keep the holdout unaugmented and uniform** so
evaluation stays against the hard case.

### `nproc` lies on a rented box

A machine advertising 384 cores gave **73** under its cgroup quota. Sizing the
pool by the host count puts five processes on every CPU. Aggregate throughput is
unchanged (the quota binds either way) but every ETA derived from it is wrong by
that factor — which is how a 20-minute job was predicted for a 100-minute one.
`--workers 0` (the default) reads the real quota and prints both numbers.

### vast.ai

- **Check the GPU before generating data.** torch cu124 has no Blackwell (sm_120)
  kernels; `torch.cuda.is_available()` returns true and it fails only when a
  kernel runs. That was discovered *after* 50 minutes of generation on a box that
  could not train. Install cu128 and run the smoke test (real model, one
  forward+backward) first.
- **Size the disk deliberately.** `dph_total` includes storage: a 100GB disk took
  a $0.112/hr box to $0.240/hr. 6M pairs is ~1GB; 20–30GB is plenty.
- **This repo is private**, so a rented box cannot clone it. A curl to the GitHub
  API from inside a session misleadingly returns 200 because the outbound proxy
  injects credentials. Code ships via the model's own Hub repo instead.
- `/api/v0/instances/` is deprecated in favour of `/api/v1/`, but **log retrieval
  works only on the v0 `request_logs` path**.
- **Archive the instance log before destroying the box.** The first run's raw
  training log was lost this way; only a reconstruction survives.
- **Avoid boxes behind a national firewall.** A machine in CN could not reach
  Hugging Face at all, and `snapshot_download` does not raise on that -- it
  returns the empty local dir, so the fetch *looks* successful until a later step
  cannot find a file, and the run exits 0 having done nothing. Such a box also
  cannot push checkpoints, so it is unusable for this workflow regardless. The
  onstart script now checks for `vast_bootstrap.sh` after the download and aborts
  loudly if it is missing.

## Not a like-for-like comparison, and say so

`serve.py` parses the 54 facelets out of the prompt and feeds the model only
those. The LLMs must read an ASCII net out of prose. That is aligned with intent
— the whole v1→v2 prompt change existed to remove net-parsing as a confound —
but it is **not identical conditions**, and any writeup has to state it.

Overfitting, by contrast, is not a plausible worry and has been checked: 4.3e19
states against 6M training pairs (~1e-13 of the space), and 200k states from the
training seed versus 200k from the holdout seed gave 400k distinct states with
**zero overlap**. `eval_canonical.py` additionally scores against
`generateScrambleSet` — a different code path from the training sampler — so a
model fitted to its sampler would show a gap there.

## The depth hypothesis is dead (2026-09-10)

The ~8-move reach wall looked like a depth limit: one step of sequential
computation per layer, 10 layers, ~8-move reach. It is not. Two arms at matched
parameters -- 720x10 (83.0M) and 512x20 (83.9M), identical data, batch, schedule
and steps -- scored the same at all nine matched checkpoints, finished at exactly
10.9%, and produced **identical per-depth results**: 100% at depth 3, 64% at
depth 6, zero from depth 10 up, down to the same single 15+ holdout solve each.

The practical conclusion inverts the hypothesis. **Prefer width.** Depth cost
3.3x the wall-clock (0.9 it/s against 3.0 -- layers run sequentially and do not
parallelise the way width does) and 36% more memory at identical batch (33.1GB
against 24.4GB, twice the layers holding activations for the backward pass), for
no gain whatsoever.

See [`docs/findings/2026-09-10-depth-does-not-buy-reach.md`](../docs/findings/2026-09-10-depth-does-not-buy-reach.md).

## Next experiments, in priority order
1. **Chunked inference.** Feed the true state back every ~8 moves. With depth
   eliminated, this is the intervention most likely to work: it sidesteps the
   compounding-error arithmetic entirely and plays directly to a model that is
   already 100% at depth 3. It makes the task interactive rather than one-shot,
   so it is a **different benchmark mode**, not a config flag, and its numbers
   must not sit in the same table.
2. **Train on the model's own rollouts (DAgger-style).** The model imitates one
   Kociemba solution per state and is never taught to recover from its own drift,
   which is exactly what a long solve requires. This is textbook exposure bias,
   and it is the remaining explanation for the wall now that architecture is
   ruled out.
3. **Re-run the discarded steps.** The benchmarked model is step 18000 of 40000;
   with the metric and checkpointing now fixed, resuming would show whether depth
   6 kept climbing past 72%. ~20 minutes, ~$0.50.
4. **RL for solution length.** Reward `solved ? (C - len) : 0`. The verifier is
   already exact and free. Closest to the "train a reasoning model" question.