code/CLAUDE.md

# Tiny cube-solving model — operating manual

Context for working in `training/`. The root `CLAUDE.md` is the benchmark's
manual; this one is only about the model. `README.md` here is the command
reference; this file is the reasoning, the traps, and the state of the work.

## What this is for

**Not a cube solver.** `packages/solver` already solves any cube perfectly in
~23ms via Kociemba, and nothing here competes with that. The model is a
**control** for the benchmark's central claim: frontier models score 0–80% on a
task with a published exact algorithm from 2008, and the obvious objection is
that the task may just be hostile to neural computation. A tiny model tests that
objection directly.

Framed as a capability claim ("we built a cube solver") it is worthless. Framed
as a learnability claim it is the control the leaderboard was missing. Keep that
distinction in any writeup.

## Where it got to (2026-09-09)

42.0M parameters, ~1 hour of training, ~$2 of GPU. Scored through `bench.ts` on
the canonical scrambles at n=200 per depth:

| depth | 3 | 6 | 10 | 15 | 20+ |
|---|---|---|---|---|---|
| solve rate | **100%** | **69%** | 1.5% | 0% | 0% |

At depth 6 that is statistically indistinguishable from `gpt-5.6-sol` (73%,
n=11) — say **matches**, not beats. Full write-up, including what the result
does *not* support, in
[`docs/findings/2026-09-09-tiny-model-reach-wall.md`](../docs/findings/2026-09-09-tiny-model-reach-wall.md).

**The finding is the reach wall.** The falloff is a cliff at a particular
solution length, not gradual decay, and it did not move across 15,000 steps of
training — the model improves *within* its ~8-move reach while the reach itself
stays put.

## Model shape

Stock `LlamaConfig`, trained from scratch — the architecture, not Llama weights.
Nothing is downloaded. Pretraining has nothing to transfer: the vocabulary is 28
tokens, the task is one fixed function rather than language, and training data is
unlimited and verified. A standard architecture is used only so checkpoints load
with plain `transformers` and push to the Hub without custom code.

- hidden 512, 10 layers, 8 heads, seq 83 → **41.97M params, 168MB fp32**
- **Embeddings are 0.03% of parameters** (14k, from a 28-token vocabulary), against
  20–40% in a normal LLM. Essentially every parameter computes rather than stores.

## Things that will bite you

### Evaluate by solve rate, and break it down by length

A cube has enormously many valid solutions and Kociemba emits one, so a model
producing a *different* valid solve scores badly on token match and perfectly on
what matters. Never use loss or token accuracy as the metric.

**And never use a single aggregate over the holdout.** It is ~85% fully-mixed
states, so a model with an 8-move reach cannot exceed ~15% on it however well it
trains. On the first run that aggregate sat at 11–12% from step 13000 to 40000
and was read as a plateau three times, while depth 6 was in fact going 52% → 72%.
`solve_rate()` now buckets by solution length for exactly this reason.

### A saturated metric must never choose checkpoints

Same run, sharper consequence: checkpoints were saved only on improvement, so
with the metric pinned the run kept a **step-18000 model and discarded steps
18000–40000**. Fixed in `5779f7e` — latest weights always written to `--out`,
best-scoring to `--out-best`. If you change the save logic, keep that property.

### Install kociemba from git, never PyPI

The 1.2.1 sdist dates from 2017 and on most machines installs "successfully"
while producing **no C extension**, silently falling back to a pure-Python
implementation ~50x slower. Nothing errors; the only symptom is that generation
crawls while a rented box bills by the hour.

```bash
pip install --no-build-isolation "git+https://github.com/muodov/kociemba.git"
```

`gen_data.py` refuses to run below native speed and `vast_bootstrap.sh` gates on
it. Two diagnoses that look right and are not, recorded so they are not
re-derived: **missing `libffi-dev`** (a real error, but not the cause) and
**python 3.10 vs 3.11** (also not the cause).

`gen_data.cjs` is the escape hatch — cubejs, pure JavaScript, no compiler, and it
is the solver the harness scores with. About a third the throughput; always works.

### Verified decoding needs a high temperature

Decoding several candidates and replaying each through the engine is free and
exact, but the obvious temperature makes it useless. Measured over greedy
failures, 16 samples each:

| temp | distinct candidates | failures rescued |
|---|---|---|
| 0.7 | 2.8 | **0/5** |
| 1.0 | 4.0 | 1/5 |
| 1.5 | 11.0 | 2/5 |

The model is *confidently wrong*, not uncertain, so mild sampling redraws the
same answer. Default is 1.5. Keep `attempts=1` (bare greedy) as the model's own
score and report anything above it separately — it is a **system** score.

### Generation is CPU-bound; the GPU is idle for it

Each pair needs a Kociemba solve, which is IDA* tree search — branchy control
flow and random pruning-table lookups, the opposite of what a GPU does. The lever
is fewer solver calls, not more GPU. `--augment 8` takes several states from each
solution path for an ~8x cut (measured 7.8x).

Augmentation also *improves* coverage rather than harming it, contrary to the
first assumption here: uniformly sampled states are ~95% distance 18–20, whereas
sampling along solution paths spans distance 1–20 evenly, matching a benchmark
whose buckets start at depth 3. **Keep the holdout unaugmented and uniform** so
evaluation stays against the hard case.

### `nproc` lies on a rented box

A machine advertising 384 cores gave **73** under its cgroup quota. Sizing the
pool by the host count puts five processes on every CPU. Aggregate throughput is
unchanged (the quota binds either way) but every ETA derived from it is wrong by
that factor — which is how a 20-minute job was predicted for a 100-minute one.
`--workers 0` (the default) reads the real quota and prints both numbers.

### vast.ai

- **Check the GPU before generating data.** torch cu124 has no Blackwell (sm_120)
  kernels; `torch.cuda.is_available()` returns true and it fails only when a
  kernel runs. That was discovered *after* 50 minutes of generation on a box that
  could not train. Install cu128 and run the smoke test (real model, one
  forward+backward) first.
- **Size the disk deliberately.** `dph_total` includes storage: a 100GB disk took
  a $0.112/hr box to $0.240/hr. 6M pairs is ~1GB; 20–30GB is plenty.
- **This repo is private**, so a rented box cannot clone it. A curl to the GitHub
  API from inside a session misleadingly returns 200 because the outbound proxy
  injects credentials. Code ships via the model's own Hub repo instead.
- `/api/v0/instances/` is deprecated in favour of `/api/v1/`, but **log retrieval
  works only on the v0 `request_logs` path**.
- **Archive the instance log before destroying the box.** The first run's raw
  training log was lost this way; only a reconstruction survives.
- **Avoid boxes behind a national firewall.** A machine in CN could not reach
  Hugging Face at all, and `snapshot_download` does not raise on that -- it
  returns the empty local dir, so the fetch *looks* successful until a later step
  cannot find a file, and the run exits 0 having done nothing. Such a box also
  cannot push checkpoints, so it is unusable for this workflow regardless. The
  onstart script now checks for `vast_bootstrap.sh` after the download and aborts
  loudly if it is missing.

## Not a like-for-like comparison, and say so

`serve.py` parses the 54 facelets out of the prompt and feeds the model only
those. The LLMs must read an ASCII net out of prose. That is aligned with intent
— the whole v1→v2 prompt change existed to remove net-parsing as a confound —
but it is **not identical conditions**, and any writeup has to state it.

Overfitting, by contrast, is not a plausible worry and has been checked: 4.3e19
states against 6M training pairs (~1e-13 of the space), and 200k states from the
training seed versus 200k from the holdout seed gave 400k distinct states with
**zero overlap**. `eval_canonical.py` additionally scores against
`generateScrambleSet` — a different code path from the training sampler — so a
model fitted to its sampler would show a gap there.

## The depth hypothesis is dead (2026-09-10)

The ~8-move reach wall looked like a depth limit: one step of sequential
computation per layer, 10 layers, ~8-move reach. It is not. Two arms at matched
parameters -- 720x10 (83.0M) and 512x20 (83.9M), identical data, batch, schedule
and steps -- scored the same at all nine matched checkpoints, finished at exactly
10.9%, and produced **identical per-depth results**: 100% at depth 3, 64% at
depth 6, zero from depth 10 up, down to the same single 15+ holdout solve each.

The practical conclusion inverts the hypothesis. **Prefer width.** Depth cost
3.3x the wall-clock (0.9 it/s against 3.0 -- layers run sequentially and do not
parallelise the way width does) and 36% more memory at identical batch (33.1GB
against 24.4GB, twice the layers holding activations for the backward pass), for
no gain whatsoever.

See [`docs/findings/2026-09-10-depth-does-not-buy-reach.md`](../docs/findings/2026-09-10-depth-does-not-buy-reach.md).

## Four fixes tried, four negatives (2026-09-11)

| hypothesis | intervention | result |
|---|---|---|
| not enough capacity | 42M -> 83M | no change |
| not enough sequential compute | 20 layers vs 10, matched params | identical, to the solve |
| blind long-horizon execution | interactive chunking every 8 moves | no change |
| exposure bias | 2M expert-labelled states from its own rollouts | 69->72% at depth 6, 1.5->2.0% at depth 10 (noise) |

**Do not re-run these.** Each is a plausible story that is now eliminated, and
one further diagnosis was checked and is also wrong: Kociemba's solutions are
strictly monotonic, so the labels do teach progress.

What is left is a cliff in *policy quality*, not gradual degradation -- excellent
within ~8 moves, actively counterproductive beyond. Every intervention kept the
same shape: one forward pass, one emitted sequence, **no search**. That is
precisely what the literature does not do. DeepCubeA (Nature MI 2019) learns a
value function and runs A* over it with a comparably small network; Kociemba is
IDA* over pruning tables.

See [`docs/findings/2026-09-11-four-failed-fixes-the-wall-needs-search.md`](../docs/findings/2026-09-11-four-failed-fixes-the-wall-needs-search.md).

## Next experiments, in priority order

1. **Add search over the existing model.** Keep the trained network as a
   heuristic and run a shallow beam or IDA* over it, with the engine as the
   transition function. The search supplies the horizon the policy cannot, and it
   reuses everything already built. Cheapest path to testing the search
   hypothesis.
2. **Train a value function instead of a policy (DeepCubeA's shape).** Predict
   distance-to-solved rather than a move sequence, then search over it. This is
   the approach with published evidence behind it, and a genuinely different
   project rather than another variation on seq2seq.
3. **RL for solution length**, only once something reaches deep scrambles at all.
   Reward `solved ? (C - len) : 0`; the verifier is already exact and free.
   Pointless while the model cannot solve the states it would be optimising.