Transformer baseline for the PhaseMesh SSM comparison (text8)
A fresh, matched Transformer baseline trained for Ash's PhaseMesh SSM comparison. Completed 5,000 optimizer updates and the final validation at ~02:15 UTC on 3 November 2026 (single run, seed 1).
| value | |
|---|---|
| Parameters | 130,575,232 |
| Training byte presentations | 327,680,000 |
| Best validation | 1.2401772355 bpc at step 4,000 |
| Final validation | 1.2575643889 bpc at step 5,000 |
| PhaseMesh SSM (reported) | best 1.2568 / final 1.2603 bpc |
Lower is better. The Transformer's best score is lower, but it worsened after step 4,000, and its final advantage is only ~0.00274 bpc. This is one seed with sampled validation (50 batches every 250 updates), so it does not establish a statistically reliable architecture winner. Evaluation sampling/configuration and hardware differ between the two runs, and the SSM checkpoint was not available for independent evaluation, so no matched hardware-speed claim is made.
Setup
text8, 90/5/5 split, context 1,024, effective batch 64 (microbatch 4 x 16 accumulation steps), BF16 training, FP32 evaluation.
Model: 11 layers, d_model 1088, 17 heads, d_ff = 2 x d_model, RoPE, tied embeddings, vocab 256 (bytes). Peak learning rate
3e-4 (min 1e-4, 300 warmup steps) versus the SSM's reported 3e-3. Full configuration in config.json; training code in
run_matched.py, idle_training.py, child_entry.py and vendor/phase_ssm/ (source commit 3ae43ffac6c0).
Files
| file | what | sha256 |
|---|---|---|
latest.pt |
final step-5000 checkpoint with resume state (model, 90 optimizer states, RNG) | 9daa4a11d354fed0bce207f5fb01c0e4b382cf4f1a2f30f31b6706f88bf98411 |
best.pt |
best step-4000 checkpoint, model-only | 4403abde8d0a2e2610d5b4e839df0cbce0d7cc59adef2afcbf1bcaaaebf7a35a |
config.json |
run + model configuration | 2ee79a2c5384459bbb9fa13d53ce8c7bc74f8fe17e9442fbe5db7ec7eb97902a |
BACKUP_MANIFEST.json |
SHA-256 manifest of every file in this bundle | |
RESTORE_VERIFIED.json / BEST_RESTORE_VERIFIED.json |
restore checks: strict CPU load, finite weights, RNG, optimizer-state count |
The final checkpoint passed a strict CPU restore, finite-weight checks, RNG verification and a check of its 90 optimizer states; the best checkpoint passed strict model loading and finite-weight checks (it carries no optimizer/RNG state).
Reference: the PhaseMesh SSM results and documented setup.
- Downloads last month
- -