PhaseMesh text8 comparison
Independent Transformer and PhaseMesh SSM runs based on Ash Hart's PhaseMesh commit 3ae43ffac6c0699857cada0865f156405fb95956.
Publication status
Both Transformer and SSM best and final model-only checkpoints are included, with configuration, measured results and reproducible training/model code. All four checkpoints passed strict CPU model loading and finite-tensor checks.
Measured results
Both runs used one NVIDIA RTX 3090 (24 GB), seed 1, text8 contiguous 90/5/5 train/validation/test split, context 1024, effective batch 64 (microbatch 4 x accumulation 16), 5000 optimizer updates and 327,680,000 byte presentations. BF16 training; FP32 sampled validation with 50 batches every 250 updates.
- Transformer: 130,575,232 parameters, best 1.2401772355061287 bpc at step 4000; final 1.257564388922874 at step 5000.
- SSM reproduction: 131,688,256 parameters, best 1.2638741508689288 bpc at step 4750; final 1.268647525903973 at step 5000.
- Upstream reported SSM reference: best 1.2568, final 1.2603. These are separate results, not this reproduction's scores.
Optimizer-update compute: Transformer 8665.858 seconds, SSM 30490.6774 seconds, approximately 3.52x longer for this SSM implementation. Excludes evaluation, saving and inference-priority pauses. One seed, different learning rates, 11 versus 12 layers and different implementations prevent a controlled architecture or universal speed conclusion.
Correction: the SSM uses the FFT backend. Phase memory is disabled. FFT itself is not disabled.
Files
transformer/best.pt and transformer/final.pt contain only model tensors, model configuration, model type, step and validation score. Neither includes optimizer or RNG state. Both load using torch.load(..., weights_only=True), pass strict model loading and finite-weight checks. Final is a model-only export of the independently verified full resume checkpoint; its hash differs from that original by design.
transformer/run_matched.py and transformer/vendor/phase_ssm contain the training runner and pinned model code. transformer/config.json preserves exact settings; transformer/progress.json contains measured evaluation history and compute timer. No text8 dataset is bundled. Supply the original extracted text8 file to the runner. --resume expects the runner's full resume-state format, not these model-only exports.
The Transformer uses width 1088, 11 layers, 17 attention heads, RoPE, feedforward multiplier 2 and tied byte embeddings. AdamW peak LR 0.0003, minimum 0.0001, 300 warmup updates. SSM uses width 1088, 12 layers, state dimension 64, expansion 1, short convolution 4, FF multiplier 2, activation checkpointing, peak LR 0.003. All are small byte-level research models, not instruction-tuned assistants.
The ssm/ directory contains best.pt (step 4750), final.pt (step 5000), config.json, progress.json, run_ssm.py, vendor source and its detailed README. SSM files likewise contain no optimizer or RNG state.
Source software is MIT licensed; the upstream copyright and license are preserved in transformer/LICENSE and ssm/LICENSE. Weight licensing is not separately asserted here. Original public Transformer release: https://huggingface.co/MarxistLeninist/phasemesh-transformer-baseline-text8