H-Net dynamic-chunking pilot models
31 byte-level H-Net checkpoints (22.5M params, 6000 steps) trained to study what a learned chunker discovers when it is free to choose its own computational granularity. Every run shares data, seed schedule and FLOP budget; only the chunker's objective varies.
Runs
| family | n | what varies |
|---|---|---|
pilot_A_s{0,1,2} |
3 | no parity loss — the natural chunker (baseline) |
pilot_B_s{0,1,2} |
3 | parity loss, per-content-matched-sentence beta |
pilot_C_s{0,1,2} |
3 | parity applied then removed (reversion arm) |
pilot_D_s{0,1,2} |
3 | stronger equalisation |
pilot_E_s{0,1,2} |
3 | entropy-matched variant |
pilot_Apc_s{0,1,2} |
3 | per-codepoint control |
pilot_Bpc_s{0,1,2} |
3 | parity loss, per-codepoint beta |
emactl_noema_s{0,1,2} |
3 | EMA ablation control (built from A) |
emactl_genreg_s{0,1,2} |
3 | generic-regularisation control, entropy-matched to B |
jitter_s{0,1,2} |
3 | threshold-jitter training (tau ~ U[0.30, 0.75]) |
baseline_en |
1 | English-only baseline |
Format
Weights are model.safetensors; config.json carries model_config, the training config,
and a tied_weights map. emb.weight and head.weight are tied, so only emb.weight is
stored — restore with:
from safetensors.torch import load_file
import json
sd = load_file("model.safetensors")
cfg = json.load(open("config.json"))
for dst, src in cfg["tied_weights"].items():
sd[dst] = sd[src] # head.weight <- emb.weight
Verified bit-exact against the original training checkpoints (max|Δ| = 0.0, 111 tensors).
final.json and log.jsonl are the trainer's own records.
What these were used to find
- Segmentation crystallises after capability: the router lags BPB descent in 36/36 language-seed pairs at half of each clock's own change.
- Learned units are not linguistic. Morpheme F1 sits at or below a rate-matched random baseline; gold word+morph features add ≤0.001 held-out AUROC over computational features in 9/9 languages.
- Chunks are not functionally localised: aligned-minus-misaligned activation patching is at the construction rate (0/28 cells above 2 across-seed SD), while killing the whole chunk pathway costs +0.34 nats/byte. Information is additive and diffuse across chunks.
- The chunk-level main network is worth only ~0.07 BPB at this scale.
Negative results are reported as results. See the companion results dataset and probe dataset.
Limitations
16.2M/22.5M params, 6000 steps, one hierarchy stage, 3 seeds. Only the final step was
retained (keep_ckpt: false), so no intermediate weights exist. Third-party models used for
comparison (allenai/Bolmo-1B, BLT, ByT5) are not redistributed here.