AdaptiveChunking/hnet-chunking-spec-82m

H-Net dynamic-chunking spec models (82.5M) — with full training trajectories

Eight byte-level H-Net runs at 82,568,832 parameters, one hierarchy stage, trained to study what a learned chunker discovers when it is free to choose its own computational granularity — and, unlike the 22.5M pilots, when it discovers it.

The pilots at AdaptiveChunking/hnet-chunking-pilots retained only a final checkpoint. These runs retain the whole log-spaced trajectory, which is the point: it is what makes training-dynamics and activation-patching questions answerable at multiple steps rather than at one. That trajectory cannot be reconstructed after the fact.

Runs

run	condition	seed	steps	training bytes	checkpoints
spec_A_s{0,1,2}	A — baseline: global ratio loss throughout, router never frozen	0,1,2	11,000	0.72 GB	21 each
spec_B_s{0,1,2}	B — the schedule: global → parity-A → freeze	0,1,2	11,000	0.72 GB	21 each
specfull_A_s0	A, at the full data budget	0	76,300	5.00 GB	30
specfull_B_s0	B, at the full data budget	0	76,300	5.00 GB	30

Data, seed schedule and FLOP budget are held identical within a scale; only the chunker's objective varies. Architecture: d_enc 384, d_main 768, n_main 10, seq_len 1024, batch 64 (specfull), corpus FineWeb2 + FineWeb-v1 (en).

What they show

The parity effect replicates at 3.7× the pilot's parameters, and at the full data budget, with essentially no likelihood cost:

scale	Gini(chunks/sentence) A → B	reduction	high-resource BPB cost
pilot, 22.5M, 0.2 GB, 3 seeds	0.2195 → 0.0831	−62.1%	+0.38%
spec, 82.5M, 0.72 GB, 3 seeds	0.2181 → 0.0997	−54.3%	−0.82% (B is better)
specfull, 82.5M, 5.00 GB, 1 seed	0.2201 → 0.1037	−52.9%	+0.29%

Same direction, same magnitude, same chunks-per-sentence vs chunks-per-character asymmetry. Mean BPB drops from 1.376 (0.72 GB) to 1.151 (5.00 GB), so the full-budget models are genuinely better trained, not just longer.

Format

Each run directory holds {step:06d}.safetensors (one per retained checkpoint), config.json, final.json and log.jsonl. config.json carries model_config, the training config, a checkpoints list mapping tag → step, and a tied_weights map. emb.weight and head.weight are tied, so only emb.weight is stored — restore with:

import json
from safetensors.torch import load_file

sd = load_file("spec_A_s0/011000.safetensors")
cfg = json.load(open("spec_A_s0/config.json"))
for dst, src in cfg["tied_weights"].items():
    sd[dst] = sd[src]

manifest.json at the repo root lists every run with its parameter count, condition, seed, final step and checkpoint count.

Probe trajectories for these runs (253 npz per 11k-step run, 361 per full run) live alongside the pilots' in AdaptiveChunking/hnet-chunking-probes under the same spec_* / specfull_* names.

Known issues — carried over from the pilots, and still true here

These are documented rather than hidden, because they change what the logged fields mean:

• mask and mask_level1 are byte-for-byte identical. Everything here is a single-stage model (S=1); mask_level1 is a placeholder, not a second hierarchy level. Any analysis treating it as one is measuring the same thing twice.
• Thai len(mask) runs 21 bytes long relative to the text it indexes.
• Boundaries phase-lock sub-character on several scripts — the router does not respect codepoint edges, which is expected for a byte-level model but matters when aligning chunks to linguistic units.
• Analyses of these runs so far find that capability crystallises before segmentation does, that learned chunks are not functionally localised, and that the units are not linguistic. Treat "chunk" as "a unit of compute allocation", not "a morpheme".

Caveats

• Conditions C, D and E were not run at this scale — only A and B. The path-dependence/lock-in result (condition C's reversion drift) exists only at 22.5M and may be an artifact of a 6000-step run with a 40%/70% phase split.
• specfull is one seed per condition. The three-seed spread exists only at the 0.72 GB budget.
• Reported Gini is over chunks per content-matched sentence, computed on 12 languages with high-resource = en, de, ru, zh. A Gini is not comparable across different language sets.

Provenance

Produced by tier1/train.py in beetle-hnet; launched by tier1/launch_spec_overnight.sh and tier1/launch_spec_full.sh, packaged by tier1/push_spec_to_hf.py. Analysis in results/exp21_tier1_spec/.