Tern-1.5
A 236 M-parameter native ternary (BitNet b1.58, weights ∈ {−1, 0, +1}) language model with a sparse hourglass architecture. Trained end-to-end on a single Kaggle TPU v5e-8 session in 2 h 37 m on 5.84 B tokens.
236,307,968 total params · 118,339,072 active/token (50.1%) · 89.3% ternary
2.64 effective bits/param · 447 MFLOP/token · val ppl 30.3 · 738,918 tok/s
post-trained: 45.4% exact-match tool calling on a 7-dialect held-out probe · 564.5 MFLOP/token · 2.52 h · see 8.4 for lineage
The root checkpoint is the post-trained tool-calling model. The pre-training-only base
is under pretrain/, this run's harness-SFT-only intermediate under ablation/, and the
previous post-training run under r3/. Sections 1-7 describe the base model and its
architecture; section 8 describes post-training.
Read 8.4 before quoting a number. This model is the product of two stacked post-training passes (pretrain -> r3 -> r4), not one, because r4 loaded the repo root checkpoint that r3 had just published. The 45.4% headline is cumulative across both. Section 8.9 lists what is and is not statistically established.
1. Why this shape
A v5e-8 delivers 1576 TFLOPS bf16. The preceding flat 15-layer model (v4) reached only 12.7 % MFU — it was leaving ~87 % of the accelerator idle while underfitting its data. Two structural causes: narrow operands (768 / 640) against a single 128×128 MXU per chip, and every layer paying full 1024-token resolution.
So the lever is not a kernel trick, it is a shape decision: put the deep part of the network at low resolution.
tokens ──embed──► [ stage A: 2 blocks ]──┐ full res T = 1024, d = 768
local windowed attn │ dense ternary SwiGLU, f = 1536
(w = 256) + dense FFN │
▼
causal patchify (shift R−1) + RMSNorm + ternary w_down
│ R = 4 → T/4 = 256, d = 1024
[ stage B: 10 blocks ] │ GLOBAL attention (cheap at T/4)
ternary MoE, 8 experts │ top-2, expert_dim 640
▼
RMSNorm + ternary w_up + reshape
│
┌──────── U-Net skip ────────────┴──► x = x + up(down(x))
└──► [ stage C: 2 blocks ] ──► RMSNorm ──► tied head
Five design choices do the work:
- Causal patchify. A naive
T → (T/R, R·d)reshape makes patch j summarise tokensR·j … R·j+R−1; upsampling it back onto positionR·jwould hand 3 of every 4 positions a summary containing their own next-token label — a collapsed train loss and broken generation. The input is shifted left byR−1first, so patch j spans[R·j−R+1, R·j]and depends on nothing afterR·j. Verified by a per-position perturbation sweep: leak =0.0e+00. - Sparsity only where it is cheap. MoE dispatch is gather/scatter — memory-bound, and
relatively costly on a v5e's weak HBM/ICI. Running it at
T/4cuts that traffic 4×, while stages A/C use plain dense ternary SwiGLU, which is pure MXU work. This inverts the usual "MoE everywhere" layout. - Local attention at full res, global at the neck.
local_window_size=(255, 0)full-res; unrestricted causal attention over 256 latent positions. Global mixing is bought where it is nearly free. - Latent patch-lookahead MTP. From latent j (which summarises tokens ≤
R·j) predict tokenR·j+1, reusing the tied embedding. It runs atT/4, so multi-token-prediction regularisation costs ¼ of a normal extra head. - U-Net skip across the bottleneck so the crown still sees token-level detail.
Every matmul dimension is a multiple of 128 for the v5e MXU.
2. Measured results
Against the flat v4 predecessor, same 3-hour budget, same accelerator, same data corpus:
| v4 flat | Tern-1.5 | ||
|---|---|---|---|
| FLOP/token (fwd+bwd) | 849.6 M | 447.4 M | 1.90× fewer |
| tokens trained | 2.179 B | 5.839 B | 2.68× |
| throughput | 282 k tok/s | 739 k tok/s | 2.62× |
| MFU | 12.7 % | 17.9 % (21.0 % steady) | 1.41× |
| micro_seq that fit in HBM | 8 (16 OOM'd) | 16 | 2× |
| total wall clock | 2 h 38 m | 2 h 37 m | — |
| val loss (total) | 4.0989 | 3.4119 | −0.687 |
| val perplexity (total) | 60.27 | 30.32 | 0.50× |
| bits/token | 5.9135 | 4.9223 | −16.8 % |
| MC average (4 tasks) | 35.18 % | 36.19 % | +1.0 pt |
| tokens / total param | 9.6 | 24.7 | 2.57× |
Run ternova_0912_022826. Zero errors, zero NaN, no OOM, no compile fallbacks. Router imbalance
0.046 → 0.013 → 0.029 (collapse would be ≈0.375), from aux-loss-free bias balancing.
budget_stop fired at step 44 550 with 82 s to spare.
Per-task MC: arc_easy 0.390 · sciq 0.487 · hellaswag 0.333 · arc_challenge 0.238.
3. Honest limitations — read before citing
The headline "perplexity halved" overstates the language-modelling gain. val_loss is the
total main + 0.3 × aux, and the auxiliary objective changed between the two runs (v4
predicted token t+2 at full resolution; this model predicts token R·j+1 from a T/4 latent
patch). Decomposing with the final train-step aux losses:
| train main CE | train aux | implied val main CE | implied val ppl | |
|---|---|---|---|---|
| v4 flat | 2.950 | 4.721 | ≈ 2.683 | ≈ 14.6 |
| Tern-1.5 | 2.979 | 3.085 | ≈ 2.486 | ≈ 12.0 |
So the honest next-token improvement is ≈ 0.20 nats (~18 % lower perplexity), not 50 %. Most of the headline comes from the auxiliary head being better at its own (differently-scaled) task. These implied figures assume val aux ≈ train aux at the LR floor — indicative, not exact.
Two further confounds:
- Tokenizer. v4 loaded a pretrained 32 k BPE; in this run Kaggle silently dropped the
dataset_sourcesmount, so the in-kernel fallback trained a fresh 32 768-vocab gigatoken BPE in 25.2 s. Same size, same recipe, sameeos=256— but not the same merges. Perplexity across different tokenizers is not strictly comparable.tokenizer.jsonin this repo is that tokenizer. - 5.0 epochs over a 1.167 B-token corpus, vs 1.87 for v4. Encouragingly, final train main CE (2.979) is no better than v4's (2.950) despite 2.68× the tokens while val improved — the opposite of the memorisation signature.
Scale. At 236 M params and 5.8 B tokens, 4-option reasoning benchmarks barely move; arc_challenge at 23.8 % is at chance. Generated text is grammatical and on-topic but repetitive under greedy decoding, which is normal at this scale.
4. Files
| file | size | what it is |
|---|---|---|
params_fp32.npz |
876.6 MB | fp32 shadow weights (130 arrays, 236 307 968 elements). This is what you load to resume or fine-tune — QAT keeps a full-precision latent copy and applies ternary() in the forward pass. |
ternova_ternary.npz |
71.1 MB | the deployable quantised artifact: 59 tensors as int8 codes ∈ {−1,0,1} + per-row/per-expert scales (211 025 920 elements, 89.3 %), plus tok_emb and the 10 routers as int8 absmax (25 247 744 elements). |
tokenizer.json |
1.18 MB | gigatoken 32 768-vocab BPE, eos = 256. |
ternova_train.py |
72 KB | the complete single-file training kernel (data → tokenizer → train → eval → export). Also the reference implementation of the architecture. |
pretrain/params_fp32.npz |
876.6 MB | the pre-training-only fp32 shadow weights this model was fine-tuned from (130 arrays; no ldl_*). |
ablation/ckpt_phase1.npz |
876.6 MB | the phase-1 (harness-SFT-only) checkpoint, saved before self-improvement, so the phase-2 contribution can be ablated. |
posttrain.py |
90 KB | the post-training kernel: TTH-1 harness, masking, self-improvement loop, BFCL-style scorer. |
posttrain_log.jsonl, result.json |
— | the post-training run's event log and final metrics. |
config.json |
— | architecture + post-training hyperparameters, measured LDL gate statistics. |
result.json, train_log.jsonl |
— | the run's own event log and final metrics. |
Dequantising is code.astype(f32) * scale. For 2-D tensors scale == mean(|w_fp32|) exactly
(absmean rule); for 3-D expert tensors the scale is per-expert, shape (E, 1, 1).
Note that code * scale will not reproduce params_fp32.npz closely (cosine ≈ 0.72–0.87 on
ternary tensors). That is expected and not corruption: with straight-through-estimator QAT the
fp32 array is the optimiser's latent variable and the forward pass always uses its ternary
projection. The int8 tensors, by contrast, reconstruct to 1/256 (cosine > 0.9998).
5. Loading it
This is a custom JAX architecture -- it is not loadable via transformers.AutoModel, and
there is no separate modeling_*.py: the reference implementation is the training kernel itself.
import importlib.util, numpy as np, jax, jax.numpy as jnp
def load_mod(name, path): # ternova_train.py / posttrain.py
spec = importlib.util.spec_from_file_location(name, path)
m = importlib.util.module_from_spec(spec); spec.loader.exec_module(m); return m
T = load_mod("ternova", "ternova_train.py") # the architecture + generate()
P = load_mod("posttrain", "posttrain.py") # TTH-1 render_prompt/render_conv + scoring
cfg = T.CONFIG_HG # d_model 768, d_latent 1024, R=4, n_a/b/c 2/10/2
p = {k: jnp.asarray(v) for k, v in np.load("params_fp32.npz").items()} # fp32 shadow weights
biases = [jnp.zeros(cfg["n_experts"]) for _ in range(cfg["n_b"])] # aux-loss-free router bias
def logits_fn(tokens):
out, _, _ = T.forward_hg(p, tokens, cfg, biases, 1.25)
return out[0] @ jnp.transpose(T.ternary(p["head_w"])) if "head_w" in p else out[0]
print(T.generate(logits_fn, tok, ["The capital of France is"], n_new=32))
For tool calling, drive it through the TTH-1 harness the model was trained on -- build the
prompt with P.render_prompt(conv), generate to EOS, parse the <tool_response> JSON,
append the tool result, repeat. posttrain.py's rollout() and _norm_arg/_key are the
exact prompt builder and scorer used to produce every number in section 8, so a local harness and
the reported evaluation cannot drift apart.
Requires jax (TPU strongly recommended; CPU works but is slow) and gigatoken for the tokenizer.
Vocab is 32 768, pretrained context 1024, post-trained tool interactions 928 (768 prompt + 160 new).
6. Training recipe
| Data | FineWeb-Edu 45 % · Cosmopedia-v2 30 % · finemath-3plus 15 % · Python (github-code-clean) 10 % — 1.167 B tokens packed, 5.0 epochs |
| Optimiser | Muon (Nesterov 0.95, 5-step Newton–Schulz, device-sharded) on all 2-D/3-D matrices; AdamW (β 0.9/0.95) on embeddings, routers and norms |
| LR | 3e-3, cosine to 10 %, 5 % warmup, grad clip 1.0 |
| Quantisation | BitNet b1.58 absmean ternary with STE shadow weights; activations bf16 |
| Balancing | aux-loss-free router bias, δ = 0.02, clip ±0.25 |
| Batch | 131 072 tokens/step (micro_seq 16 × 8 devices × 1024) |
| Hardware | Kaggle TPU v5e-8, jax 0.10.2, 354.7 s to compile |
7. Reproducing
ternova_train.py is the entire pipeline. Run it on a TPU v5e-8 with TERNOVA_MODE=train; it
downloads and tokenises the data itself, calibrates the micro-batch ladder against live HBM,
solves its own step budget from measured step time, and stops on the clock. TERNOVA_MODE=tiny
runs a CPU smoke test in ~25 s.
The FLOP/param model was validated two independent ways: an offline calculator that first
reproduces the flat model's 850 MFLOP/token exactly, and the kernel's own flops_per_token_hg.
They agree to < 0.01 %.
8. Post-training: tool calling and self-correction
The base model above is a raw LM. It was post-trained on the same TPU v5e-8 into a tool-calling model that can drive an agent harness, in a single 2.52 h session.
Sub-sections 8.1-8.9 below; the base-model sections 1-7 are unchanged. Read 8.4 (lineage) and 8.9 (limitations) before quoting any number here.
8.1 The harness format (TTH-1)
All seven tool corpora are normalised into one canonical format, so the model learns a single contract rather than seven dialects:
<tools>[{"name":...,"description":...,"parameters":{...}}]</tools>
<user>...</user>
<assistant>
<tool_response>{"name":"f","arguments":{...}}</tool_response>
</assistant>
<tool_response>{"name":"f","content":...}</tool_response>
<assistant>final answer</assistant>
A turn ends with the real EOS token, so a harness loop is just generate to EOS -> parse the call -> append the tool result -> repeat. Only <assistant> spans (and the EOS that closes them) carry loss; every prompt, schema and tool response is masked to zero.
8.2 Data
- 635,334 packed rows = 651 M tokens
- 7 tool corpora + 3 retention corpora. Two disjoint held-out sets are carved out before any training: 994 probe items, which are never trained on in any phase and are the only thing every score below is measured against, and 5,999 improve items, withheld from phase 1 and used only in phase 2 -- the model rolls out on them, and only its own verified output and the machine-written critique of its failures become training documents. The gold calls themselves are never a phase-2 target.
- The probe is stratified across all 7 tool dialects (142 items each), as is the improve set. An earlier run drew the probe from one corpus and the improve set from another, which made its headline number dialect-specific; see 8.4.
- Packing is document-aligned: every row contains whole interactions padded with unsupervised EOS. No conversation is ever split across a row boundary, which is what would otherwise strand a tool schema on a different row from the turn that uses it.
| source | rows | weight | kind | supervised frac |
|---|---|---|---|---|
| ret_web | 182,492 | 0.2 | ret | 1.0 |
| hermes_reason | 42,382 | 0.14 | tool | 0.3582 |
| apigen | 66,627 | 0.12 | tool | 0.1972 |
| xlam | 37,482 | 0.11 | tool | 0.1886 |
| glaive_v2 | 61,913 | 0.1 | tool | 0.5241 |
| ret_synth | 78,978 | 0.08 | ret | 1.0 |
| ret_code | 94,927 | 0.07 | ret | 1.0 |
| dolci | 54,923 | 0.07 | tool | 0.243 |
| hermes_fc | 8,623 | 0.06 | tool | 0.326 |
| toolace | 6,987 | 0.05 | tool | 0.2013 |
8.3 Results
BFCL-style scoring on the held-out probe. match = right tool and semantically equal arguments (5 == "5"); exact = byte-identical argument JSON.
| stage | well-formed | closed tag | name acc | match | exact |
|---|---|---|---|---|---|
| base (see 8.4 -- the previous post-trained model) | 91.2% | 92.0% | 41.6% | 28.9% | 28.5% |
| after phase 1 (harness SFT) | 88.4% | 89.2% | 57.4% | 40.7% | 39.5% |
| after phase 2 (self-improvement) | 93.0% | 93.3% | 59.5% | 45.4% | 44.9% |
Overall, this run took match 28.9% -> 45.4% (+0.165, z = 7.73, 95% CI [+0.123, +0.207]).
Per-dialect breakdown. This is the table that matters: an aggregate over a single corpus hides exactly the failure mode this design is meant to fix.
| dialect | n | base match | final match | delta | z |
|---|---|---|---|---|---|
| hermes_reason | 142 | 0.310 | 0.732 | +0.423 | +7.86 |
| xlam | 142 | 0.225 | 0.437 | +0.211 | +3.88 |
| apigen | 142 | 0.282 | 0.486 | +0.204 | +3.62 |
| toolace | 142 | 0.091 | 0.232 | +0.141 | +3.28 |
| hermes_fc | 142 | 0.035 | 0.120 | +0.084 | +2.70 |
| dolci | 142 | 0.176 | 0.239 | +0.063 | +1.32 |
| glaive_v2 | 142 | 0.901 | 0.930 | +0.028 | +0.86 |
Every dialect improved, and the gains are largest where the base was weakest (hermes_reason, xlam, apigen, toolace) and smallest where it was already saturated (glaive_v2). Isolating phase 2's own contribution (phase 1 -> final) shows it is concentrated in one dialect:
| dialect | n | phase 1 match | final match | delta | z |
|---|---|---|---|---|---|
| hermes_reason | 142 | 0.444 | 0.732 | +0.289 | +5.17 |
| toolace | 142 | 0.148 | 0.232 | +0.084 | +1.83 |
| hermes_fc | 142 | 0.106 | 0.120 | +0.014 | +0.38 |
| glaive_v2 | 142 | 0.922 | 0.930 | +0.007 | +0.23 |
| xlam | 142 | 0.458 | 0.437 | -0.021 | -0.36 |
| apigen | 142 | 0.507 | 0.486 | -0.021 | -0.36 |
| dolci | 142 | 0.268 | 0.239 | -0.028 | -0.55 |
Bold z clears 1.96. Only hermes_reason does; the other six per-dialect deltas are individually indistinguishable from noise at n = 142, even though the pooled phase-2 effect (8.9) does clear it.
Held-out loss on supervised tokens only: tool 0.7956, retention 3.273. Per source, phase 1 -> shipped:
| source | phase 1 | shipped | delta |
|---|---|---|---|
| apigen | 0.0976 | 0.1104 | +0.0128 |
| xlam | 0.1436 | 0.1684 | +0.0248 |
| toolace | 0.5067 | 0.5078 | +0.0011 |
| dolci | 0.4741 | 0.5081 | +0.0340 |
| hermes_fc | 1.1406 | 1.078 | -0.0626 |
| glaive_v2 | 1.4069 | 1.4975 | +0.0906 |
| ret_code | 1.5698 | 1.5686 | -0.0012 |
| hermes_reason | 1.6412 | 1.6989 | +0.0577 |
| ret_synth | 3.5074 | 3.5049 | -0.0025 |
| ret_web | 4.7325 | 4.7455 | +0.0130 |
ret_web stays near 4.7455 nats: the retention mixture is not teaching open web text, it is only slowing its decay.
General ability, same four multiple-choice tasks as the base model (0.3619 before any post-training): arc_challenge 0.194, arc_easy 0.3833, hellaswag 0.33, sciq 0.49 -> average 0.3493.
8.4 Lineage -- what was actually trained on what
Two post-training runs have been executed, and they are not independent:
| run | started from | probe | match | weights |
|---|---|---|---|---|
| pretrain | scratch, 5.84 B tokens | n/a (0.0% tool-call) | 0.000 | pretrain/params_fp32.npz |
r3 ternova_0912_233104 |
pretrain | 1000, apigen only | 0.368 | r3/params_fp32.npz |
r4 ternova_0913_035918 (this card) |
r3's output | 1000, 142 per dialect x 7 | 0.454 | params_fp32.npz |
r4 was meant to re-run the same recipe from the pre-training base with a corrected, dialect-balanced probe. It did not: base_file defaulted to the repo root, and r3's weights had been published to the root an hour before r4 launched. So r4 is a second post-training pass stacked on r3, and the base row in 8.3 is r3's shipped model rather than the raw pre-trained LM. r4 also re-initialised the LDL gate (w=0, b=-3) instead of inheriting r3's trained one.
What this means for the numbers:
- Valid: r4's model is better than r3's on the balanced probe (28.9% -> 45.4%), and phase 1 -> phase 2 within r4 is a clean comparison, since both use the same weights and the same probe.
- Not valid: any claim of the form "0% -> 45.4% from pre-training in one run". The cumulative path is pretrain -> r3 -> r4, across two sessions totalling 4.76 h.
- Unmeasured: the pre-trained base on the balanced probe. It scored 0.000 with 0.0% well-formed output on r3's apigen probe, so it is ~0 on any tool probe, but that was never re-measured on the stratified one.
- r3's 0.368 and r4's 0.454 are not comparable to each other -- different probes.
The cause is fixed: PT_BASE_FILE now defaults to pretrain/params_fp32.npz, so cumulative post-training has to be opted into explicitly.
8.5 Phase 1 -- masked harness SFT
- 13,442 steps, 1762 M tokens, 72.8 min at lr 0.0005
- Epoch-capped, not just clock-capped: replaying a few hundred million SFT tokens a dozen times overfits and destroys the base model's general ability, so the step budget is
min(time, epochs * tool_rows / rows_per_step).
8.6 Phase 2 -- learning from its own mistakes
For each round the model is rolled out on held-out prompts it has never seen, its generations are verified against the gold call, and the failures are turned into new training documents:
- generate 160 tokens from the prompt with the current weights;
- parse the turn, compare name + normalised arguments to gold;
- for every failure, emit a document containing the prompt, the model's own wrong call and a machine-written critique of what was wrong -- all masked to zero loss -- followed by the correct call, which is the only supervised span;
- train on a mixture of those correction documents and the phase-1 data, so the model sees its own error distribution rather than a generic one.
The wrong call is context, never a target. That is what makes it self-correction rather than self-distillation of mistakes.
| round | rollouts | correct | wrong | malformed | correction docs | packed rows | steps |
|---|---|---|---|---|---|---|---|
| 0 | 5999 | 2249 | 3310 | 897 | 9230 | 6998 | 3998 |
| 1 | 5999 | 2888 | 2991 | 657 | 8594 | 6415 | 2421 |
Rollout correctness rose 37.5% -> 48.1% and malformed output fell 14.9% -> 10.9% between rounds. Phase 2 total: 120.4 min at lr 0.00025.
Note the rollouts got harder, not easier, versus the previous run: with the improve set now spread over seven dialects instead of one, round 0 started at 37.5% correct. That is the intended effect -- more distinct failure modes to learn from -- and it is why phase 2 trained 6,419 steps here against 2,622 before.
8.7 The Latent Deliberation Loop (the novel part)
Post-training adds exactly one architectural component, active only in the post-trained checkpoint: after the hourglass trunk runs, it is re-run with its own weights (tied recurrence, Universal-Transformer style) and blended back through a per-position gate the model learns:
g = sigmoid(rms(z; ldl_n) . ldl_w + ldl_b) # [B, T/R, 1] per latent position
z = z + g * (trunk(z) - z)
g is a learned decision of how much more to think at this latent position, which turns a fixed-depth stack into an adaptive-depth one -- the property multi-step tool planning needs, since hi and book a flight warrant very different amounts of deliberation.
- 2,049 parameters (0.0009 % of the model):
ldl_nandldl_wof size d_latent, plus one scalarldl_b. - cost: 447.4 M -> 564.5 M FLOP/token (+26.2 %), counted by the kernel's own
flops_per_token, not estimated. The loop runs atT/Rresolution, so it costs a quarter of what an extra full-resolution pass would. - initialisation:
ldl_w = 0,ldl_b = -3, so at step 0 the loop is a ~4.7 % perturbation of the checkpoint rather than a re-randomisation (verified inldltest.py, 43 assertions). Nothing is lost if the loop never learns to open.
Measured gate on held-out data at the end of training. This run re-initialised the gate from scratch and it re-learned to open further than the previous run did:
| r3 (previous run) | r4 (this run) | |
|---|---|---|
ldl_bias |
-2.9024 | -2.9076 |
ldl_w_absmean |
0.02098 | 0.02598 |
gate_mean |
0.8068 | 0.9259 |
gate_p90 |
0.9424 | 0.9899 |
gate_frac_open |
0.948 | 0.9875 |
gate_at_zero_context |
0.052 | 0.0518 |
gate_frac_open = 98.8% while gate_at_zero_context = 0.0518: the loop opens on real context and stays shut on empty context, so the gate is input-dependent rather than uniformly on. That the bias moved only 0.092 from init while the gate opened this wide means ldl_w is doing the work, not the bias.
8.8 Cost
- whole post-training session: 2.52 h on one Kaggle TPU v5e-8 (limit 3 h)
- 527,644 tok/s at 0.2484 s/step · 564.5 MFLOP/token · 131,072 tokens/step = B 128 rows x 1024 (micro_seq 16 x 8 devices)
- shipped weights: 70.1 MB ternary (0.297 bytes/param), 876.7 MB fp32 shadow
- XLA compile 1084.9 s + calibration 2.6 s. The calibration figure is worth noting: the previous run burned 962 s here because
measurefed the step's own outputs into the next iteration, making iteration 1 a second XLA program. Passing identical argument objects every iteration removed it (iter_s = [0.319, 0.249, 0.248, ...], no spike).
8.9 Honest limitations of the post-training numbers
1. The absolute level is modest, and the bottleneck is tool selection, not format. Well-formed output is 93.0% but name accuracy is 59.5% and match 45.4%. The harness contract is essentially solved; picking the right function from a schema of similar API names is not. At 236 M params that is the expected failure mode.
2. Phase 2's effect on match now clears significance, but only just. Unpaired two-proportion test at n = 994 (per-item outcomes were not logged, so a paired McNemar test was impossible; unpaired is the conservative choice):
| metric | phase 1 | shipped | delta | 95% CI | z | verdict |
|---|---|---|---|---|---|---|
| well-formed | 88.4% | 93.0% | +0.045 | [+0.020, +0.071] | 3.49 | decisive |
| match | 40.7% | 45.4% | +0.046 | [+0.003, +0.090] | 2.09 | significant |
z = 2.09 is past 1.96 but not comfortably -- p ~ 0.037. The same comparison in the previous run gave z = 1.36 with a CI spanning zero. Two changes explain the difference: the balanced probe measures a real mixture of dialects instead of one the model had already been tuned on, and phase 2 trained 6,419 steps here versus 2,622 before. Treat "self-improvement helps" as supported, not settled.
Per dialect the phase-2 gain is concentrated in hermes_reason (+0.289, z = 5.17); the other six are individually noise at n = 142. So the pooled significance is driven largely by one corpus, and it is not established that the mechanism generalises across dialects.
3. The lineage confound (8.4) is the biggest caveat. This is a second post-training pass on an already-post-trained model, so the phase-1 and phase-2 deltas measure additional training on top of a model that had already seen harness data. A clean single-pass run from pretrain/params_fp32.npz on this balanced probe has not been executed, and until it is, the honest headline for this repo is "two stacked post-training passes reached 45.4% match", not "post-training reaches 45.4%".
4. General ability regressed and has not recovered. The four multiple-choice tasks averaged 0.3619 before any post-training, 0.3435 after r3 and 0.3493 after r4 (-0.0126 versus pre-training). Per task: arc_challenge 0.194, arc_easy 0.3833, hellaswag 0.33, sciq 0.49. The retention mixture (35 % of rows by weight) slows the decline but does not stop it, and ret_web val loss stays at 4.7455. There is a real alignment tax here.
5. The LDL is still not ablated. 8.7 shows the gate is engaged and re-learns consistently across two independent initialisations, which is mildly reassuring, but nothing isolates its contribution to the score. The gain must be attributed to the whole recipe. An LDL-off rerun is the obvious next experiment and is cheap -- same budget, one config flag.
6. The scorer penalises a data artifact. Some gold turns contain the same function name twice; emitting it once scores as a miss even when the call is correct. Two of the eight generations logged in the previous run's result.json failed only for this reason. True match is therefore slightly higher than reported. Not corrected, to keep the scorer identical across stages and runs.
7. Multi-call turns remain the dominant structural failure. Where gold requires two chained calls (get_id then search_torrents), the model emits one. Neither run addressed this; it needs multi-call emphasis in the training mixture, which is a data change rather than a plumbing fix.
8. Single seed per configuration, no repeats. No error bars across seeds, and the base model itself was trained once. Everything above is one sample from a distribution that has not been characterised.
Apache-2.0.
- Downloads last month
- 624