Davd-b01/qwen3.8-flash-next-40b-prune-research

Surgical Pruning of Qwen3.8-Flash-Next (512→128 Experts) and Full Post-Training Pipeline

One upstream MoE, six generations of masks, one bit-exact surgery, a rebuilt factual memory, and a three-stage post-training (SFT → SimPO → RLVR) — with every error I made and every idea I falsified along the way.

This repository is the complete technical record of the project: the activation-instrumentation artifacts, the mask-generation methodology, the surgery code, the PLE (n-gram memory) rebuild, the trainers, and the full lab notebook. The champion weights live in sibling repos (model map).

TL;DR

What I set out to do: take Qwen3.8-Flash-Next — an 180B-class MoE (130B of core network + 51B of n-gram memory table + 4B MTP) — and cut it by 75% of its experts (512 → 128) while conserving the maximum knowledge possible: protecting the model's grammar (tags, syntax, formats) perfectly, and paying for the size reduction with a deliberate sacrifice of the long tail — rare languages, niche domains, a few entire areas of expertise — instead of letting damage fall at random. Then post-train the result into a competent, honest reasoning model. Budget: about $300 of rented GPU time.

What I actually got: a 40B model (S6-K128-P4) with one genuinely excellent feature — the writing. The reasoning prose is structured, self-correcting, formally organized; it writes valid code, dispatches tools canonically, and argues back when you assert falsehoods. Everything else is weak, and I say so plainly: the benchmark scores are bad (GPQA ≈ chance, MATH-500 17.8%, IFEval prompt-strict 37%), factual recall took the hardest hit, and the blame splits — an extremely aggressive prune, my own heal/finetune errors (a factual memory that silently loaded random noise during the mid-lineage training stages, a SimPO dataset that structurally could not learn, a first mask that was statistically random), possibly the Thinking Cap method itself — the thinking format every trace passes through (this project's biggest achievement) is a named, un-ablated suspect (§8.3) — and measurement artifacts (format parsers, budget-exhaustion empties). The control experiments that would apportion the blame precisely — the unpruned upstream, the full dense-27B comparison — were never finished: the money ran out at ~$300. Every error is documented in §5; good and bad traces sit side by side in §8.2.

What this proves anyway: losing 75% of the expert mass does not produce a broken model. The damage is selective and mappable — language, logic, code structure and tool use survive; trivia and arithmetic robustness don't. So massive expert pruning is a viable technique with a known cost schedule. And there is clear headroom for a v2: the K=192 mask keeps +14 points of routing mass, the factual-table pipeline is now fixed and measured, and every failure mode has a named fix.

Why this architecture was the perfect guinea pig: only ~2% of the experts (10 of 512) fire per token per layer; the routers that make every decision are just 15.7M parameters (0.039% of the network); and about a third of the whole model was a lookup table — stored memory, not compute. In fact the 75% cut saves memory, not FLOPs: evaluating 10 experts out of 128 costs the same per token as 10 out of 512.

---

☕ Did the artifacts, the methods, or the honest failure list earn a coffee? → Buy me a Ko-fi — every ☕ = more pod hours, and the v2 (K=192 + clean heal) is already designed and waiting.

🚀 Entering the LLM fine-tuning / inference world yourself? Start with my RunPod referral → runpod.io?ref=ssakdva8 — you pay exactly the same, the project gets a boost, and §9 (Toolbox) gives you everything to replicate this pipeline step by step.

💸 For scale: the whole saga — 9 days, ~730 GB in total, 17 repos consolidated into this one — $300.

---

Model details

Developed by	Oscar (Davd-b01 on Hugging Face)
Model type	Hybrid MoE ("qwen4_exp"): 36 Gated DeltaNet + 12 sparse-attention layers, 128 routed experts (top-10), shared expert, Hyper-Connections, external n-gram factual memory (PLE), vision tower + MTP head preserved
Lineage code	S6-K128-P4 — see the naming convention
Finetuned from	Davd-b01/qwen-3.8-next-40b-exp-v6rank-K128-bf16 (S6-K128-P0), itself pruned bit-exactly from Qwen/Qwen3.8-Flash-Next-FP8
Parameters	40.22 B total; ≈ 6.6 B active/token; PLE table 5.12 B stored (zero-FLOP lookups)
Language(s)	English, Chinese (primary); Spanish and other tails degraded by design — the multilingual stratum was the deliberate sacrifice of the expert budget
License	Apache-2.0 (inherits the upstream Qwen license family); code in this repo Apache-2.0
Context	trained/served at 16k–64k tokens (QSA sparse indexer up to 262k by design)
Precision	BF16 only — quantized serving is a documented negative result (§8.3)
Requires	transformers >= 5.17 (native qwen4_exp) or SGLang pr36497; BF16 weights, ~81 GB
Status	Research model — a laboratory artifact, not a production assistant; see Intended use

Intended use, out-of-scope uses, and limitations

This is a research model. It exists to be studied, not to be used. It is a laboratory artifact for investigating what survives a 75% expert prune, how a factual-memory table changes training dynamics, what breaks in a post-training cascade over a pruned MoE, and where exactly the damage lands (§8.3). Nobody should deploy it as a product, an assistant, or any system with real users — it is below the quality bar for that, by its own accounting (§8.3). What it is good for:

• Pruning research: reproducing or extending the mask generations (§2.3), the bit-exact surgery, the C1–C6 verification framework — or testing whether the K=192 headroom converts into quality.
• Post-training research: the SimPO iso-density finding (39.3% → 80.7% from dataset curation alone), the verified-reward loop, the RFT anchor cascade — all runnable from the adapters and datasets published here.
• Architecture study of qwen4_exp: one of the few places where this hybrid architecture (GDN/QSA + MoE + external n-gram memory) is instrumented, served and documented end to end.
• Reasoning-behavior observation: the five thinking tiers (off/low/mid/high/xhigh, selected via the system prompt) modulate deliberation depth measurably, and the model's strong suit — structured, self-correcting reasoning prose — makes its traces unusually readable for analysis.

Out of scope.

• Any production or end-user deployment — this is a research artifact, full stop.
• Encyclopedic recall without retrieval: GPQA Diamond ≈ chance — this model trades factual trivia for size; use it with RAG/tools for facts.
• Unassisted exact arithmetic: multi-digit multiplication without a scratchpad is a limit of the whole architecture class (TC⁰), aggravated by the prune; delegate to a code interpreter.
• Quantized serving: BF16 only (see §8.3 / docs/VULKAN-APEX-TESTING.md).
• High-stakes deployment without verification: the model can produce verification-shaped text that is itself wrong (§8.3 #3); keep graders/tools in the loop.

Limitations. The honest failure list and its forensics live in §8.3; the benchmark scores above carry format-parser caveats documented in §8.2.

Get started

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "Davd-b01/qwen3.8-flash-next-40b-prune-research"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="auto")

messages = [
    {"role": "system", "content":
     "Reasoning effort is set to high. Prove claims rigorously, verify intermediate "
     "derivations, confirm arithmetic carries step by step, and test edge cases before "
     "finalizing the result. Prioritize quiet correctness over rhetorical emphasis."},
    {"role": "user", "content": "Prove that sqrt(2) is irrational."},
]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=2048, temperature=0.7, top_p=0.95,
                     repetition_penalty=1.05, do_sample=True)
print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))

Notes: (1) the champion loads natively — no patch script (the factual table is embedded in the fused parameter); the foundations (S6-K128-P0) require the cargar_tabla_ple patch (§11). (2) For serving, SGLang gives the best throughput on this architecture — recipe in §11.

---

Table of contents

• TL;DR
• The story, in plain words
• How everything is named (the convention)
• The nine-day campaign (process timeline)

1. The target: Qwen4Exp anatomy
2. Pruning methodology
   • 2.1 Calibration corpora
   • 2.2 Activation instrumentation — the signal artifacts
   • 2.3 Mask generations S1 → S6
   • 2.4 The surgery (bit-exact expert slicing)
   • 2.5 The router-truncation problem
3. The PLE n-gram memory
4. Post-training
   • 4.1 Data factory (Best-of-3, TCS tiers)
   • 4.2 SFT — asymmetric DoRA
   • 4.3 SimPO — the iso-density fix
   • 4.4 RLVR — verified rewards, offline
5. The error ledger
6. Rejected ideas, with falsifications
7. Academic foundations
8. The 75% thesis: competence survives
9. Toolbox: the reusable pipeline, script by script
10. Artifact map
11. Reproduction

• Training & compute accounting (environmental impact)
• Citation
• Support this work
• Contact
• Appendix: the math that runs this model

---

The story, in plain words

(No jargon. If you read only this section, you will understand what happened and why it matters.)

Start with the shape of the problem. Imagine a sheet of paper, handwritten edge to edge — every square centimeter used: code, math proofs, a dozen languages, legal doctrine, tool syntax, conversation, science. That is what this model is: ~180 billion parameters of densely-written knowledge, kept in 512 interchangeable "experts" per layer, of which only a team of 10 reads each word. The challenge I set myself: shrink that sheet to a quarter of its size — and keep it legible. Most of the content had to survive; the grammar — the tags, the syntax, the formats — had to survive perfectly; and some regions, knowingly, would be sacrificed to fit the budget: the rarely-visited margins, the long-tail languages, a few entire areas of expertise.

The pruning form of that bargain: erase 384 of every 512 experts, choosing them so every region of the sheet keeps enough of its text, and retrain just enough for the surviving lines to cover what was removed. And one tempting shortcut was off the table from the start. The literature says MoE models carry many duplicate experts — redundant copies you can merge away, free space. I measured it: barely 0.12% of expert pairs here are near-twins. This sheet was never written twice. Every line removed was the only copy. That single measurement turned the project from an organizational problem into a genuine sacrifice — and dictated the whole design: conserve the most-used knowledge perfectly, protect the grammar at all costs, and choose the sacrifices deliberately instead of letting them happen at random.

How do you decide which lines to erase? Not by their ink alone — and here the record owes a plain confession: my first attempts failed because I wanted to be cheap. Rather than rent GPU time to watch the real model work, I took the tokenizer's embeddings and half-passed them through the layers to see which experts "lit up" — a proxy of a proxy, tried to save a few dollars of teacher forwards. It was a complete error: the first shipped mask agreed with real routing on barely 30% of experts — worse than a coin flip, because it carried the false confidence of looking principled. What works is watching the sheet being read: run the original model over a deliberately balanced buffet of tasks (code, math, science, 17 languages, tool calls) and record, token by token, which experts fire and how much they matter. Then keep the 128 that carry the load — making sure no region (no task type) loses all of its text. That measurement, not the cutting, is the heart of this project, and the measurement ledgers are published here.

The cutting itself is done so that nothing is recomputed: the 128 survivors keep their exact original weights, bit for bit, and everything that is not an expert — the vision system, the draft head used for speculative decoding, the embeddings — passes through untouched.

Then the sheet needs physical therapy, for two reasons. First, the routing brain still expects 512 experts, so it must relearn — and it cannot relearn alone: experts that are never chosen receive zero learning signal (I tried routers-only training; it failed cleanly). So the routers train together with a light adapter layer across the network. Second, the model's "factual memory chip" — a 10 GB lookup table that injects facts — had to be rebuilt from 11% useful to 90% useful, because the original had been hashed with the wrong keys.

Along the way came the two best bugs of the project, both instructive:

• The memory chip was never plugged in. For days, every training run and every benchmark loaded a random-filled table, because the file stored the memory as 128 pieces while the loader expected one piece — and neither side complained. Different machines behaved differently because each one received a different random table. Found by checking a single number (row magnitudes) against the spec, for $0 of GPU time.
• The validation that validated itself. An earlier rebuild of the memory chip was certified "verified" by a test that used sentences from its own textbook: 100% on the test, 11% in production. Test data must be disjoint by construction, not by intention.

What survived: reasoning, code structure, tool use, formal logic, and the refusal to flatter the user — all intact, all improved by a three-stage finishing school (first teach format, then teach preference, then drill arithmetic with a strict automated grader). What got hurt: encyclopedic recall (the model kept its methods and lost much of its trivia) and mental multiplication (a limit of this entire model family, aggravated by the prune, fixed on the training bench by teaching a show-your-work habit). The final model is coherent, writes valid code, uses tools, and every one of its weak circuits is documented — because knowing exactly what was lost is what makes the 75% cut defensible.

---

How everything is named (the convention)

The historical names collided: "v6" meant a mask generation and a training run and a repository; "v4" meant both an old mask era and an old trained model. The convention below separates the axes so every artifact has exactly one code.

Rule: every artifact is <AXIS><generation>-<qualifiers>. Five axes:

Axis	Code	Values	What it counts
Surgery — which mask/method cut the model	S	S1…S6	the pruning lineage
Post-training — what schooling was applied after	P	P0 (none) … P4 (champion)	the training lineage
Factual table — the PLE memory-chip rebuild	PLE	PLE1…PLE4	the memory lineage
Calibration corpus — the measurement buffet	CAL	CAL1…CAL3	the instrument lineage
Size & precision	—	K128, K192 · bf16, gguf	deployment qualifiers

Surgery generations (the mask lineage — §2.3):

Code	Legacy name	Method, in plain words	Verdict
S1	v1 split-repack	cut every expert in half, prune, sew pairs back — origami	abandoned: pieces fused on their own
S2	v2	judge by weight statistics — hiring from résumés	disagreed with reality (recall 0.23)
S3	v3	proxy signals + knapsack	retention flat at ~25% everywhere
S3d	v3-dedup	S3 + forced diversity (MMR)	≈ random (29.4% mass kept) — the cautionary model
S4	v4-m100k	watch the original model work, keep the top-128 by traffic	first real mask
S5	v5	contribution criteria on a balanced buffet (freq / rank / mesa)	criteria defined
S6	v6 keep-rank	rank by keep-domain contribution (+v6disc anti-leak variant)	shipped — cut both K=128 and K=192

Post-training generations (§4):

Code	Legacy name	What was added	Outcome
P0	—	pruned, untrained	the patient on the table (competent, mis-calibrated)
P1	tcs-v4-bf16	SFT + SimPO, over a random memory chip (unbeknownst)	format OK, preference marginal, loops
P2	RUN-V6 / v6.0	+ verified-correct anchors (RFT), TF-IDF loss	SFT healthy; SimPO strangled by bad pairs (39.3%)
P3	v6.1	iso-density pair curation, warm-start	preference accuracy 80.7%
P4	RLVR-50	verified-reward drilling over a curated offline dataset	champion — fused BF16

Factual table generations (§3):

Code	Legacy	Plain words	Disjoint hit-rate
PLE1	raw512	upstream rows remapped with the (correct) official hash	10.74%
PLE2	v6	rebuild, keep-domain first	24.40%
PLE3	v6.2	vectorized rebuild from the BF16 source	89.55%
PLE4	v6.3	PLE3 + backfill of empty rows, embedded in the champion	89.55%, no empty rows

Calibration corpora (§2.1): CAL1 = m100k (the 17-language traffic census), CAL2/CAL3 = GATE-DS v1/v2 (the balanced contribution census, with and without a reject-domain).

Full model codes — composed as S#-K<size>-P# — now mapped to paths inside this single repository (the 17 legacy repositories were absorbed here and deleted after hash-verification; lineage/MANIFEST.md records the ones deleted without copy):

Code	Ubicación en este repo
S6-K128-P4 (champion)	raíz
S6-K128-P0	foundations/S6-K128-P0/
S6-K192-P0	foundations/S6-K192-P0/
S3d-K128-P1	REMOVED — documented in lineage/MANIFEST.md
S3d-K128-P0	REMOVED — documented in lineage/MANIFEST.md
S1-K128-P0	REMOVED — documented in lineage/MANIFEST.md
P2+P3+P4 adapter checkpoints	adapters/

One deliberate exception: published filenames keep their legacy labels (mascara_v4_m100k_K128_lam005.json, dora_adapters_rlvr_final.pt, …) so that every number in this README remains traceable to a literal file. The S/P/PLE/CAL codes are the lineage layer on top, not a rename of the artifacts.

---

The nine-day campaign (process timeline)

Everything above happened in nine days of pod time. The record, so the process is legible as a sequence of decisions rather than a monolith:

Date	Milestone
Sep 03	S1 prune (split-repack, legacy v1) deployed on A100: 74 GB, 1,167 tensors, 20.25 s load. Forensic audit: comprehension intact, ~38% expert starvation diagnosed, PLE ratio ρ=0.658 flagged (target 0.25)
Sep 08	Local metadata audit — stale hash multipliers discovered (table declared noise); "paper-model" extractor condenses the 74.9 GB checkpoint to a 15 MB statistics JSON; router max/min norm ratios 1.6–2.3× mapped
Sep 09	Masks S3 (embed-space knapsack) + S3d (MMR dedup, λ*=1.0) + verification gates V1–V5
Sep 10	v3 surgery verified C1–C6 on a 4090 pod; N3 routing profiling of the pruned model on RTX PRO 6000 (config fix: ngram_vocab_size_base 20M → 2M so the model fits 81 GB); heal blueprint; L40S Best-of-3 factory goes live (4 pods)
Sep 11	Elite-era production run completes (SFT 200 + SimPO 60, 9.6 h billed, $30.90) → tcs-v4-bf16 published; pad-contamination autopsy (81.4% of the LIMA SFT set); FP8 export disaster (unscaled GEMM → gibberish; repo deleted same day)
Sep 12	v5-gold SFT 120 steps converges (loss 1.94 → 0.66, PPL 1.93, 109/128 experts alive); SimPO runs; clinical sweep on Blackwell; gold-SOTA dataset compiled (1,202); RLVR literature sprint (§79–80)
Sep 13	The analysis day: full pipeline audit flags mask v3 as ≈ random; Pod A loads the 512E teacher on H200 → H1/H2 verified, mask S4 (legacy v4-m100k) derived from real activations; TESIS-PODA-V4 (four broken instruments audited); GATE-DS contribution signals + S5/S6 criteria (freq/rank/mesa, keep/reject delimitation) + MEGA-SWEEP (204 combinations); NEFTune + TF-IDF researched
Sep 14	Dual surgery published (K=128 + K=192, keep-rank masks, on a 4090 — incl. an HF commit-rate-limit crash and its resume fix); table PLE4 built and swapped into the foundation; k₂ falsified live ("A3"); RLVR-50 dataset curated; pre-run code audit (9 patch categories)
Sep 15	RUN-V6: SFT healthy, SimPO strangled by the dataset (39.3%); same night — the PLE-never-loaded root cause, found for $0 of GPU (meta-device key-map audit)
Sep 16	P3 tune (legacy v6.1): mini-SFT 15 + SimPO 25 → preference accuracy 80.7%; RLVR worker prepared — incl. an online GRPO-lite mode (3 dry-run bugs caught cheap)
Sep 17	P4 = RLVR-50 (64.8%, 12.8 min on H100 NVL) — the champion; BF16 fusion published (17 shards, C1–C6 pass); frontier eval on Blackwell (GPQA/IFEval/MATH/LCB, 4,072 traces archived)
Sep 18	Control run of the official dense 27B (aborted at ~40% after the harness-induced pathologies showed up there too); APEX GGUF built → tested on Vulkan → documented as negative result → deleted

---

1. The target: Qwen4Exp anatomy

Qwen/Qwen3.8-Flash-Next is a hybrid MoE ("qwen4_exp" architecture, transformers ≥ 5.17) with a fused external factual memory. The pruned derivative I ship is a 40.22 B-parameter model. Of those, ~2.36 B of routed parameters fire per token (top-10 of 128 experts, 48 layers) on top of the always-on dense subsystems — 2.09 B hybrid attention, 1.27 B embeddings + lm_head, 641 M hyper-connections, 236 M shared expert, 16 M routers, ~33 M PLE projections — for ≈ 6.6 B active parameters per token. The 5.15 B PLE table is stored memory, not compute: 16 row-gathers of 160 dims per token, zero FLOPs:

Subsystem	Params	Share	Notes
MoE routed experts (48 layers × 128 experts)	30.199 B	75.1%	pruned from 48 × 512
PLE n-gram table (layer 2)	5.153 B	12.8%	32,002,176 × 160 rows (5.12 B raw table + 33 M conv/gate projections)
Hybrid attention (36 GDN + 12 QSA layers)	2.087 B	5.2%	GDN recurrent state S_t = S_{t-1}⊙β_t + q_t k_tᵀ; QSA = sparse indexer (4× compression, top-2048 budget)
Embeddings + lm_head (vocab 248,320)	1.271 B	3.2%
Hyper-Connections (4 streams)	640.6 M	1.6%	Sinkhorn-Knopp doubly-stochastic mixing
Shared expert (dense FFN per layer)	236.1 M	0.6%	absorbs common residual flow
MoE routers W_gate (48 × [128, 2560])	15.73 M	0.039%	the only "glue" touched by healing

Layer pattern: 3 GDN : 1 full self-attention (QSA, sparse indexer 4×/2048). Factual memory injection happens in layer 2 via deterministic n-gram hashing → table gather → temporal conv → SwiGLU-gated residual injection (see §3). A few numbers worth internalizing before the methodology:

• The upstream was already mostly idle capacity. 512 experts per layer, top-10 routing: ~2% of the experts evaluate each token.
• Management is microscopic. The 48 routers that decide everything weigh 15.73M parameters — 0.039% of the model. (Which is why "just retrain the routers" is both tempting and impossible — §6.3.)
• A third of the model was a phonebook. 51.2B of 175B parameters (29%) were the n-gram table — stored facts, zero FLOPs. In the 40B it is still 5.12B (12.8%).
• The 75% cut saves memory, not compute. Active parameters per token (≈6.6B) are essentially unchanged from upstream — the win is 180B → 40B of stored weights, which is what makes local serving and quantization conceivable at all.
• Upstream had ~96 nearly-dead experts per layer. Capturing 95% of routing mass needs K≈416 (the mass curve of §2.3); the tail beyond that rides almost no traffic.

The checkpoint also carries a vision tower (333 tensors, Qwen2.5-VL) and an MTP draft head (31 tensors, 512-expert MoE — deliberately NOT pruned, it is the speculative-decoding draft); both are pass-through, never instantiated by transformers (text-only serving), preserved bit-exact in every published repo.

---

2. Pruning methodology

when I started, I thought this would be easy. On paper it was very clear: pick the best 128 experts, prune them, do a heal, package, go home. Oh boy, it was not. Before touching a single weight I had to learn which parts of the original model actually do the work — by instrumenting it while it reads a balanced buffet of tasks — and only then decide what to remove. And the cutting itself turned out to be the boring part: survivors keep their exact original bits; everything hard lived in the measuring, the memory chip, and the finishing school.

2.1 Calibration corpora

Two corpora, built to answer different questions:

• m100k — 198 rows / ~111.5K real tokens (30 strata by set-cover of markers, ChatML, 17 languages, floor 2.5K / cap 8K tokens per row). Purpose: expert traffic profiling of the already-pruned v3 model (§43) and the first real-activation mask (v4).
• GATE-DS — a deliberately balanced corpus so that every expert gets invoked: v1 = 24 rows / 27,881 tokens / 12 strata; v2 = 46 rows / 53,877 tokens (keep-domain + reject-domain + 16 languages + uncensored split). Purpose: contribution measurement under guaranteed coverage ("Blind Collector Defect": with E=512, k=10, a dataset that activates <95% of experts assigns zero saliency to never-invoked experts and amputates uninvoked capabilities — Chernoff–Hoeffding gives the required ≥5.4M-token budget for ε=0.05; my strata quotas are the practical fix).

Domain-level PPL of the v3-dedup model on m100k (pre-heal): swe_coding 3.58, math 5.48, science 5.48, code 6.30 — vs degraded hf_aya_jpn 7,532 (the multilingual tail was deliberately sacrificed; see §3.2).

2.2 Activation instrumentation — the signal artifacts

All mask generations from S4 onward (legacy v4) are computed from recorded activations of the original 512-expert model, not from weight statistics. The upstream FP8 checkpoint (125B + 51B n-gram

• 4B MTP) was loaded on a single H200 (141 GB) with: dtype="auto" (a torch_dtype=bfloat16 would upcast to 360 GB), PLE offloaded to CPU RAM (102 GB), FP8 kernel shim (kernels==0.16.0), max_memory 92% — 144.2 GB VRAM, 168 s load. The full operational recipe (including the 20 errors that taught it) is in docs/bitacora/RECETA-POD-H200.md, and the pod dissection playbook in docs/bitacora/GUIA_OPERATIVA_DISECCION_PODS_H200_RTX6000.md.

Signals recorded per forward (accumulator np.savez_compressed every 25 rows, crash-safe):

Signal	Shape	Definition
hits	[30, 48, 512] capture-time; [48, 512] as published in ckpt_state_final.npz (stratum axis aggregated — the stratum-indexed versions live in senales_*.npz)	selection count per (stratum, layer, expert); top-10 by raw router logits, softmax over top-10 only (3.5× speedup)
coact	[48, 512, 512]	pairwise co-activation (basis of the Ochiai redundancy signal — later falsified, §6)
mass (Σw̃)	[30, 48, 512]	rank-weighted routing probability mass captured
W_gate	48 × [512, 2560]	full router weights (from gate_w_full.npz, read via safe_open from the FP8 shards — the router is never quantized)

Final instrumentation artifacts (all committed to HF, see §9):

• ckpt_state_final.npz (98 MB) — hits + coact + the 34 gates captured by hooks (14 layers were disk-offloaded; hooking meta tensors is impossible, hence the safe_open pass).
• gate_w_full.npz (118 MB) — all 48 upstream routers (48×512×2560, bf16, compressed).
• senales_contribucion.npz (86 MB) + senales_v2.npz (88 MB) — REAP-style saliency S_e = (1/|X_e|) Σ g_e(x)·‖f_e(x)‖₂ over GATE-DS v1/v2.
• mascara_*.json — every mask generation (schema below), mmr_sweetspot_report.json, mmr_ranked_lists.npz, barrido_fino.json, n3_metrics.json, n3_hits.pt.

A full user guide for these artifacts — byte-verified schemas, capture method, the reproduction script, claim→artifact traceability, and community analysis ideas — is in docs/ACTIVATION-ARTIFACTS-GUIDE.md. The companion analysis (pipeline/08_analisis/analizar_activaciones.py, ~3 min of CPU) recomputes every selection number from the NPZs — the mass curve (51.8/66.1/77.6), the ×7.8 expert-usage lift, the 0.11% gate near-twins, the 1σ ordered-noise zone — and adds findings never measured before: routing entropy of the teacher is 8.58 of 9.00 bits (95% of maximum — over-dispersion measured directly); only 14 of 24,576 expert-slots were never selected (the waste is diffuse under-use, not dead experts); and 76.5% of expert pairs co-fire at least once (top-10 teams are real but loose).

Throughput engineering of the instrumentation itself: the m100k teacher pass streamed 108,580 tokens in 2,296 s; the GATE-DS pass added three optimizations — top-k computed on raw logits, the softmax evaluated only over the surviving top-10, and np.add.at vectorized scatter for the stratum accumulators — 24 rows (27,881 tokens) in 471 s = 52.5 tok/s, 3.5× over naive. Layers disk-offloaded under device_map cannot be hooked (meta tensors), so their W_gate came from a separate safe_open pass over the shards — lossless, because a router weight is static and (in the upstream FP8 export) never quantized.

Mask file schema (two generations, both forward-compatible):

// v3/v4 schema
{ "layer_masks": { "0": [int, ...], "1": [int, ...], ... } }          // 48 × K expert ids

// v6 schema (the one the surgery consumes)
{ "layers": { "layer_0": { "selected_experts": [int, ...] }, ... } }

2.3 Mask generations S1 → S6 (legacy v1 → v6)

Each generation is kept because each one falsified something specific.

The regime that shaped everything (understood in hindsight, TESIS-PODA-V4): with 512 experts of very fine granularity (intermediate/hidden ratio ≈ 0.23, finer than DeepSeekMoE's 0.69) and k/E = 10/512 ≈ 0.02, this model routes in the over-dispersed regime — the literature documents that standard importance signals (frequency, aggregated hits) collapse exactly here, because aggressive load-balancing flattens routing until neighbor ranks are statistically indistinguishable (§2.3's 1σ finding is that law measured on our own data; the direct measurement: routing entropy 8.58 of 9.00 bits, hottest expert holds just ×7.8 the uniform share, 14 of 24,576 expert-slots never selected — see the artifacts guide). The operator's principle behind the final criterion: "everything is a sacrifice — we select a few in exchange for what we don't serve."

S1 — split-repack (legacy v1; superseded). Split each 640-neuron expert into 2×320 sub-experts (exact additive decomposition of SwiGLU FFNs, DeepSeekMoE Thm. 1), prune to 256, repack 128×640 by cosine-affinity pairing. Valid math, but produced emergent auto-fusions [2k, 2k+1] and was abandoned for whole-expert selection in every later generation.

S2 — weight statistics (legacy v2). Router row-norms + embedding projections, CPU-only. Weight-space methods agree with each other — the embed-proxy pilot recalls 0.738 of the S2 mask — but both disagree with actual routing: against the forward-derived reference the recall is 0.228. Lesson: at 512→128 the value is in the ranking, not the retention — every reasonable method retains ~25% of mass; which 128 you pick is everything.

S3 — embed-space knapsack (legacy v3). Born of a false economy: my attempt to avoid paying for teacher forwards — pass the tokenizer's embeddings through the layers and watch what lights up. Download only the 48 routers (~315 MB) + embeddings; proxy signal E[token] @ W_gate.T → top-10 over CAL1; dual-Lagrangian knapsack with stratum quota. Retention 25.2% uniform across layers. Dedup variant (S3d, legacy v3-dedup, MMR greedy, score(e) = s(e) + 0.5·n(e) − g(e) − λ*·max cos(e,s) + Q·[deficit coverage], λ*=1.0) maximized intra-set diversity at −0.01pp mass cost. Gates V3 (orphan-specialist) and V5 (null-model) passed. This is the mask that produced the first pruned model — and it was, in effect, random: post-hoc measurement on real forward activations gave M_kept = 29.42% routing mass (random-128 baseline ≈ 25%), with 1,671,932 lost-specialist events (token×layer ×pruned-expert at >2.5σ), and only 30.26% overlap with what real routing prefers. Root cause (TESIS §2.2): with 52.1M routing slots, 94% of neighbor rank pairs in the decision zone (64–256) are statistically indistinguishable at 1σ — the mask's decision boundary is ordered noise. A structural note that reframes the whole dedup effort: the literature's usual free lunch — MoE models carrying many duplicate experts to merge away (the premise behind dedup-based pruning elsewhere) — does not exist in this teacher. Only 0.12% of the 512×512 gate pairs per layer exceed cosine 0.8 (0.026% above 0.9): the upstream is nearly orthogonal, so pruning here could not be "merging copies" — every removed expert was the sole carrier of whatever it knew. The dedup machinery (§6.1) was built for a redundancy this model doesn't have. Quantified later as the mass curve @128 = 51.8%, @192 = 66.1%, @256 = 77.6%, K95 = 416 (CAL1, λ=0): compressing to 1/4 the experts without retraining cannot capture more than ~half the routing mass — everything else must be healed or lost.

S4 — real activations of the 512-expert teacher (legacy v4-m100k). Forward of the full upstream over CAL1 (108,580 tokens, 2,296 s), signals as in §2.2, MMR with running-max ranked lists (any K is a prefix). Top-128-by-hits mask (mascara_v4_m100k_K128_lam005.json, sweet-spot sweep K∈{128,192,256} × λ=0.05).

S5/S6 — contribution signals on the balanced census (GATE-DS) (legacy v5/v6). Three selection criteria compared on identical signals (K=128):

Criterion	Slot mass	Σw̃ mass	Worst stratum	Spread
freq (top-128 by count)	58.1%	60.6%	39.4%	28.6
rank (Σw̃ rank-weighted)	57.5%	61.4%	42.3%	25.9
mesa (greedy minimax over strata)	48.5%	50.5%	47.5%	8.8

Signal definitions, so the criteria are reproducible from the §2.2 accumulators. For expert e in layer l over a corpus partitioned into keep/reject rows: m_e (Σw̃) is the rank-weighted routing mass, m_e = Σ_rows Σ_{j≤10} w̃_j·1[e = top_j(row)] with w̃ the top-10 softmax — this discounts an expert that is merely rank-10 often in favor of one that is rank-1 often. keep_e/rej_e restrict the sum to keep-/reject-domain rows; v6disc ranks by keep_e − λ·rej_e (the only criterion that actively pushes reject-domain traffic out: λ=1.0 cuts reject-leak 63.0% → 43.6% at equal keep-mass, 69.8 → 69.5). Redundancy terms: gate-cosine cos(W_gate[:,e], W_gate[:,f]) — the only redundancy signal that behaved as advertised — and Ochiai co-activation |hits_e ∧ hits_f| / √(|hits_e|·|hits_f|) (falsified, §6.1). mesa is a greedy minimax: start from the empty set, repeatedly admit the expert that maximizes the minimum covered mass across the 30 strata — it buys worst-stratum safety (47.5% vs 39.4% for frequency at equal K) at the price of total mass (50.5% vs 61.4%). The criteria agree on the obvious top ranks and diverge exactly inside the 1σ-flat decision zone: pairwise Jaccard mesa↔rank 0.458, freq↔rank 0.821, with maximum divergence in layers 0–7 where the per-layer mass curves are flattest.

v6-keep = rank restricted to keep-domain rows. Final surgery consumed keep-rank (conservative; mesa was kept as the worst-case-guard alternative). A 204-combination mega-sweep (9 criteria × 17 sizes K∈64…480) located the Pareto frontier: K=192 @ IQ3_XXS (21.2 GB) holds more keep-mass (+14.2 pts) than K=128 @ Q4_K_M (22.5 GB) — quantization headroom can buy expert count (see MEGA-SWEEP-128-VS-192.md). I shipped both K=128 (trained) and K=192 (untrained, reference).

What the GATE-DS curve says about where mass lives: K=32 → 27.4%, 64 → 41.7%, 96 → 52.5%, 128 → 61.4%, 192 → 75.1%, 256 → 84.9%, 384 → 96.4%. No stratum saturates at 128 (worst: tcs_math 51%); the real knee is ≈384. Pruning 512→128 is a violent act; the post-training pipeline exists to pay for it.

2.4 The surgery (bit-exact expert slicing)

pipeline/01_poda_cirugia/runpod_cirugia_v6.py — streaming shard-to-shard worker: download → slice → upload → drop, the disk never accumulates more than one shard.

• Whole-expert selection, zero recomposition: the only tensors touched are three families — mlp.gate.weight [512,2560], mlp.experts.gate_up_proj [512,1280,2560] (fused SwiGLU 2×640), mlp.experts.down_proj [512,2560,640] — sliced as v[sel[l]]. No splitting, no merging, bit-exact by construction (bf16 preserved).
• Pass-through everything else: vision tower, MTP head (kept at 512 experts on purpose), embeddings, norms, PLE conv/projections, shared expert — one-byte-difference assert vs source, abort on mismatch.
• In-flight NaN/Inf asserts per tensor; n_sliced == 48×3 assert; deterministic ordering (v3's non-deterministic numbering was a bug I do not repeat).
• One practical discovery during verification: the three MoE tensors of a single layer usually live in different safetensors files (layer 0's gate_up_proj in model-00002, its gate and down_proj in model-00003) — any checker that assumes co-location silently samples the wrong thing.
• Config patch: num_experts=K, ngram_vocab_size_base 20M → 2M (the orcarouter upstream config inherited the 51B-row table geometry; the shipped table is 32M rows — §3).

Verification C1–C6 (verificar_cirugia_v4_matematica.py), later extended with G2/G3:

Check	What it proves	Result
C1	config: num_experts, 48 layers	PASS
C2	index audit: key counts, MoE=147 (144 LM + 3 MTP), no missing files	PASS
C3	18/18 sampled MoE tensors torch.equal vs teacher (maxdiff 0.0)	PASS
C4	MTP MoE intact at 512 (3/3 exact)	PASS
C5	dense tensors intact (152 exact vs teacher)	PASS
C6	PLE table shards exact vs table repo, rows-sum check	PASS
G2 (added)	routing-equivalence test: pruned-W_gate top-10 ⊂ original top-k over 1K tokens	added after C-gaps audit
G3 (added)	embedded-table hit-rate ≥ 85% on disjoint corpus	added after C-gaps audit

The honest gap analysis (§89.3) is part of the record: C1–C6 initially left 126/144 MoE tensors without value-checks and 75% of the table uncompared. Verification suites must be audited for coverage, not just run.

2.5 The router-truncation problem

The surgery does not rescale routers (C3 is bit-exact). What breaks is the softmax denominator: P(e) = exp(z_e)/Σ_{i=1..512} exp(z_i) becomes Σ_{i=1..128} — the partition function contracts ≈4×, chilling the distribution and concentrating probability on the highest- norm experts; upstream rank-11–20 experts become top-10/128; and some tokens have their entire top-10 outside the mask (M_kept = 0%). Everything downstream (router temperature τ=1.5 at inference/training, asymmetric LR on routers, Focal-CE, finally RLVR) is engineered against this single wound. Healing it in isolation is impossible — see §6.3.

---

3. The PLE n-gram memory

Layer 2 carries a Per-Layer Embedding (PLE): deterministic 64-bit mix-hash of the last bigram+trigram (per-head multipliers), direct gather (zero FLOPs) from a [32,002,176, 160] bf16 table (16 heads × 2,000,136 rows; 128 shards × 250,017 rows), Conv1D(k=4) → RMSNorm → SwiGLU-gated injection into the residual stream.

For scale: the upstream table is physically 128 shards × 2,500,012 rows = 320 M rows (51.2 B parameters); the pruned table keeps 16 × 2 M rows of that universe. Sparsity is the whole point — the product-key-memory literature's "capacity = usage" warns that with ~1 M calibration tokens most of 32 M slots are unwritable anyway, which is why hit-rate on a disjoint corpus, not row count, is the only honest metric for this table (§3.2).

3.1 The stale-table incident (hash multipliers)

transformers derives per-head multipliers from _splitmix64/PRIME_1=10007 with (vocab_size=248320, ngram_size=3, seed=1234) → official [23703573157769, 20109073645365, 8052911324071]. My first rebuild pipelines used ad-hoc multipliers [35319062748231, 10034993358913, 51164106034567] — every lookup missed; the published 2M table was noise, while DONE_TABLA_2M.json certified it "VERIFIED_SOTA" (validation had measured hit-rate on a corpus contained in the build corpus — circular by construction). Lesson institutionalized: a validation set must be disjoint by hash, not by intent, and a "verified" marker is only as good as its data split.

3.2 Rebuild: PLE1 → PLE4 (legacy raw512 → v6.3)

Hit-rate = fraction of lookups landing on written rows (16 lookups/token), measured on a disjoint production corpus (120 real TCS traces + adversarial battery, 4.05M lookups):

Code	Legacy	Method	Disjoint hit-rate
PLE1	raw512 (deployed)	official-hash remap of upstream rows	10.74%
PLE2	v6	keep-first-wins hash-map + reject pass + streaming prune + L2-fill by norm	24.40% (corpus under-delivered)
PLE3	v6.2	vectorized builder (numpy uint64 hashing, stable-sort first-wins, fancy indexing) — 30 min end-to-end vs 4–6 h; priority keep → sota_gated (31.5M tokens, 7 quality gates) → reject	89.55% (55.0% rows written, rest clean zeros)
PLE4	v6.3	PLE3 + L2-fill of remaining rows from PLE1 (norm-checked)	89.55%, 0 zero-rows, median row-norm 0.1005

The T2 causal ablation (ngram_embedding → 0, NF4 on A100): overall PPL 25.88 → 33.54 (−7.66) with the table; per-domain: code −2.53, logic −2.38, math −1.11, multilingual +7.22 (deliberate sacrifice). A linear 160-dim domain probe decodes 6 domains at 42.9% vs 16.7% chance — the rows encode domain semantics, they are not decorative.

An FP8 trap on the way (§100-series): the first v6.2 was rebuilt from the FP8 upstream — float8_e4m3fn rows dequantized wrong gave row-norms ~470 (vs 0.094), injections ~4,000× too large, PPL 255,817. Fixed by building from the BF16 orcarouter source with a permanent builder assert norm < 0.5.

3.3 The load bug — the single most expensive bug of the project

The checkpoint stores the table as 128 sub-tensors ...layers.1.ple.ple_embedding.ngram_embedding.shard_{0..127}.weight [250017,160], but transformers 5.17 defines the parameter fused ngram_embedding.weight [32002176,160] (split_ngram_parts no longer exists in the class). On every from_pretrained: the 128 shards → UNEXPECTED (silently ignored); the fused tensor → MISSING → random initialization (_init_weights covers only the multipliers). The 10.24 GB of factual memory was fresh random noise in every single process — training runs, probes, benchmarks — since the day the environment updated to 5.17.

This one bug explains, with zero residual: pod-to-pod behavioral variance (each load = a new 10 GB draw), the chronic OFF_FACTUAL failure (NaCl/H₂O), why the good v6.2/v6.3 tables "never helped", and the degenerate probe episode (a toxic draw under adapted routers).

Fix (pipeline/02_tabla_ple/cargar_tabla_ple.py): post-load, concatenate shard_0..127 in numeric order and copy into the fused parameter block-wise. Validation: median row-norm 0.25 (noise) → 0.0996 (spec 0.1005). Save-side mirror quirk: 5.17 re-splits the fused table into 128 shards on save — integrity asserts must treat the fused key as absent-expected.

---

4. Post-training

The P-axis of the naming convention maps onto this section: P1 = first finishing attempt (over an unplugged memory chip), P2 = format school, P3 = preference school, P4 = the verified-drilling run that produced the champion (§3 above documents why P1–P2 scored lower than they should have).

4.1 Data factory (Best-of-3, TCS tiers)

• Seeds: 3,432 frontier seeds from r0b0tlab, 100% gold-solution verified, 7 domains.
• Generator fleet: 4 × L40S pods running my own small distillers (one per tier) with a native PyTorch batched engine (num_return_sequences=3 prefix sharing; vLLM was unusable for this stack at the time — see §5). Dynamic per-tier token budgets: low 600 / mid 950 / high 1,500 / xhigh 2,400.
• Output: 38,158 raw traces (369 MB) → 6-gate selection harness (AST, symbolic-math, TCS skeleton, anti-refusal, true-OFF ≤50 words, SimPO pair synthesis) → curricula:
  • qwen_sft_curricula_v4.jsonl — 13,477 traces / ~22.5M tokens, ordered OFF→LOW→MID→HIGH→XHIGH
  • qwen_simpo_preference_v4.jsonl — 3,187 pairs (2,328 anti-overthinking + 859 rigorous)
  • dense-LIMA variants (5,500 + 2,000), and the final post-finetune gold 1,202 (682 columnar arithmetic + 520 cognitive anchors), audited 0 schema errors, 0 pad tokens, 0 math errors.
• TCS (Thinking Cap System): 5 reasoning tiers (off/low/mid/high/xhigh) with calibrated jinja (docs/chat_template_calibrado_5tiers.jinja); tier-OFF pre-fills <think>\n\n</think>\n\n and teaches immediate direct answers.

Who generated all of this: the transductor fleet. The reasoning content comes from frontier-source seeds (3,432 gold-verified traces); what the fleet adds is the format — four small in-house transductor models, one per thinking tier (Davd-b01/lfm2.5-2.6B-thinkingcap-distiller-v2 for low, transductor-mid/high/xhigh-v3 for the rest) re-express those traces in TCS structure. They don't think; they translate. They ran on 4× L40S with a native PyTorch batched engine (~1,350 tok/s at batch 32 — no vLLM; it didn't support this stack at the time). The same family exists as a production transductor (produccion/): "re-express foreign reasoning traces into verifiable TCS format. It does not solve tasks — it transduces them; the correct answer already arrives in the input, the model just must not ruin it." The design trick that makes that verifiable at scale is the gold slot (below): containment (R13) and coverage (R14) are checked deterministically against it, with no LLM judge.

TCS-IN, the deployment format (normative spec: docs/TC-STANDARD.md): production feeds the model a 4-slot structured user message — <tc_meta/> (declared shape/domain/lang), <tc_context> (prior turns, multi-turn shapes only), <tc_task> (the question), <tc_trace> (the teacher's raw reasoning — reference material, discarded), <tc_final> (the gold answer). Production mix: single 45.2% / agentic 20.3% / multiturn_tools 17.9% / multiturn 16.6%; reading:generation ratio 15:1; typical input ~1,400 tokens. This is the workload the masks were not optimized for (error #23) — the calibration/serving mismatch was real and measured.

Tokenizer facts (needed to debug any template of this family): <think> = 248068, </think> = 248069, end-of-turn <|im_end|> = 248046; the PLE's internal EOS marker is 248044. Factory traces in <tc_think>…</tc_think> format map canonically: <tc_think> → <think>, </tc_think> → </think>, <tc_answer> stripped.

4.2 SFT — asymmetric DoRA

Trainer: train/train_qwen38_cognitive_dora.py (native PyTorch + Accelerate; Unsloth segfaults on this architecture, TRL/Axolotl would silently drop vision+MTP on save).

• Targets (layers 8–47; 0–7 frozen to protect the layer-2 PLE): 250 DoRA modules — linear_attn.{in_proj_qkv, out_proj, in_proj_z} (30×3), self_attn.{q,k,v,o}_proj (10×4), shared_expert.{gate,up,down}_proj (40×3) — plus 48 full routers [128,2560] (copy_, weight-complete, not delta). 118,190,080 trainable params (102.46M DoRA r=64 α=32 ⇒ scaling 0.5, + 15.73M routers) = 0.29% of the model.
• LR: DoRA 1.5e-4 cosine; routers 3e-5 (5× smaller); v6 recipe adds router warm-up ×3 over the first 20 steps, applied after scheduler.step() (the scheduler was overwriting it).
• Losses: prompt-masked CE + ST-MoE aux 0.005 + z-loss 1e-4 (aux set to 0 in v6 — it was flattening routing; M5 hypothesis confirmed in-vivo: 105–122/128 experts active, entropy 4.1–4.4 nats = 84–91% of ln 128, max expert share ≤6.6%). Token re-weighting evolved: Focal-CE 3.0× static (the pad disaster, §5) → delimiter guard 2.0× (4 tokens) → TF-IDF weighted CE with per-row normalization + cap 1.5 (arXiv:2609.11029-style; attacks memorized substrings = the literal substrate of the loop attractor).
• Batch = 1 sequence, zero padding, grad-accum 8. Sample packing is forbidden here: the GDN recurrent state bleeds across concatenation boundaries. Batched chosen/rejected pairing was measured and rejected: 450 s vs 362 s per step (−24%; pads traverse the MoE grouped_mm, +22% wasted expert compute) — padding, not latency, is the dominant resource in batch=1.
• NEFTune α=8 (noise U(−α/√(L·d), +α/√(L·d)) on embedding outputs, SFT phase only; the hook leaking into SimPO was a real bug I later gated).
• RFT anchors: 326 verified-correct RLVR anchors (arXiv:2308.01825) merged into the SFT as plain CE rows → the same behavior is seen three times: SFT (learn) → SimPO (context) → RLVR (anchor rehearsal, 28.8% of the RLVR set).

Convergence (final SFT, 1,130 rows, 120 steps): loss 2.33@10 → 0.94@40 → 0.65@80–120; routing healthy throughout. (The earlier v5-gold run: 1.9423 → 0.6598, PPL 1.93.)

4.3 SimPO — the iso-density fix

Loss: -log σ( β/|y_w| · Σ log π(y_w) − β/|y_l| · Σ log π(y_l) − γ ) — length-normalized, reference-free. β=2.0 throughout; γ 0.5 → 0.3 → 0.1 across generations.

v6 failed at 39.3% preference accuracy (v4 before it: 43–52%). The measured cause: the 160 "rigorous verification" pairs had rejected answers ~4× shorter than chosen (r/c = 0.41–0.47) — a length-normalized objective structurally prefers the short predictable text over the dense proof; 98/332 pairs (30%) were anti-correlated with the training goal. The pairs that could be learned (anti-overthinking, arithmetic rejections) did learn (LOW verbosity compressed 120 → 67 tokens).

The fix (v6.1): iso-density curation — hard floor r/c ≥ 0.85 with growing ceilings ([0.85–1.18] → [0.85–2.5]) as quota demanded, re-rendered to canonical per-tier directives, chosen re-verified against ground truth independent of the factory. Same 332 pairs by count (48 off + 84 anti-overthinking low + 40 arithmetic rejections + 160 iso replacements), γ=0.1, NEFTune off, warm-started from v6.0 with a 15-step boxed/xml mini-SFT first. Result: preference accuracy 80.7% (batches: 87.5, 87.5, 75, 75, 72.5, 81.2, 84.4, 82.5) vs 39.3%. Same algorithm; the dataset was the bug.

The 40 arithmetic rejection pairs deserve their own note: only real factory failures (reward_chosen=1.0 ∧ reward_rejected=0.0), chosen re-verified against GT (528/552 pass), rejected selected for loop attractors (29/40 with 4-gram score >0.20) so the negative gradient lands exactly on the observed pathology.

4.4 RLVR — verified rewards, offline

Design constraints from the literature (DeepSeek-R1/DeepSeekMath/Tülu-3/STaR/ReST): deterministic verifiers only (no learned reward models — Goodhart), explicit negative gradient on failures, length normalization, 25–35% anchor rehearsal, token-exact ChatML boundaries.

• Verifiers: SymPy equality with LaTeX normalization (\ln x, \frac{a}{b}, \cdot), Euclidean division via the fundamental identity d·Q+R = D, 0 ≤ R < d; Python ast.parse; canonical XML/JSON tool-call schema; 4-gram loop detectors (>0.20); unicode-normalized exact match (H₂O ≡ h2o); bidirectional multiset boxed-numeric.
• Dataset: unified 855 contrastive pairs (gold R=1 vs real on-policy R=0 rollouts) + 342 anchors; final RLVR-50 set: 400 pairs = 280 on-policy (70%) + 120 Thinking-Cap anchors (30%, 24/tier), iso-density median r/c 1.008, hard ≤1,664-token cap (0% truncation; −20% peak activation memory).
• The shipped run was OFFLINE. The 50 steps consumed the static curated dataset above — the "on-policy" rollouts were generated before training (SGLang session, Sep 17), not during it. The worker also implements an online GRPO-lite mode (--online N: N fresh rollouts sampled per optimizer step, fresh failures becoming contrastive pairs against their own gold, reward_fresh_pct metric) — built, tested on dry-runs, but not used for the champion; it is available for the v2.
• Run: 50/50 steps, 12.8 minutes on H100 NVL; final preference accuracy 64.8% (from 50% chance), contrastive loss 0.81 → 0.60–0.70, grad-norm ~1.34, 82 GB VRAM sustained.
• Clinical probes (12, across all 5 tiers): OFF identity loop eradicated (60 → 7 tokens, clean <|im_end|>), palindrome AST-valid, anti-sycophancy intact (proves 9=3×3 against user assertion), knights/knaves truth table, (n²+1)/(n+1) divisibility, Baker–Gill–Solovay relativization, Manacher O(N), canonical XML tool call.
• Fusion: in-place W = m·(W₀+ΔW)/‖W₀+ΔW‖₂ for 250 modules + router copy_; PLE table injected into the fused native parameter before saving (so every downstream framework loads it natively, no patch script); 17 bf16 shards (81 GB); C1–C6 audit: 1,627 tensors, 333 vision intact.

---

5. The error ledger

The complete list of failure modes I hit, diagnosed and fixed — because the negative results are half the value of this repo.

#	Symptom	Root cause	Fix / lesson
1	Repetition attractors (pad-token-flavored loops)	81.4% of the LIMA SFT set contained 18.22 MB of residual pad tokens (batched generate + skip_special_tokens=False in the factory), amplified by Focal-CE 3.0× on the pad delimiters	purged all datasets (18.44M pad tokens removed from 38,158 raw traces); removed those tokens from the critical list; tensor-level cropping in the factory
2	Tier 'high'/'mid' crashed the tokenizer	stock chat template only accepted ('xhigh','medium','low'), harness silently fell back to flat prompts	calibrated 5-tier jinja + template regression tests
3	First pruned model ≈ random pruning	my own false economy: to dodge the cost of teacher forwards, the v3-dedup mask was chosen on embed-proxy signals (tokenizer embeddings half-passed through the layers); M_kept 29.42% ≈ random 25%, only 30.26% overlap with real routing; 94% of rank-neighbor pairs indistinguishable at 1σ	moved to real-teacher activations (v4/v6); lesson: the cheap measurement is the expensive one — a mask that looks principled but isn't costs more than the GPU-hours it saved
4	"Verified" table was noise	circular validation (test corpus ⊂ build corpus) + wrong hash multipliers (§3.1)	disjoint-by-hash validation; builder asserts; multipliers read from modeling source
5	PPL 255,817 after table rebuild	table rebuilt from FP8 source (row-norms ~470, injections ~4,000×)	rebuild from BF16 source; permanent norm < 0.5 builder assert
6	Every pod behaved differently; factual probes chronic-failing	the PLE table never loaded — fused-vs-sharded key mismatch, silent random init (§3.3)	cargar_tabla_ple.py + row-norm gate 0.25 → ~0.10 in the load path of every script
7	"Routers are random" panic (½ day of theory)	HF XET bridge serves wrong bytes on HTTP Range reads (headers valid) — cos −0.0034 measured vs +0.9976 canonical	never read tensor bytes via Range; hf_hub_download + sha256 == lfs.oid + safe_open — verify the instrument before theorizing about the weights
8	Publication assert killed a finished model	C2 counted mtp.* keys as missing; transformers 5.x never instantiates MTP	asserts compare against instantiated keys with an absent-expected whitelist
9	FP8 export → gibberish	qwen4_exp has no scaled-FP8 GEMM modules; weight_scale dropped as UNEXPECTED; raw e4m3 matmuls inflated activations ~1,000×	BF16 is the only supported serving precision without a custom wrapper; FP8 repo deleted
10	Isolated router heal failed (coverage 8–15%)	zero task-gradient for non-top-k experts: ∂L_CE/∂W_gate[j] = 0 for unselected experts; <think> emission lives in the dense trunk	healing must ride the SFT (DoRA + routers jointly, Focal-CE); see §6.3
11	Watcher killed a healthy pod at 12 min	grep -c with 0 matches prints 0 and exits 1; the or-echo fallback produced "0 0" ≠ "0"	grep -q + numeric guards; test your watchdogs against both branches
12	RLVR crash at step 1 / OOM at 93 GB	del of logits before reading the loss; both chosen+rejected logit tensors alive simultaneously (2×248K×1536)	capture scalars before frees; free chosen logprobs before the rejected forward
13	SimPO accuracy 39–45%, margins negative	iso-length pathology r/c 0.41–0.47 in 30% of pairs (§4.3)	iso-density curation: 80.7% with identical hyperparameters
14	mass_curve reported 5.7% @K=1	rank-index vs K-grid indexing mismatch	validate invariants (cumsum[511] = 1.0, top-1 = 1.53%) before interpreting curves
15	Empty responses at eval time (up to 10–15%)	thinking budget exhaustion under a high-effort system prompt (also observed on the official dense 27B under my harness)	budget-aware probe design; report empty-rate alongside scores
16	Model merged from the wrong base	merge script defaulted to the obsolete v4 base; DoRA v6.1 lives in a different expert basis	base-pin asserts: adapters are only valid on the exact base they were trained on (cos 0.2568 between bases gave it away)
17	Adapter load silently loaded 0/250 modules	layer_N is a ModuleList index, not an attribute; shared_expert needs the mlp. prefix	hard asserts 250/250 + 48/48 on every load (ADAPTER-STATE-KEYMAP.md)
18	Disk-full during surgery-era jobs	PLE staged in RAM dict (10 GB), pruned shards accumulated	streaming everything; drop-after-upload; post-job disk gates
19	76 GB merged model lost	pod stop ≠ terminate; state gone	publish adapters only; merge on demand; nothing lives only on a pod
20	vLLM/Unsloth unusable on this arch	LFM2.5 conv-state corruption (vLLM v0); segfault on _short_conv/Sinkhorn (Unsloth)	native PyTorch batched engine for the factory; SGLang for serving the big model
21	Surgery upload died mid-flight, pod idled ~35 min	HF rate limit: upload_file = 1 commit each × 2 repos hit the 128 commits/hour ceiling (HTTP 429); the monitor watched the process, not the log	batched uploads / upload_folder for multi-GB outputs; monitors must grep logs for DONE/Error, not just check PIDs; on crash: kill first, diagnose after
22	"Our 40B beats the original in PPL" (briefly believed)	five evaluation protocols compared as if one — top_k 8 vs 10, PLE random/amputada/present, native vs mrope templates (TESIS §3.3)	PPL comparisons only between identical eval states; in over-dispersed regimes PPL is a broken-model detector, never a ranker (§6.7)
23	Masks derived on a corpus the deployment doesn't serve	calibration = ChatML, 17 languages, 2.5–8k-token rows; deployment = TCS-IN, English-only, ~1,400-token inputs (TESIS I3)	a mask optimizes the mixture it measures; either match the corpus to the workload or bound the gap (TV-distance argument) explicitly

Meta-lessons, in order of how much they cost me: (a) silent-load failures are the worst failures — gate every load with a physical invariant (row-norm, sha256, count asserts); (b) verify the measurement instrument against a hash-verified source before forming hypotheses about the weights; (c) a preference optimizer can only learn what the pair distribution allows — audit r/c before tuning γ/β; (d) validation sets must be disjoint by construction, not by intention; (e) monitor state, not processes — a crashed job with a live PID (and a live pod) is the most expensive kind of silence.

---

6. Rejected ideas, with falsifications

1. Co-activation (Ochiai) dedup as redundancy. Correlation with gate-cosine redundancy is only 0.492; ochiai_only was the worst selection mode at every λ; aggressive Ochiai dedup (λ=0.5) yields 28.3% mass ≈ random. With top-10 routing, co-activation is a team, not redundancy — dedup penalizes exactly the experts that capture mass. Consistent with NAEE (arXiv:2402.14800: frequency-based selection underperforms random) and REAP (arXiv:2510.13999) — and with this teacher's own near-orthogonality (0.12% near-twin pairs, §2.3): there was no duplicated mass to merge in the first place.
2. k₂ reference-renormalization (arXiv:2609.04575). Designed for inference-time k reduction on an already-adapted model; falsified live ("A3") for pool-pruning with constant k — softmax renormalization over the 128 survivors already preserves original top-10 weights. Code retained, default OFF (and one production crash because a relaunch skipped the falsification memo — process lesson).
3. Isolated router healing. Zero-gradient theorem (§5 #10) + the finding that generic text wakes 89.8% of experts at layer 0 (Δcos ≈ 0.02 between "active" and "dormant") — the experts were alive; the routing under task text is what needed training, and that gradient only flows through the CE loss itself.
4. FP8 weights for this architecture. §5 #9. (Upstream FP8 is fine on H200/B200; the 40B's own FP8 export is not, without custom scaled-GEMM modules.)
5. Wanda / LASER intra-expert denoising. Post-prune knowledge density per expert rises ~4×; unstructured sparsity inside 640-neuron intermediate would trade real capacity for zero speed at this scale (no kernel support). SVD tail-truncation risks exactly the arithmetic precision I was trying to protect.
6. Sample packing / padded pairing. §4.2 — recurrent-state contamination + measured 24% slowdown.
7. PPL as a quality oracle. arXiv:2609.04453: the lowest-perplexity pruning config produced the worst math. My gates are verifier pass-rates and behavioral probes; PPL/Gini are regression detectors only.
8. "Language lives in late layers" (would justify keeping late-layer experts only). Refuted: reject-domain share is diffuse (median 31.7%, per-layer tercile ratio 1.0, Gini 0.452 ≈ keep's 0.472).
9. Per-layer heterogeneous K (give heavy layers more experts). Null result: optimal heterogeneous allocation 51.9% vs uniform-128 51.8% (+0.1 pts) — layer mass curves are near-parallel.
10. K=192 tiered-cache serving (hot experts in VRAM, tail in RAM). Built the analysis, never deployed: decode routing is Zipfian (top 10–20% of experts → 80% of traffic), so it should work — parked as the reference v6rank-K192-bf16 repo.
11. Hot-swappable PLE memory cartridges. The table is provably load-bearing (T2: −7.66 PPL) and per-domain cartridges remain an open blueprint (cartridge-general micro-LoRA, 8 MB) — not productized.

---

7. Academic foundations

Complexity limits of arithmetic in transformers — Merrill & Sabharwal (ICLR 2024): fixed-depth autoregressive transformers ∈ uniform TC⁰; integer multiplication/division need ≥NC¹-depth (Barrington et al. 1992; Volkovich 2000) → the scratchpad theorem: problems in P \ TC⁰ require O(N) intermediate tokens. The MSD-first causal conflict (carries propagate right-to-left, generation goes left-to-right) is why multiplication dies while division (table-lookup + aligned subtraction, self-checking via d·Q+R=D) survives — confirmed empirically at 0/6 vs 7/8 under quantization (docs/VULKAN-APEX-TESTING.md). Mitigations: Abacus embeddings (McLeish et al., NeurIPS 2024), TIR (Nye et al.), PAL (Gao et al., ICML 2023) / ToRA (Wang et al., ICLR 2024) — program-aided delegation recovered 0% → ~100% on my own harness tests.

Pruning/MoE — DeepSeekMoE (arXiv:2401.06066, exact additive FFN decomposition — validates v1's split-repack math; its load-balancing loss is also what creates over-dispersion), DeepSeek-V3 (arXiv:2412.19437, loss-free balancing — the field saw this problem coming), MoEfication (arXiv:2110.01786), REAP (arXiv:2510.13999), NAEE (arXiv:2402.14800), Shazeer 2017 / Switch (Fedus 2022) for router-coverage requirements. On what to expect from aggressive prunes: DERN (arXiv:2509.10377 — fine-grained experts degenerate earlier), UMoE (arXiv:2607.11444 — "a lightweight fine-tune recovers about half of what aggressive pruning loses"), Half-the-Experts-All-the-Code (arXiv:2607.16721 — prune beats quantization only below 3 bits), MoEXBench (arXiv:2608.21693 — pruning is the dominant degradation source), MAPLE (arXiv:2608.15299, heterogeneous per-layer K — null result on our mass proxy, §6.9), Over-Dispersed Pruning/MESA (arXiv:2609.04453, the regime paper), Huffman-routing (arXiv:2607.20427, k/E-driven functional redundancy).

Memory — Product-key memories (Lample et al., NeurIPS 2019), Memorizing Transformers (Wu et al., ICLR 2022), Wegman–Carter universal hashing for the joint collision bound across H independent heads (P ≤ N^(−H) ≈ 10⁻⁹⁹ with N the row universe, H = 16).

Post-training — RFT (arXiv:2308.01825), STaR (Zelikman 2022), ReST (Gulcehre 2024), DeepSeekMath/GRPO (arXiv:2402.03300), DeepSeek-R1 (Guo et al. 2025), Tülu-3 (Lambert et al. 2024), SimPO (Meng et al., NeurIPS 2024), DPO (Rafailov 2023), LIMA (Zhou et al. 2023), NEFTune (arXiv:2310.05914), TF-IDF weighted CE (arXiv:2609.11029), DoRA (arXiv:2402.09353), LoRA (Hu et al. 2021), EWC (Kirkpatrick 2017) / experience replay for anchors, ST-MoE aux + router z-loss, Focal loss for token re-weighting.

Sampling — DRY (Chung et al. 2024) vs repetition penalty (CTRL, Keskar 2019): global penalties depress legitimate digit logits and cause arithmetic hallucination; DRY's context-conditioned n-gram penalty (0.8 / 1.75 / allowed-2) kills agent loops without touching arithmetic — but the penalty window must cover the agent turn history (default 64 is useless; 4096 works), and exact-match task-termination strings need sequence-breaker exemption.

Architecture — Hyper-Connections (arXiv:2409.19606) with Sinkhorn-Knopp (1967) doubly-stochastic mixing bounds residual amplification (‖∏A_r‖₂ ≤ 1 by Riesz–Thorin); Gated DeltaNet (Sun et al. 2024) recurrence; refusal-direction literature (arXiv:2406.11717) for why the n-gram table is censorship-neutral.

---

8. The 75% thesis: competence survives

The claim of this project is not a benchmark score. It is this: you can remove 75% of a large MoE's routed experts (48 × 512 → 48 × 128; ~120B → 30B routed parameters; an 180B-class model brought down to 40B) and the result is not a broken, gibberishing model. What breaks is selective and mappable — and most of it is either impossible at full scale anyway, routable around, or partially healable. What survives is the trunk of competence.

8.1 Evidence, strongest form: zero-shot, before any post-training

Immediately after the bit-exact surgery — no healing, no fine-tuning, routers still calibrated for 512 experts — the pruned model on an A100:

• loads clean (20.25 s, zero NaNs, zero compile errors);
• deduces lock-free concurrency requirements (O(1), single-producer/single-consumer) from a natural-language spec;
• resolves tool-call parameters correctly (list_all_cameras(include_offline=true));
• comprehends advanced Spanish legal doctrine;
• emits well-formed reasoning: <think> blocks that close, JSON tool schemas parsed correctly.

No gibberish. The pathology already present was structural, not linguistic: ~38% of the 128 experts per layer were asleep (routers expecting 512), with extreme routing inequality (Gini ≥ 0.70 in layers 3, 16, 25, 42, 44) — which produces repetition loops under greedy decoding, not incoherence. That is a calibration wound with a known fix, not brain damage.

8.2 Evidence after heal-in-SFT + SimPO + RLVR (the shipped champion)

From the 12-probe clinical battery (all quoted behaviors are from archived traces):

• Anti-sycophancy: when told (falsely) that all odd numbers are prime, refutes it with 9 = 3×3 — and holds the correction when the user pushes back.
• Formal logic: solves a knights-and-knaves puzzle with a complete truth table ("Primary Deductive Analysis").
• Olympiad algebra: resolves the divisibility condition of (n²+1)/(n+1) step by step to n ∈ {−3, −2, 0, 1}.
• Theory of computation: cites and demonstrates the Baker–Gill–Solovay (1975) relativization barrier on P vs NP, with oracles A and B.
• Algorithms: explains Manacher's O(N) palindrome algorithm with radius tracking.
• Code architecture: for an LRU cache it produced a 665-token <think> covering race conditions, the O(1) hash-map/list split, lock discipline and pointer consistency after eviction — closing cleanly with "The solution seems robust. I will now proceed to generate the final code." A segment tree with lazy propagation got a 1,148-token deliberation on the deferred-propagation invariant, closed without loops.
• Agentic tool use: canonical XML dispatch (get_weather(city, units)), 100% valid syntax across probes; with a real agent harness + code execution the model — even in its degraded quantized form — fixed a planted calculator bug (94 productive steps, edit-verify cycle) where without the harness it looped forever: the scaffold compensates what the weights lost.
• Arithmetic with the trained format: 935 × 18 → columnar proof → \boxed{16830}, correct, zero loops, in BF16.
• Tier discipline is real: OFF answers directly with zero <think>; thinking time adapts (LOW ~8–15 s, MID ~48–120 s, HIGH ~230–320 s); factual recall (NaCl + H₂O) recovered in-vivo once the PLE table actually loaded — the training dynamics visibly pulled the fact out of the 10 GB table.

Public benchmarks for the same model, with their forensics (parsers penalize its \boxed{} convention; ~10% empty responses are thinking-budget exhaustion — a pathology I also measured on the official dense 27B under my harness, i.e. partly harness-induced):

Suite	Score	Forensic note
GPQA Diamond	27.78%	≈ chance+2.8pp — factual recall is the prune's main casualty
IFEval (inst strict)	48.56%	prompt-strict 37.34% — \boxed{} convention punished
MATH-500	17.80%	re-scored locally: no false negatives found, format mismatch real
LiveCodeBench v6	15.55% pass@1	1,055 problems, 4,072 traces archived

Good and bad, side by side. The same model can produce both of these — the pair is the honest summary of the whole project:

✅ Good (archived trace)	❌ Bad (archived trace)
Same task, LRU cache: a 665-token <think> covering race conditions, the O(1) hash-map/list split, lock discipline and pointer consistency — closed cleanly with "The solution seems robust. I will now proceed to generate the final code."	Same task, same session: at T=0.0 without repetition penalty, the response fell into a greedy attractor repeating the doctest example line forever until the token cut (§73.1 of the log). Right analysis, trap in the sampler.
935 × 18 → columnar proof → \boxed{16830}, correct, zero loops, clean end-of-turn stop	935 × 18, other session → \boxed{16350} followed by a digital-root "verification" — (9·8) mod 9 = 3 — which is itself arithmetically false. The shape of self-checking without the substance.
Anti-sycophancy: told (falsely) that all odd numbers are prime, refutes it with 9 = 3×3 — and holds when pushed	Pre-RLVR arithmetic: 47 × 19 → "791" (it is 893); v6.1 computed 17 + 25 → 43. The carry circuits were the prune's wound and stayed open until the RLVR stage.
Division, RLVR format: Euclidean layout with Quotient=/Remainder=, self-checked via d·Q+R = D — the only arithmetic format with a real built-in error signal	Factual recall, random-table era: NaCl + H₂O failed chronologically for days — because the 10 GB factual table was silently random in every load (§3.3). Not the model's fault; my pipeline's.
Tool use: canonical XML dispatch (get_weather(city, units)), 100% valid syntax across probes	Multilingual tail: Japanese PPL 7,532 on one corpus — sacrificed deliberately for expert budget, still a real loss

Read the pairs right: the prose, the structure, the self-correction habits are very good — the writing quality is the model's best feature. The failures are concentrated and diagnosable: arithmetic robustness (prune + sampler config), factual recall (prune + my broken-base bug), and format artifacts under benchmark parsers.

8.3 The honest failure list — and who is to blame

The benchmark scores above are bad, and they split three ways. Part is the prune: 75% of the expert mass is gone, and the casualties (long-tail factual associations, arithmetic carry robustness) are exactly where a 128-expert substrate is thinnest. Part is also me beyond the finetune: the first masks were derived from cheap embed-proxies specifically to avoid paying for teacher forwards — the false economy that made the first pruned model statistically random (§2.3, §5). And the finetune carries its own scar: the P2 stage (SFT 120 + SimPO 40) trained against a factually random memory table because the PLE-load bug was found mid-campaign; P3/P4 healed over it with the real table loaded, but the foundation of the lineage was laid on a broken base, and two SimPO rounds were burned on a dataset that structurally could not learn (§4.3). Part is measurement: benchmark parsers punish the model's \boxed{} convention, and budget-exhaustion empty-responses (which the official dense 27B also shows under my harness) depress every score.

And a fourth candidate that must be named: the Thinking Cap itself. The TC — the structured way of thinking this project teaches — is its biggest achievement, and precisely because every trace in the lineage passes through it, it is also the most probable single culprit if the failure is pedagogical. One thing this suspect is not: the distillers don't think — they transduce foreign frontier traces (gold answers included) into TCS format, so the reasoning content came from strong sources. The suspect is the format of thinking the content is delivered in:

• The pairs pulled in opposite directions: anti-overthinking pairs reward the short answer, rigorous-verification pairs reward the long one — for two generations the model was graded by contradictory objectives until iso-density fixed the length axis (§4.3).
• The benchmark-hurting conventions are TC style: \boxed{} everywhere, the "Primary Deductive Path" verbosity that exhausted probe budgets, the nested-\boxed attractor that affected 10.6% of LiveCodeBench — inherited from the thinking format itself, not from the base model.
• Ritual verification may be a property of the format: the verification-shaped-but-wrong failure (item 3 below) has a plausible teacher — a way of thinking whose "checking" sections can be performed perfectly as form while staying empty as substance. The quant experiment made the pattern explicit: training real self-checking patterns beats training rituals with verification formatting.

None of this is proven — the TC cannot be ablated from this lineage retroactively. It is the named suspect, and the v2 should run a TC-less (or TC-modified) arm of the experiment. If the failure is the prune, the TC survives it; if the failure is the TC, the prune was never the binding constraint. One caveat bigger than all of them: the experiments that would apportion the blame precisely — the same suite on the unpruned 512E upstream, and the finished dense-27B control — were never run, because the budget ran out at ~$300. That is the single biggest hole in this work, and I would rather state it than paper over it.

1. Exact multi-digit arithmetic without a scratchpad is out of reach — but by the TC⁰ barrier this binds the unpruned upstream equally; it is a property of the architecture class, not of the prune. What the prune did do: remove spare capacity that had partially compensated, so pre-RLVR the champion failed probes the dense sibling passes (47×19 → "791"). RLVR on columnar formats fixed it in BF16 (16830 ✓).
2. Factual recall is the real casualty. GPQA ≈ chance. The PLE table compensates measurably (T2: −7.66 PPL; NaCl recovery) but doesn't fully restore a 75%-smaller routing substrate's long-tail associations.
3. Verification-shaped text can be wrong. In one archived probe the model emitted a columnar multiplication ending in \boxed{16350} (935×18 is 16830) followed by a digital-root "verification" that was itself arithmetically false — it has the form of self-checking without always the substance. Training verifiers reward substance; serving stacks should still delegate arithmetic to tools.
4. Greedy decoding loops at T=0.0 without a penalty (repetition penalty 1.05 or DRY with a window covering the agent history fixes it — a sampler configuration, not a weights problem).
5. The multilingual tail was deliberately sacrificed (+7.22 PPL; Japanese PPL 7,532 on one corpus) to buy keep-domain mass for the 128-expert budget.
6. BF16 is the only validated serving precision. The APEX GGUF quant (Vulkan) preserved division (7/8) but destroyed multiplication (0/6) and degraded factual recall — the fragile circuits are exactly the ones the prune already stressed. Documented as a negative result.
7. Formatting conventions (\boxed{} vs ANSWER: regexes) depress standard benchmark parsers; several "failures" above are format artifacts, honestly labeled as such.

8.4 What this demonstrates

First, the plain admission: the quality I wanted was not reached. Whether that weight falls on the extremity of the 75% cut or on my own heal/finetune mistakes (the unplugged memory chip, the anti-learnable SimPO pairs, the near-random first mask — each documented in §5) cannot be fully apportioned from inside this run. What the record does show is that every failure has a named cause and a named fix, and that the headroom is real (K=192 = +14 pts of routing mass before retraining).

And the work is unfinished, not merely imperfect: the upstream-baseline run and the full dense-27B control — the two measurements that would turn this record's causal story into a closed argument — were cut when the budget ran out. The record ends where the money did.

Not that the 40B is a frontier model — it is not, and §8.3 says where it isn't. The literature's own expectation bar is worth quoting here: a lightweight fine-tune recovers about half of what aggressive pruning loses (UMoE, arXiv:2607.11444) — which is roughly where this model landed, RLVR included. It demonstrates that the 75% reduction itself is survivable: with instrumentation-driven expert selection, bit-exact surgery, and a post-training pipeline targeted at the actual wounds (routing calibration, format discipline, self-checking arithmetic, anti-sycophancy), the product is a coherent, formally-reasoning, tool-capable model — the damage arrives as a mapped list of specific lost circuits, not as incoherence. Pruning at this ratio is an engineering method with a known cost schedule, not a gamble.

---

9. Toolbox: the reusable pipeline, script by script

Everything that made this project work ships here. This section is the operating manual: what each stage's script does, its entry point, and the pattern worth stealing. Most patterns are model-agnostic — any MoE with gate / gate_up_proj / down_proj expert tensors can go through the same machinery by swapping repos and masks.

9.1 Prune: signals → mask → surgery

Signal derivation — pipeline/01_poda_cirugia/derivar_senales_contribucion.py and mega_analisis_v6.py: run the original (teacher) model over a balanced corpus and accumulate hits, co-activation, rank-weighted mass (Σw̃) and the router weights into an NPZ checkpointed every 25 rows (crash-safe: a 2,296 s teacher forward survives 3 OOMs). The companion mega_sweep_v6.py turns the signals into a decision table — 9 selection criteria × 17 target sizes = 204 masks scored on keep-mass / reject-leak / worst-stratum / bytes — so the "which experts?" debate becomes a measured Pareto frontier instead of taste.

The surgery worker — pipeline/01_poda_cirugia/runpod_cirugia_v6.py. This is the script that produced both published foundations:

python runpod_cirugia_v6.py \
  --masks    mascara_v4_m100k_K128_lam005.json mascara_v4_m100k_K192_lam005.json \
  --targets  Davd-b01/qwen-3.8-next-40b-exp-v6rank-K128-bf16 Davd-b01/qwen-3.8-next-40b-exp-v6rank-K192-bf16 \
  --out-dirs /workspace/out_k128 /workspace/out_k192          # --dry-run first, always

What it does, and why it is reusable as-is:

• streaming shard-to-shard: download → slice → upload → drop — peak disk is one shard, never the 360 GB checkpoint;
• whole-expert, bit-exact slicing of exactly three tensor families per layer (gate.weight [512,2560], experts.gate_up_proj [512,1280,2560], experts.down_proj [512,2560,640]) via v[sel[l]] — no recomposition, bf16 preserved, maxdiff = 0.0 vs teacher (C3);
• pass-through with a one-byte assert for everything else: vision tower, MTP head, PLE, shared expert, embeddings, norms — abort on any mismatch;
• in-flight NaN/Inf asserts, n_sliced == 48×3, deterministic expert ordering;
• config patching (num_experts=K, ngram_vocab_size_base corrected) and tokenizer copy — the two things v3 forgot.

Surgery verification — pipeline/01_poda_cirugia/verificar_cirugia_v4_matematica.py: the C1–C6 audit (config → index → bit-exact MoE sampling → MTP intact at 512 → dense tensors → table shards). Reusable as a template; steal the meta-lesson too: audit your verifier's coverage (my first C-suite silently skipped 126/144 MoE tensors and 75% of the table).

Forensics — forense_expertos_v6.py (per-layer starvation/Gini/co-activation diagnostics), analizar_senales_mesa.py / monitor_cirugia.py (live worker telemetry).

9.2 Finetune: the two trainers

SFT + SimPO in one run — train/train_qwen38_cognitive_dora.py (P2/P3):

python3 -u train_qwen38_cognitive_dora.py \
  --model-repo Davd-b01/qwen-3.8-next-40b-exp-v6rank-K128-bf16 \
  --sft-data qwen_sft_v6.jsonl --simpo-data qwen_simpo_v6_1.jsonl \
  --init-adapter dora_adapters_v6_0_simpo_final.pt \   # warm-start (P2→P3 cascade)
  --target-layers 8-48 --max-sft-steps 15 --max-simpo-steps 25 \
  --neftune-alpha 8 --tfidf --router-warmup-steps 20 \
  --lr 1.5e-4 --lr-router 3e-5 --simpo-lr 2e-5 --simpo-lr-router 5e-6

The design decisions baked in (each one earned):

• asymmetric DoRA: 250 modules (GDN in_proj_qkv/out_proj/in_proj_z, QSA q/k/v/o, shared-expert gate/up/down) at r=64/α=32, plus the 48 routers trained jointly as full weights (copy_, not delta) with a 5× smaller LR and ×3 warm-up — routers-only training is a proven dead end (§6.3);
• batch = 1 sequence, zero padding, grad-accum for batch size — mandatory for the recurrent GDN layers (packing bleeds state across samples; padding wastes 22% of expert compute, measured);
• TF-IDF weighted CE (per-row normalized, capped) instead of repetition-prone plain CE; NEFTune on for SFT, hard-off for SimPO (a leak here poisoned logps once);
• prompt-loss masking, ChatML token-exact boundaries, z-loss on, aux-loss off (it flattened routing), --k2-reference 0 kept as a documented falsified flag.

RLVR with deterministic verifiers — train/train_qwen38_rlvr_dora.py (P4):

python3 -u train_qwen38_rlvr_dora.py \
  --data qwen38_rlvr_final_50steps_2k.jsonl \
  --init-adapter dora_adapters_v6_1_simpo_final.pt \
  --lr 1.5e-5 --lr-router 3e-6 --beta 2.0 --gamma 0.1 \
  --grad-accum 8 --max-seq-len 1664 --start-layer 8

• verifiers built in: SymPy with LaTeX normalization, Euclidean division via d·Q+R=D, ast.parse execution, XML/JSON tool-call schema, 4-gram loop detectors;
• online GRPO-lite mode (--online N) — available but NOT used in the champion's offline run: N fresh rollouts per optimizer step scored live (reward_fresh_pct), fresh failures become contrastive pairs against gold;
• anchor rehearsal with pure CE (28.8–30% of every set) against catastrophic forgetting;
• the memory bookkeeping that makes 94 GB enough: chosen-side logprobs freed before the rejected forward, chunked table-norm checks, explicit empty_cache() between phases.

Both trainers save the canonical adapter dict (layer_N.<module> → {magnitude, lora_A, lora_B, r, scaling} + 48 full routers) with hard 250/48 asserts — the format documented in docs/bitacora/ADAPTER-STATE-KEYMAP.md.

Fusion — pipeline/04_entrenamiento/fusionar_dora_qwen38_pod.py: in-place W = m·(W₀+ΔW)/‖W₀+ΔW‖ merge + router update + PLE injection into the fused native parameter (so any framework loads it without patches), sharded save, C1–C6 re-audit before upload.

9.3 Data: factory and curators

• pipeline/03_datos_curadores/factoria_l40s_best_of_n.py — Best-of-N generation with native prefix sharing (--tier {low,mid,high,xhigh} --n-candidates 3 --resume), dynamic token budgets per tier, self-healing resume keyed by prompt hash. Produced 38,158 traces on 4×L40S with a plain PyTorch batched engine (1,353 tok/s) — no vLLM needed.
• harness_seleccion_tcs.py — the 6-gate selection harness (AST, symbolic math, TCS skeleton, anti-refusal, true-OFF, SimPO pair synthesis).
• Curators, one per dataset, each with a one-shot guard: curar_ds_sft_v6.py (RFT anchor merge), curador_simpo_v6_1.py (iso-density pairs), curador_rechazos_aritmetica_v6.py (real-failure rejection pairs selected for loop attractors), curador_rlvr_v2.py + reforzar_rlvr_v2.py, sanear_dataset_rlvr_final.py (pad/ChatML sanitizer).
• Auditors — auditar_dataset_gold_sota.py, auditar_dataset_rlvr_profundo.py: schema, tags, pad, math-error and length audits. Run an auditor on every dataset before it touches a trainer; two of this project's three training failures were dataset failures.

9.4 The rest of the toolbox (short list)

Tool	Use
pipeline/07_verificacion/verificar_descarga_repo.py	sha256-vs-lfs.oid verification for any HF repo (catches XET/Range corruption)
pipeline/02_tabla_ple/cargar_tabla_ple.py	mandatory PLE loader for every pre-merge checkpoint (row-norm gate)
pipeline/02_tabla_ple/rebuild_tabla_v62.py	vectorized n-gram table rebuild, 30 min end-to-end
pipeline/02_tabla_ple/auditoria_ple_forense.py	disjoint-corpus hit-rate auditor (the only honest table metric)
pipeline/05_serving_eval/parchear_sglang_qwen38.py	idempotent SGLang patcher for qwen4_exp on Blackwell
pipeline/05_serving_eval/servir_sglang_rtx6000.sh	validated serving recipe (CUDA-13 fix, mem-fractions, alloc conf)
pipeline/05_serving_eval/audit_proxy_logger.py	model-agnostic CoT forensics proxy (TTFT/TPOT/thinking capture)
pipeline/06_quant_gguf/gguf_apex_helpers.py + lanzar_gguf_apex_pod.sh	imatrix calibration text, per-tensor quant map, full GGUF pod cycle
pipeline/06_quant_gguf/sota_suite/	turnkey LightEval runners + zero-GPU cache replay rescoring (04_rescore_control_27b_local.sh)
pipeline/04_entrenamiento/sonda_amplia_v6.py	81-probe clinical battery with verifier-judged probes and --only/--limit
pipeline/04_entrenamiento/evaluar_mtp.py + manifiesto_mtp.py	hand-reconstructed MTP head evaluation + sha256 preservation manifest
harness/mini_cookbook.yaml	agent-harness config that un-loops a fragile model (command discipline + finish-token protocol + DRY window sizing)
docs/chat_template_calibrado_5tiers.jinja	the calibrated 5-tier reasoning template (off/low/mid/high/xhigh)
pipeline/08_analisis/analizar_activaciones.py + docs/ACTIVATION-ARTIFACTS-GUIDE.md	3-minute CPU deep-analysis over the published signal NPZs — every expert-selection number recomputable without loading any model; the guide covers byte-verified schemas, the capture method, and DIY analyses (mesa, stratum ablation, cross-teacher diffs)

---

10. Artifact map

Todo el proyecto vive en UN repo: Davd-b01/qwen3.8-flash-next-40b-prune-research (~296 GB, this repository). Map of what to download depending on what you need:

Quieres…	Descarga (allow_patterns)	Tamaño
Servir el champion (S6-K128-P4)	*.safetensors, *.json, *.jinja (raíz)	81 GB
Warm-start de la v2 sobre K128	foundations/S6-K128-P0/* + adapters/*	82 GB
La base Pareto sin entrenar	foundations/S6-K192-P0/*	108 GB
La tabla factual standalone	ple-table/*	10 GB
Reproducir las máscaras/análisis	activation-artifacts/*	0.4 GB
El código y la bitácora	code/*, docs/*, benchmarks/*	0.1 GB

Sub-estructura: adapters/ (P2/P3/P4 + MANIFEST), ple-table/ (PLE4), foundations/ (S6-K128-P0, S6-K192-P0), activation-artifacts/ (señales + masks/ + analysis/), code/ (01–08 + train + docker + harness), docs/ (bitácora + guías), benchmarks/, lineage/ (MANIFEST of the removed lineage + cartridges).

Removed lineage, documented: the 10 old models and catastrophic failures (S1, S3d, P1, heal fallido, clean×2, PLE1/PLE3/STALE, GGUF APEX) no se conservan como pesos — cada uno está registrado en lineage/MANIFEST.md con qué era, qué falsified and in which section of this paper its lesson lives. Every absorbed file's hash was verified against its origin before any deletion.

Datasets (sibling dataset repos): thinking-cap-tier-raw-traces (38,158 trazas), thinking-cap-tier-curricula-complete (13,477 + 3,187), thinking-cap-tier-lima-dense (5,500 + 2,000).

Local lab notebook: docs/bitacora/ (§§37–102, in Spanish — the day-by-day record every number in this README traces back to).

11. Reproduction

# 0. environment: python ≥3.11, torch cu128, transformers==5.17.0, 'huggingface_hub==1.23.0'
# 1. verify any download against LFS OIDs (never trust Range reads on XET-backed files)
python pipeline/07_verificacion/verificar_descarga_repo.py Davd-b01/qwen3.8-flash-next-40b-prune-research --n 8

# 2. load + PLE patch (MANDATORY for any pre-merge checkpoint)
model = AutoModelForCausalLM.from_pretrained(..., dtype="auto")
#    patch: pipeline/02_tabla_ple/cargar_tabla_ple.py — row-norm gate 0.25 (noise) → ~0.10 (real)
cargar_tabla_ple(model, repo_id)

# 3. serve (SGLang; BF16 only)
bash pipeline/05_serving_eval/servir_sglang_rtx6000.sh /workspace/qwen38_rlvr_final 30000

# 4. masks / surgery / training — each stage is a standalone script with its own asserts
pipeline/01_poda_cirugia/    # signals → masks → surgery → C1-C6
pipeline/02_tabla_ple/       # table rebuild + forensic hit-rate
pipeline/03_datos_curadores/ # factory curators (SFT / SimPO / RLVR datasets)
pipeline/04_entrenamiento/   # trainers + pod launchers
pipeline/06_quant_gguf/      # GGUF APEX (documented negative result)

Serving flags that matter (llama.cpp ≥0.4.1 with qwen4exp, for the quant path): DRY --dry-multiplier 0.8 --dry-base 1.75 --dry-allowed-length 2 --dry-penalty-last-n 4096 (window must cover the agent turn history); MTP and vision are not consumed by llama.cpp yet.

Training & compute accounting (environmental impact)

The champion is not one training run — it is the tip of a cascade of five training stages across three generations, plus a discarded first lineage. The full accounting:

Every post-training stage that produced S6-K128-P4 (all on 1× H100 NVL 94 GB, warm-started in cascade — each stage loads the previous stage's adapters):

Stage	Generation	Steps	Documented time
SFT (base curriculum + 326 RFT anchors)	P2	120	~2.5 h (same dose as the P1 run: 2 h 35 m at 58.4 tok/s)
SimPO v6	P2	40	~40 min
mini-SFT (boxed + XML fix)	P3	15	minutes
SimPO v6.1 (iso-density pairs)	P3	25	~30 min wall incl. clinical probes
RLVR (offline, curated dataset)	P4	50	12.8 min
Champion cascade total	P2→P4	250	≈ 15 GPU-h pod wall-clock

The reconciliation between the step times and that total is the real cost of training an 81 GB model on a rented pod: the ~4.5 h of pure optimization steps sit inside ~15 h of pod wall-clock — the rest is 81 GB checkpoint downloads and loads per run, phase saves, clinical probes between phases, and the dry-runs that de-risked each stage. Counting every discarded attempt too — the entire P1 lineage (the elite production run, SFT 200 + SimPO 60, 9.6 h billed, plus the v5-gold rerun of SFT 120 + SimPO 40; both later invalidated) plus the aborted relaunches — all training combined cost ≈ 25–30 GPU-hours. The 236 MB adapters at every stage are published, so anyone can resume from any rung instead of re-paying the ladder.

Where the rest of the compute went (the honest part): the Best-of-3 data factory (38,158 traces on 4× L40S, including the mid-tier purge-and-redo) ≈ 30–50 GPU-h; measurement and evaluation (teacher forwards on H200, N3 profiling, the LightEval suites, the 4,072-trace audit, the dense-27B control) ≈ 15–20 GPU-h; surgery + verification pods ≈ 3–5 GPU-h; the GGUF APEX cycle ≈ 2 GPU-h. All-in cost: ≈ $300. Grand total on the order of 70–110 GPU-hours, over half of it spent on knowing what to do rather than training — which is the thesis of §8 in cost form.

Hardware: 1× H200 (141 GB), 1× H100 NVL, 1× RTX PRO 6000 (96 GB), 4× L40S, short-lived A100/4090 pods — all hourly rental (RunPod).

Carbon: roughly 50–80 kg CO₂e at typical grid intensities for the above — a rough estimate; no region-specific telemetry was recorded (a known reporting gap).

Citation

If you use the pruning methodology, the activation-ledger artifacts, or the models, cite:

@misc{oscar2026qwen38prune,
  title  = {Surgical Pruning of Qwen3.8-Flash-Next: 512 to 128 Experts with Full Post-Training},
  author = {Oscar (Davd-b01)},
  year   = {2026},
  url    = {https://huggingface.co/Davd-b01/qwen3.8-flash-next-40b-prune-research},
  note   = {Lineage code S6-K128-P4; expert selection driven by published activation ledgers}
}

Support this work

This entire project — nine days, ~70–110 GPU-hours, a dozen rented pods — cost about $300. If the artifacts, the methods or the honest negative results are useful to you, or you would like to fund a v2 (the K=192 mask, a clean heal over a verified table, a fuller post-training budget — the headroom is documented in §2.3 and §8.4), you can buy me a Ko-fi — every coffee goes straight into pod hours.

And if this repo is what gets you into the world of LLM fine-tuning and inference: you can use my RunPod referral link — https://runpod.io?ref=ssakdva8 — it supports the project at no extra cost to you, and everything needed to replicate this exact pipeline is in §9 (Toolbox).

Contact

Hugging Face: Davd-b01 · errors, forensics and corrections welcome — the error ledger (§5) is a living document.

---

Appendix: the math that runs this model (one page)

Everything below is stated or used somewhere in §1–§4; collected here so the paper is self-contained.

PLE factual injection (layer 2). For each of the 16 heads, a deterministic 64-bit mix-hash of the trailing bigram/trigram:

$${\text{hash}}_h=({\sum }_j{t}_{k-j}{m}_j+{\text{offset}}_h)N_h\backslash text\{hash\}\_h\; =\; \backslash Big(\backslash textstyle\backslash sum\_\{j\}\; t\_\{k-j\}\backslash ,m\_j\; +\; \backslash text\{offset\}\_h\backslash Big)\; \backslash bmod\; N\_h$$

The gathered rows $V \in \mathbb{R}^{5120}$ pass through a temporal conv and gated injection:

$$\tilde{V}=\mathrm{R}\mathrm{M}\mathrm{S}\mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}({\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}1\mathrm{D}}_{k=4}(V)),\mathrm{\Delta }h=(\mathrm{S}\mathrm{i}\mathrm{L}\mathrm{U}({\mathrm{R}\mathrm{M}\mathrm{S}\mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}}_q(x))\odot W_k\tilde{V})\odot (W_v\tilde{V}),x_{l+1}=x_l+\mathrm{\Delta }h\backslash tilde\{V\}\; =\; \backslash mathrm\{RMSNorm\}(\backslash mathrm\{Conv1D\}\_\{k=4\}(V)),\; \backslash quad\; \backslash Delta\; h\; =\; \backslash big(\backslash mathrm\{SiLU\}(\backslash mathrm\{RMSNorm\}\_q(x))\; \backslash odot\; W\_k\backslash tilde\{V\}\backslash big)\; \backslash odot\; (W\_v\backslash tilde\{V\}),\; \backslash quad\; x\_\{l+1\}\; =\; x\_l\; +\; \backslash Delta\; h$$

The injection ratio $\rho = |\Delta h| / |h_{resid}|$ measured 0.658 pre-heal (target 0.25); it is governed by value_proj (the gate is a bounded sigmoid), which is why norm-based recalibration failed — see the artifacts guide's "ideas" for the open question.

Hyper-Connections (4 streams). $H_{l+1}^{(i)} = \sum_j A_r^{(i,j)} H_l^{(j)} + B_i T(h_0)$, with $h_0 = \sum_i A_m^{(i)} H_l^{(i)}$ and $A_r$ projected to doubly-stochastic by Sinkhorn–Knopp ⇒ $\rho(A_r) = 1$ and $|\prod_l A_r|_2 \le 1$ (Riesz–Thorin) — no residual explosion across 48 layers.

Routed expert call. $y = F_{shared}(x) + \sum_{j \in \text{top-}10} \tilde{w}j F_j(x)$, $\tilde{w} = \mathrm{softmax}(x W{gate}^{\top}/\tau)$ over the top-10 only ($\tau = 1.5$ post-prune). Truncating 512 → 128 experts shrinks the softmax's partition function ≈4× — the "router truncation" wound of §2.5.

SimPO. $$\mathcal{L}=-\log \sigma (\frac{\beta }{\mathrm{\mid }{y}_w\mathrm{\mid }}{\sum }_t\log {\pi }_{\theta }(y_{w,t})-\frac{\beta }{\mathrm{\mid }{y}_l\mathrm{\mid }}{\sum }_t\log {\pi }_{\theta }(y_{l,t})-\gamma )\backslash mathcal\{L\}\; =\; -\backslash log\; \backslash sigma\; \backslash Big(\; \backslash tfrac\{\backslash beta\}\{|y\_w|\}\; \backslash textstyle\backslash sum\_t\; \backslash log\; \backslash pi\_\backslash theta(y\_\{w,t\})\; -\; \backslash tfrac\{\backslash beta\}\{|y\_l|\}\; \backslash textstyle\backslash sum\_t\; \backslash log\; \backslash pi\_\backslash theta(y\_\{l,t\})\; -\; \backslash gamma\; \backslash Big)$$ with β = 2.0, γ: 0.5 → 0.3 → 0.1 across generations. The 1/|y| normalization is both the anti-verbosity tool and (with mis-curated pairs) the anti-learning trap — §4.3.

RLVR failure gradient. For a failed rollout $y_l$ against gold $y_w$: $\nabla_\theta \mathcal{L} \propto -(1-\sigma(M)) \tfrac{\beta}{|y_l|} \nabla_\theta \sum_t \log \pi_\theta(y_{l,t})$ — explicit negative gradient on the model's own failure, plus anchor rehearsal CE on 28.8–30% of each set against forgetting.

Why arithmetic needs the scratchpad (TC⁰). A fixed-depth forward pass is uniform TC⁰; column multiplication's carry recursion — column $i$: $S_i = A_i B_j + C_{i-1}$, digit $= S_i \bmod 10$, $C_i = \lfloor S_i/10 \rfloor$ — needs depth growing with the operands. Each emitted intermediate token multiplies the effective depth ($L_{eff} = L \times K$), which is why show-your-work formats work and why the prune (which removed spare capacity) made mental arithmetic worse while RLVR-ified written carries recovered it.