YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
RetNet-0.5B — FAILED training checkpoint
⚠️ DO NOT USE THESE WEIGHTS
01_v21_export/is a failed training run, published only so the failure stays auditable. It loads, but it is not a usable model. If you want a working model, useAyaka157/RetNet-0.5B.
| Folder | Produced by | Why it fails |
|---|---|---|
01_v21_export/ |
the v21 DDP training run | It inherits 338,120,704 dead parameters from the width-growth step, and msr.gamma was left as a bare parameter under weight_decay=0.1 because --no_decay_1d defaulted off and the v21 config never set the field. Median gamma over 440 heads fell to 0.5656, with 10 heads frozen at 0.213622 through pure decay with zero gradient. |
What is deliberately NOT in this repository
Publishing only training-produced checkpoints is a deliberate policy here, so two other defective artifacts from the same lineage are not published:
probe_500m— its weights are random. Its ownconfig.json_notereads: "probe checkpoint: random weights, only for VRAM and step-time measurement, not usable for evaluation." It is not a trained model and does not belong in a model repository.retnet_500m— the 272M -> 500M width-growth product. It has notrainer_state.jsonbecause no training produced it; it is the output ofscripts/grow_500m.py. This is the step where the dead parameters were actually created.
The mechanism, because it explains both failures: widening a checkpoint sets the new input
columns to zero so the wider model computes exactly what the narrower one did. But if the
new gate gamma_new is also initialised to zero, then dL/dW_newcol = upstream x 0 = 0,
and since the new inputs are themselves zero, gamma_new receives no gradient either. The
new capacity is permanently inert. Initialising gamma_new = 1.0 would have been equally
function-preserving at step 0 and would have let the channels wake up. That counterfactual
is derived, never tested — see the main repository's docs/00_错误总表.md (E01–E03, E07)
and docs/00F1_修复对照.md.
Verification
config.json ships alongside the weights so the architecture travels with them. SHA-256 of
model.safetensors is recorded in MANIFEST.sha256. These are the exact bytes on disk when
the failure was audited; nothing was repaired before publication.
Licence
MIT, same as the parent project. Provided strictly as a record of failure, with no warranty and no fitness for any purpose.