Full-duration CE versus Brier study
Exploratory paired nanoGPT experiments. Five main seeds per loss. Learning rates are selected by final validation NLL after a full-duration sweep; test scores never select rates.
| Study | CE test NLL | Brier test NLL | Paired Brier minus CE |
|---|---|---|---|
| scale-tinystories | 0.55443 ± 0.00124 | 0.63605 ± 0.00225 | +0.08162 |
| v2-shakespeare | 1.56379 ± 0.01165 | 1.68114 ± 0.00924 | +0.11735 |
| v2-tinystories | 0.62480 ± 0.00209 | 0.73783 ± 0.00512 | +0.11303 |
Scores use dataset-specific token units: characters for Shakespeare, UTF-8 bytes for TinyStories. Do not compare absolute NLL across tokenizations. Bands are sample SD, not confidence intervals.
Mechanism measurements
Each final test record includes conditional CE/Brier logit-gradient norms, target-probability bins and confidently wrong predictions. These are analytic gradients of unreduced per-token scores on the same logits, checked against autograd. They describe score geometry, not parameter-gradient norms or a causal explanation of all training differences.
Reproduction and limits
See source/configs/research-plan.json and source/docs/RESEARCH_PLAN.md for the frozen protocol. Dataset preparation is pinned and checksummed. All validation histories, pilots, paired seeds and samples are included under runs/. Models are small; the TinyStories subsets are source-order prefixes. Five seeds and a small LR grid do not establish universal superiority.
Source-order data, byte tokenization, fixed final-checkpoint selection and potential Shakespeare overfitting limit generalization. Baseline v1 used shorter pilots and is kept separate.
Artifacts
All source S3 artifacts were checked against their recorded SHA-256 before extraction. See S3_MANIFEST.json and FILES_SHA256.json. Native PT checkpoints and safer tensor-only safetensors exports are both included. This is custom nanoGPT code, not an AutoModel checkpoint.
Loading weights
Use source/experiments/sample.py with a trusted native PT checkpoint. Each model also has a tensor-only model.safetensors file and config.json; load it with safetensors.torch.load_file into the vendored GPT class. Tokenizer metadata is alongside the weights. No remote custom code is automatically executed.