TR-HASH MoE 100M - Corrected 70B Agentic Refinement
Status: prepared, training not started yet. This repository is the destination for the corrected refinement run. No model checkpoint should be considered available until weights and an evaluated release manifest are published here.
This run restarts from the verified final/ weights of TR-HASH MoE 100M - 125B Agentic Pretraining. It loads model weights only and creates a fresh optimizer and learning-rate schedule.
Corrected optimizer-update contract
The complete update regime is matched to the released 200M refinement instead of copying only its learning rate.
| Setting | Corrected value |
|---|---|
| Source data | same audited 70B unique-token core |
| Replay | none |
| Parameters | full model |
| Tokens per optimizer update | 3,932,160 |
| Planned optimizer updates | 17,801 |
| Scheduled token exposure | 69,996,380,160 |
| Unused audited tail | 3,407,872 tokens |
| Optimizer | fresh AdamW |
| Peak learning rate | 1e-4 |
| Scheduler | cosine |
| Warmup | 500M tokens |
| Weight decay | 0.1 |
| Sequence length | 2,048 |
| Intended hardware | 8 x NVIDIA RTX 5090 |
| Batch per GPU | 16 sequences |
| Gradient accumulation | 15 |
The Complexity framework now prints an OPTIMIZER UPDATE CONTRACT block before data loading and rejects a 100M refinement launch if LR, scheduler, warmup, weight decay, or tokens per update differ from this contract. Runtime metrics retain the historical step fields and also expose optimizer_update, optimizer_updates_planned, and tokens_per_optimizer_update.
Provenance note
The earlier 100M refinement archive is deprecated because it used 229,376 tokens per optimizer update, causing many more AdamW updates per source token. It remains public only for auditability and must not be used as the initialization point for this run.
License
Released under CC BY-NC 4.0. Source datasets retain their own licenses and terms.