Apex-3-59B-stable (raw pretraining state)
Apex-3 is a 21B-parameter mixture-of-experts language model (3.36B active per token; 30 layers, 64 experts top-4 plus a shared expert, SuperBPE vocabulary of 204,800) trained from scratch. This repository holds the raw training state at the end of the stable phase: 59.0B tokens, step 10886, DiLoCo sync 15 (3 workers), before the learning-rate decay. It is meant for continued pretraining and research on training dynamics, not for direct inference: it is not in the Transformers format and the model has had no instruction tuning.
step_00010886/ is the torch.distributed.checkpoint (offload layout: weights + optimizer state, trainer_state.json);
diloco/ the DiLoCo outer state. global_15.bin and momentum_15.bin (84 GB each) are split into 40 GB parts
(the Hub limit is 50 GB per file):
cat diloco/global_15.bin.part* > diloco/global_15.bin
cat diloco/momentum_15.bin.part* > diloco/momentum_15.bin
sha256sum -c SHA256SUMS # checks every ORIGINAL file (run in the repo root after joining)
Resume (see provenance.json): --resume step_00010886 --branch-from-diloco --branch-global diloco
Some training sources carry terms of their own (for example CC BY-SA for Wikipedia, ODC-By for FineWeb-Edu and peS2o); check them for derivatives.