Mini Qwen-Next 31B pretraining
Training starts with one complete pass over Ba2han/tokenized_mix_0507 and stops at its exhaustion.
The learning-rate schedule spans 31 billion input tokens; the packed long dataset is a later continuation.
Two GPUs use 1 packed row each per forward and 32 accumulation passes: 655,360 input tokens per update.
Activation checkpointing is disabled. N-gram gradients accumulate in FP32 and communicate in BF16.
The full 1.8B n-gram table remains on GPU; measured steady throughput is about 41k input tokens/s.
Adam groups use AdamW8bit at a peak LR of 0.001; Muon groups use the selected 8-step Polar Express recipe at 0.024.
Warmup lasts 500 updates. Linear decay starts at 24.8 billion input tokens.
Full resumable checkpoints are uploaded every 1 billion input tokens and on corpus exhaustion.
The repository retains the latest two checkpoint directories; latest points to the last complete upload.
Each includes backbone weights, both optimizer states, n-gram weights/moments, per-rank RNG and the data cursor.
Use distributed_training.py --resume auto to restore the same corpus, or --reset-data-loader with a newly packed corpus to continue at the preserved global step.
This custom architecture requires the included source and tokenizer; these are not standard Transformers AutoModel checkpoints.