algomaster99/pretrain-2.2e18

Pretrain run pretrain_2.2e18_bf16_qknorm (see src/model/train.py).

Summary

A decoder-only GPT-style causal language model, pretrained from scratch (no pretrained-weight initialization) on Spanish, Portuguese, and Hindi text. Model shape (172M params) and training token budget were sized to a 2.2e18 FLOPs compute budget via an IsoFLOP scaling-law fit (scripts/fit_isoflop_scaling_law.py). Trained in bf16 autocast with QK-LayerNorm bounding the attention logits -- earlier bf16 attempts without it diverged from unbounded attention-logit growth (see scripts/slurm/11_pretrain_2.2e18_qknorm.sh).

Data

Pretrained on andre15silva/pretrain-pt-es-hi, a cross-source MinHash-deduped corpus built from:

• Spanish, Portuguese: HuggingFaceFW/fineweb-2 (es/pt)
• Hindi: HuggingFaceFW/fineweb-2 (hi) + ai4bharat/sangraha's "verified" split (human-verified sites, OCR'd PDFs, transcribed speech -- the one Hindi source that isn't itself another Common-Crawl derivative)

No language/quality filtering beyond that dedup is applied -- sources are used as-is (see the repo's own README for the full pipeline). Tokenized with a shared multilingual byte-level BPE tokenizer (vocab_size 32000) trained on the same corpus.

Model config

GPTConfig(vocab_size=32000, block_size=1024, n_layer=11, n_head=16, n_embd=1024, dropout=0.0, bias=True, qk_norm=True)

Final training state

• iter: 259653
• final_train_loss: 3.1813
• final_val_loss: 2.6673

Files

• ckpt_final.pt -- final trained weights (model_state_dict, model_cfg, iter_num, final_train_loss, final_val_loss)
• ckpt.pt -- best-val-loss checkpoint seen during training
• ckpt_iter*.pt -- periodic snapshots (no optimizer state)

ckpt_last.pt (optimizer state, for resuming training) is intentionally not published here -- it's redundant with the files above for anyone just loading weights, and adds ~2GB per snapshot for state nobody outside the original training job needs.

Load with torch.load(path, weights_only=False) and src.model.gpt.GPTConfig/GPT.