Drew168/azmx-one-v1

AZMX One v1 (in-training checkpoint)

1.87B dense base model for India — English + 22 scheduled Indian languages, custom 128K Indic tokenizer. Apache-2.0. Training is in progress; this repo tracks the live run.

Progress

Step	33,040 / 457,763
Tokens	130.15B / 1,800B (7.23%)
Global batch	3.94M tokens/step (SEQ 2048, MICRO 16, ACCUM 30)
Schedule	WSD — warmup 6,866 / stable to 389,098 / cooldown to 457,763
Precision	bf16 + Liger fused kernels, 4x H200

Files

• latest.pt — full torch checkpoint (model + optimizer + step) for resuming training.
• model.safetensors + config.json — weights-only HF export for inference, from AZMX-One-milestone-13. Note this export lags latest.pt slightly (milestones are cut every 10B tokens).
• Tokenizer: Drew168/azmx-6t-data/azmx_tokenizer_v2.json

Architecture

AZMX (Qwen3-subclass): vocab 128,000 - hidden 2,048 - 32 layers - GQA 16/8 - head_dim 128 - SwiGLU (intermediate 6,144) - RoPE theta 10,000 - tied embeddings - seq len 2,048.

Data

Drew168/azmx-6t-data — Indic (Sangraha, CulturaX, IndicCorpV2, books, news) + English web + code + math. Mix shifted toward Indic partway through the run, which raises raw training loss (Indic text carries more entropy per token than the code-heavy earlier mix); compare bits-per-byte on held-out data, not loss across different mixtures.

Resume

Build AZMXForCausalLM from config.json, load latest.pt, continue from its step against the dataset shards. Target 1.8T tokens.