ArzLM-300M-Base

303,353,856-parameter decoder-only base model trained from scratch on 6,000,214,016 tokens.

Not instruction tuned. Not a chat model.

Code and loader: github.com/Entourage397/ArzLM

Do not use Transformers AutoModel

This repository is not a stock LlamaForCausalLM export. Tensors are LitGPT / ArzLM (lm_head.weight, fused attn.qkv, QK-Norm). This will not load correctly:

AutoModelForCausalLM.from_pretrained("Kymaris/ArzLM-300M-Base")  # unsupported

Supported loader (from the GitHub repo). After hf download or by Hub id:

from arzlm.infer import generate, load_inference_checkpoint

loaded = load_inference_checkpoint("Kymaris/ArzLM-300M-Base", device="cuda", dtype="bf16")
print(loaded.parameters, loaded.config.block_size)  # 303353856, 2048
print(generate(loaded, "The capital of France is", max_new_tokens=32)["completion"])

Or a local snapshot:

loaded = load_inference_checkpoint("./ArzLM-300M-Base", device="cuda", dtype="bf16")

Download:

hf download Kymaris/ArzLM-300M-Base --local-dir ./ArzLM-300M-Base
python -m arzlm verify-checkpoint --checkpoint ./ArzLM-300M-Base/litgpt

Canonical inference weights: litgpt/lit_model.pth (579 MiB, no optimizer). model.safetensors (642 MiB) is the same LitGPT tensors.

Model

Parameters	303,353,856
Context	2048
Vocab	32,000 (arzlm-stem-32k-v2)
Layers / width	24 / 1024
Attention	16 Q heads, 4 KV groups, head_dim 64, QK-Norm
MLP	SwiGLU, intermediate 2816
RoPE	full, theta 10000
Embeddings	tied

Training

• 6,000,214,016 tokens, 45,778 optimizer steps, batch 64×2048
• Mixture 55% general / 20% math / 15% code / 10% science
• FineWeb-Edu sample-10BT (safe shard) + Cosmopedia-v2, FineMath-4+, Stack-Edu/SWH permissive, peS2o v2
• Muon hybrid + AdamW, cosine, warmup 800 steps, seed 42
• 1× NVIDIA A100-SXM4-40GB, PyTorch 2.11.0+cu128, LitGPT 0.5.13, BF16, torch.compile

Validation (training packed val, not WikiText)

• loss 2.6909
• perplexity ~14.75
• by domain: general 3.300, math 1.800, code 1.472, science 2.955

Intended use

Research, tiny-LM experimentation, architecture / optimizer / data-mixture studies, educational reproduction of a 300M pretrain.

Limitations

300M base model, 2048 context, 6B tokens. Factually unreliable, weak math, repetition, template-like drift, incorrect code. Not aligned for assistants.

License

Apache-2.0 for these weights and the training code, with dataset attribution in the GitHub NOTICE. See that file for ODC-By / Common Crawl / Software Heritage notes.