SLM-300M

A 304M-parameter decoder-only language model trained from scratch on 6.08B tokens of FineWeb-Edu, the Chinchilla-optimal budget for its size, on a single A100. Everything here was built from scratch: the tokenizer, model, data pipeline, Muon optimizer and the fault-tolerant training system. Code: github.com/Darpan-Maurya/SLM.

On zero-shot benchmarks it beats OPT-350M and BLOOM-560M and comes within a point of Pythia-410M on average, with 30-60x less training data, and leads all of them on ARC-Easy.

This is a base model: it continues text and has not been instruction-tuned or safety-tuned.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "Drm39/slm-300m"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.bfloat16)

inputs = tokenizer("Photosynthesis is the process by which", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=80, do_sample=True,
                     temperature=0.7, top_p=0.9, repetition_penalty=1.1)
print(tokenizer.decode(out[0], skip_special_tokens=True))

trust_remote_code=True loads modeling_slm.py from this repository: the architecture (per-head QK-norm together with NoPE layers) is not one that transformers ships. That file reproduces the training implementation exactly: logits are bit-identical, and greedy decoding agrees token for token with and without the KV cache and in left-padded batches. Tested with transformers 5.18.

Model

Parameters	304.1M (270.6M non-embedding), embeddings tied
Layers / width	24 layers, d_model 1024, SwiGLU FFN 2816
Attention	16 query heads, 4 KV heads (GQA), head dim 64, RMSNorm on queries and keys
Positions	RoPE (theta 10,000) in 3 of every 4 layers; every 4th layer is NoPE
Normalisation	pre-norm RMSNorm, statistics in fp32
Context	2,048 tokens, trained with intra-document attention masking
Tokenizer	byte-level BPE trained on FineWeb-Edu, 32,768 tokens, GPT-4 split pattern

Training

Data	FineWeb-Edu sample-10BT, files 000-007: 5.45B tokens (about 1.1 epochs; the second pass is reshuffled)
Tokens	6.08B = 11,596 steps x 524,288 tokens (256 sequences of 2,048)
Optimiser	Muon (lr 0.02, momentum 0.95) on the 96 hidden matrices; AdamW (lr 6e-4, betas 0.9/0.95, wd 0.1) on embeddings and gains
Schedule	warmup-stable-decay: 1,000 warmup steps, constant, then a 1-sqrt decay over the last 20% to 10% of peak
Regularisation	z-loss 1e-4, gradient clipping 1.0, no dropout
Precision	bf16 autocast, fp32 master weights, torch.compile, FlexAttention document masks
Hardware	one A100 40GB on Google Colab, about 26 GPU-hours; 62-69k tokens/s at 48-54% MFU
Final validation	loss 2.86 nats/token, 0.897 bits per byte

Checkpoints went off the machine every ~10 minutes, to two Kaggle datasets that take turns so one complete copy always survives a failed upload; resumes are exact (data order, RNG, optimiser state, schedule) and every file is sha256-verified before it is loaded.

Evaluation

Zero-shot, scored by the log-likelihood of each answer continuation on the full evaluation sets (10,042 HellaSwag, 2,376 ARC-Easy, 1,172 ARC-Challenge, 1,838 PIQA, 1,267 WinoGrande, 500 OpenBookQA). The metric per task follows the usual lm-eval convention (acc or length-normalised acc_norm).

Model	Training tokens	ARC-E	ARC-C	PIQA	HellaSwag	OBQA	WinoGrande	Avg
SLM-300M	6B	56.6	27.5	65.2	36.5	31.6	51.7	44.9
OPT-350M	180B	41.9	25.7	64.8	36.2	33.3	52.4	42.4
BLOOM-560M	366B	43.7	27.5	65.1	36.5	32.6	52.2	42.9
Pythia-410M	300B	47.1	30.3	67.2	40.1	36.2	53.4	45.7
MobileLLM-350M	1T	53.8	33.5	68.6	49.6	40.0	57.6	50.5

LAMBADA (OpenAI version, exact greedy last word): 32.3%.

Baseline numbers are from Table 3 of the MobileLLM paper; they come from a different evaluation setup, so treat differences of a couple of points as noise. The pattern is expected from the data: an educational corpus helps knowledge questions (ARC) most and everyday narrative completion (HellaSwag) least, where sheer token count matters more.

Limitations

• A base model: it completes text and does not follow instructions.
• Small and briefly trained: fluent and on-topic, but often factually wrong, for example "Plants do not need sunlight to grow."
• English only; context limited to 2,048 tokens.
• Trained on filtered web text and not safety-tuned, so it can reproduce biases in that data.

Citation

@misc{maurya2026slm300m,
  title  = {SLM-300M: a small language model trained from scratch},
  author = {Darpan Maurya},
  year   = {2026},
  url    = {https://huggingface.co/Drm39/slm-300m}
}

Built on FineWeb-Edu (Penedo et al., 2024) and the Muon optimiser (Jordan et al., 2024).