MicroMe-125M-Base

The pretrained base language model behind MicroMe-125M — 125M parameters, 3B FineWeb-Edu tokens, trained from scratch on a single RTX 4060 laptop (8 GB) in ~40 hours with hand-written PyTorch and the Muon optimizer.

This is the raw language model (not chat-tuned). For the chat + RAG assistant, use Ankitgdes/microme-125m. Full project + training code: github.com/ankit-rawani/microme.

Benchmarks (0-shot, lm-evaluation-harness)

ARC-Easy	HellaSwag	PIQA	LAMBADA
52.2	31.9	62.2	25.5

Competitive with GPT-2-124M / Pythia-160m on ~10–100× less training data — ARC-Easy notably strong from FineWeb-Edu's educational text.

Files

File	Use
model.safetensors	bf16 weights — load with the bundled model.py, config PRESETS["micro_125m"]
base_training_state.pt	full checkpoint (weights + Muon/AdamW state + iter) — resume pretraining
sft_training_state.pt	the SmolTalk chat-SFT checkpoint (intermediate, resumable)

Load (base model, next-token generation)

import torch, importlib.util
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file

d = hf_hub_download("Ankitgdes/microme-125m-base", "model.py").rsplit("/", 1)[0]
spec = importlib.util.spec_from_file_location("mm", f"{d}/model.py")
mm = importlib.util.module_from_spec(spec); spec.loader.exec_module(mm)
model = mm.GPT(mm.PRESETS["micro_125m"]).eval()
model.load_state_dict(load_file(hf_hub_download("Ankitgdes/microme-125m-base", "model.safetensors")))

License

Apache-2.0. Trained on FineWeb-Edu (ODC-By).