nanogpt-100m

A 101M-parameter decoder-only GPT trained from scratch with cameronervin/nanogpt in three stages:

1. Base: pretraining on 2.0B FineWeb-Edu tokens.
2. SFT: fine-tuning on SmolTalk chat, GSM8K and synthetic addition.
3. GRPO: reinforcement learning with an exact numeric reward on 2-digit addition.

The whole pipeline cost about $26 of GPU time and storage. This is an educational model that demonstrates the training stack and the GRPO method. It is not a general assistant. The full write-up is docs/results.md.

Results

On 500 held-out 2-digit addition prompts:

Metric	SFT (release/sft)	GRPO (release/grpo)
Greedy exact match	32.0%	46.6%
Sampled pass@1 (T = 0.8)	9.3%	19.0%
Sampled pass@8	48.6%	63.8%
Format valid	100%	100%

The base model reaches validation loss 3.250 on FineWeb-Edu (perplexity 25.8, 1.019 bits per byte).

Files

Folder	Contents
release/base	Base model after pretraining
release/sft	SFT model, the GRPO starting point
release/grpo	GRPO model (recommended)
release/tokenizer	32,768-token byte-level BPE
release/eval_report.json	Full eval report with release gates
release/eval/	The pass@k check before GRPO and the learning-rate sweep
release/logs/	SFT and GRPO training logs

Each checkpoint folder holds model.safetensors, model_config.json and run_manifest.json. The manifest records the sha256 of the weights, the data, the config and the parent checkpoint.

Checkpoint	sha256 of model.safetensors
release/base	156d9cbbecde9177a06b78d663d6518584ad6fd3ae483330b6b8133c65db69e1
release/sft	979bfc16b8ec1da653e3c6e165b73b5a9940ecaa9b5af35ef5d860f0d24e39e3
release/grpo	fc9dd38bca2ea498631850b4fb4929ffd69fc1154b0ffa8dea28ff0aeb7d08c9

The other top-level folders hold checkpoints and logs from earlier experiments in the same run. The write-up describes them:

Folder	What it is
sft-v1, sft	SFT with bare-number answers
grpo	GRPO from sft; held-out accuracy fell from 30% to 10%
sft-carry, sft-carry3	Carry-format SFT on 2-digit and 2–3 digit sums (100% accurate, so no room for GRPO)
base, tokenizer	Same files as release/base and release/tokenizer
v1/, eval/, logs/, manifests/	Eval outputs, training logs and data manifests from all runs

Usage

The weights use this repo's own model code, not transformers.

pip install "git+https://github.com/cameronervin/nanogpt" "huggingface_hub<2"

import json

from huggingface_hub import snapshot_download
from safetensors.torch import load_model

from nanogpt.config import ModelConfig
from nanogpt.generate import sample_chat
from nanogpt.model.gpt import GPT
from nanogpt.tokenizer import load_tokenizer

path = snapshot_download(
    "cameronervin1/nanogpt-100m", allow_patterns=["release/grpo/*", "release/tokenizer/*"]
)
with open(f"{path}/release/grpo/model_config.json") as file:
    model = GPT(ModelConfig.from_mapping(json.load(file)))
load_model(model, f"{path}/release/grpo/model.safetensors")
model.eval()
tokenizer = load_tokenizer(f"{path}/release/tokenizer")

messages = [{"role": "user", "content": "What is 47 + 38? End with <final>answer</final>."}]
_, (completion,) = sample_chat(model, tokenizer, messages, num_samples=1, max_new_tokens=96)
print(tokenizer.decode(completion))

The model answers column by column. This is real output from the snippet above. Here it drops the carry in the tens column, which is typical of its errors at 46.6% accuracy:

ones: 7 + 8 = 1 5, write 5 carry 1
tens: 4 + 3 + 1 = 7, write 7
digits: 7 5
<final>75</final>

The correct answer is 85.

Training

Stage	Data	Steps	Hardware	Cost
Base	FineWeb-Edu sample-10BT, 2.0B tokens	7,630	1× H100 SXM	~$4.85
SFT	SmolTalk (200k), GSM8K (4.5k), carry-format sums (4k)	2,000	1× A40	~$1.50
GRPO	2-digit addition prompts, 8 samples each, exact reward	150	1× A40	$1.11

Architecture: 12 layers, width 768, grouped-query attention with 12 query heads and 4 KV heads, RoPE, SwiGLU, RMSNorm, QK-norm, tied embeddings, and a 1,024-token context.

GRPO settings: 64 prompts × 8 samples per step, learning rate 1e-6, KL weight 0.02 to the frozen SFT model, PPO clip 0.2, temperature 0.8.

Limitations

• The model was trained and evaluated on 2-digit addition only. Accuracy is 0% on longer numbers, likely because each multi-digit operand is one token.
• Chat and general text quality were not evaluated. Expect the limits of a 100M model trained on 2B tokens.
• GSM8K was part of the SFT data, but GSM8K accuracy was not measured.
• All results come from single runs.

License and data

The model code and these weights are released under Apache-2.0. The training data keeps its own terms:

• FineWeb-Edu: ODC-By;
• SmolTalk: Apache-2.0;
• GSM8K: MIT.

FineWeb-Edu is filtered open-web text and may contain copyrighted, harmful, biased or private content. The model can reproduce such content.