sai-gpt-from-scratch

A 163M-parameter decoder-only Transformer (GPT-2 style), implemented in PyTorch and trained end to end on a single consumer GPU (RTX 5060 Laptop): pretraining on ~3B tokens of FineWeb-Edu, then supervised instruction fine-tuning on ~877k examples.

This is a learning and engineering project, not a production model. It writes fluent text but is often factually wrong (see Limitations).

Model details

Architecture	Decoder-only Transformer, pre-LayerNorm, learned absolute positions
Parameters	163,009,536
Layers / heads / width	12 / 12 / 768
Context length	1024
Vocabulary	50,257 (GPT-2 BPE via tiktoken)
Weight tying	No (token embedding and output head are separate, which is why this is 163M and not 124M)
Precision	bf16 autocast, fp32 loss

The model is custom PyTorch code, so AutoModel.from_pretrained will not work. Load the weights into the GPTModel class from my_own_llm.ipynb (see Usage).

Training

Pretraining

• Data: FineWeb-Edu, 3.003B GPT-2 BPE tokens (3.000B train / 3M held-out validation), stored as a memory-mapped uint16 stream.
• Optimizer: AdamW, peak LR 4e-4 decaying to 4e-5 (cosine), 200 warm-up updates, weight decay 0.1, gradient clipping 1.0.
• Batching: micro-batch 2 x 1024 tokens x 16 gradient-accumulation steps = 32,768 tokens per update, 60,000 updates in the final phase.
• Run across several resumed sessions using checkpoints.

Instruction fine-tuning (SFT)

• Data: ~1.03M instruction/response pairs from conversation data, split 877,086 train / 51,594 validation / 103,186 test.
• Prompt format: Alpaca-style (below). Padding positions are masked with -100.
• Optimizer: AdamW, LR 5e-5, weight decay 0.1, batch size 1, gradient clipping 1.0.
• Run in three resumed phases (152,500 + 160,937 + 563,649 examples), roughly one epoch.

Results

Stage	Metric	Value
Pretraining	Loss at last evaluation (update 60,000)	train 3.284, validation 3.318 (perplexity about 27.6, estimated on 50 validation batches)
SFT	Validation cross-entropy / perplexity	2.037 / 7.67 (full validation set)
LLM-as-judge	Mean score on 100 test examples	37.6 / 100 (std 39.1)

The judge was Llama 3.1 8B (via Ollama, greedy decoding), scoring each response against the reference from 0 to 100. The large spread means many near-0 and many high scores. Treat it as a rough signal, not a benchmark. No standard benchmarks (HellaSwag, ARC, MMLU) have been run yet.

Before SFT, the model ignores the prompt format and loops, repeating its answer and the ### Response: marker. After SFT it produces fluent, on-format answers.

Usage

Prompt format:

Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
{your instruction}

### Response:

Loading and generating (run the class and helper cells from my_own_llm.ipynb first: GPTModel, generate, format_input, text_to_token_ids):

import torch, tiktoken
from huggingface_hub import hf_hub_download

GPT_CONFIG_163M = {
    "vocab_size": 50257, "context_length": 1024, "emb_dim": 768,
    "n_heads": 12, "n_layers": 12, "drop_rate": 0.0, "qkv_bias": False,
}

path = hf_hub_download("SAISIDD/sai-gpt-from-scratch", "my-finetuned-gpt.pth")
model = GPTModel(GPT_CONFIG_163M)
model.load_state_dict(torch.load(path, map_location="cpu", weights_only=True))
model.eval()

tokenizer = tiktoken.get_encoding("gpt2")
prompt = format_input({"instruction": "Explain what a Python dictionary is.", "input": ""}) \
         + "\n\n### Response:\n"
ids = text_to_token_ids(prompt, tokenizer)
out = generate(model, ids, max_new_tokens=150,
               context_size=1024, temperature=0.7, top_k=40, eos_id=50256)
print(tokenizer.decode(out[0, ids.shape[1]:].tolist()))

Files

• my-finetuned-gpt.pth: final instruction-tuned weights (state dict, about 650 MB)
• my_own_llm.ipynb: full project notebook (model, data pipeline, pretraining, SFT, evaluation, chat widget)

Limitations

• Small model, small data. 163M parameters and a few billion tokens is far below what reliable knowledge needs. Answers are fluent but often wrong, and longer generations can become repetitive. Do not rely on it for facts.
• SFT loss includes prompt tokens. Only padding is masked, not the instruction part, so the reported SFT loss and perplexity cover the whole sequence and are not comparable to response-only numbers.
• Contiguous split. The SFT train/validation/test split follows file order and was not shuffled by source, so source mix may differ between splits.
• "One epoch" is approximate. SFT was interrupted and resumed on re-shuffled subsets, so examples were not partitioned exactly once.
• Thin evaluation. Single run, single seed, no confidence intervals, no standard benchmarks, and a small judge model on 100 examples.
• Naming. Some file and notebook names say "124M" (GPT-2 small). The correct figure for this untied model is 163M.

Acknowledgements

• Architecture and overall training structure follow Sebastian Raschka's Build a Large Language Model (From Scratch) (Manning, 2024). This project scales it up and adds a memory-mapped data pipeline, gradient accumulation, cosine LR schedule, bf16 training, resumable checkpoints, large-scale SFT, and perplexity and LLM-judge evaluation.
• Pretraining data: FineWeb-Edu.
• Instruction data: SmolTalk (based on the source tags in the data).
• Tokenizer: GPT-2 BPE via tiktoken.
• Judge model: Llama 3.1 8B, served locally with Ollama.

License

Apache-2.0 for the code and weights in this repository. The training datasets keep their own licenses.