arihntk/gpt-from-scratch

gpt from scratch

Summary

A small GPT2 style language model with 11,139,072 parameters, implemented and trained from scratch in PyTorch. The tokenizer is a byte level BPE model with a 1024 token vocabulary trained on the same corpus. The weights are stored in safetensors format.

Status

This checkpoint was saved at step 750 of a 2500 step training plan. Validation loss at this point is 4.0174 and validation perplexity is 55.56.

Plots

Training and validation loss. The dashed line marks the best checkpoint.

Validation perplexity on a log scale. The same best checkpoint is marked.

Learning rate across the run: linear warmup for 100 steps, then cosine decay.

Gradient norm, with a 25 point rolling mean.

Throughput. The sharp dips are measurement windows that included an evaluation, not slower training.

Files

• model.safetensors: model weights in safetensors format
• config.json: model architecture settings
• metadata.json: checkpoint step, validation loss, and run details
• best.pt: PyTorch fallback checkpoint
• tok.json: BPE merge table
• config.py, model.py, tokenizer.py, generate.py: the code needed to run the model
• requirements.txt: Python packages for generation
• plots/: training figures shown above
• history.json and log.txt: evaluation history and the full training log
• LICENSE: MIT license
• README.md: this model card

Usage

Install the dependencies and generate text:

pip install -r requirements.txt
python generate.py --ckpt model.safetensors --data-dir . --prompt "ROMEO:" --max-new 300

The safetensors file is the primary format. If you prefer PyTorch, use the fallback:

python generate.py --ckpt best.pt --data-dir . --prompt "ROMEO:" --max-new 300

Pass --temperature and --top-k to control sampling.

Architecture

• 6 layers, 6 heads, 384 embedding width
• context length 256 tokens
• vocabulary 1024 tokens
• GELU activation and LayerNorm before each block
• tied input and output embeddings
• 11,139,072 parameters in float32

Training

• data: Tiny Shakespeare, 1,115,394 characters
• tokens: 399,354 train and 44,373 validation
• optimizer: AdamW with learning rate 0.001, warmup, and cosine decay
• batch: 32 sequences of 256 tokens
• planned steps: 2500

Hardware and training time

MacBook Air M4 with 16 GB unified memory. The full 2500 step run took 1412.7 s, about 23.5 minutes. The best checkpoint was reached at step 750.

The complete training log is in log.txt, the evaluation history is in history.json, and the code is in the source repository.

Evaluation data

Loss and perplexity at every evaluation step:

step	train loss	val loss	val perplexity
0	6.9762	6.9726	1067.02
50	5.3354	5.4735	238.29
100	4.4261	4.7035	110.34
150	4.2366	4.5510	94.72
200	4.1064	4.4764	87.92
250	3.9861	4.4158	82.75
300	3.8904	4.3265	75.68
350	3.7822	4.3172	74.98
400	3.6354	4.1879	65.88
450	3.5299	4.1555	63.78
500	3.3827	4.1277	62.03
550	3.2522	4.0629	58.14
600	3.1742	4.0752	58.86
650	3.0133	4.0296	56.24
700	2.9245	4.0407	56.86
750	2.7700	4.0174	55.56
800	2.6463	4.0603	57.99
850	2.5210	4.1157	61.29
900	2.4365	4.1489	63.36
950	2.2620	4.1803	65.39
1000	2.1707	4.2614	70.91
1050	2.0256	4.2995	73.66
1100	1.8903	4.3505	77.52
1150	1.7721	4.4653	86.95
1200	1.6428	4.5365	93.36
1250	1.5062	4.6036	99.84
1300	1.3698	4.6525	104.85
1350	1.2460	4.7526	115.89
1400	1.1566	4.8612	129.17
1450	1.0668	4.9519	141.44
1500	0.9616	5.0425	154.86
1550	0.8860	5.0922	162.74
1600	0.7766	5.1590	173.99
1650	0.7177	5.2373	188.15
1700	0.6459	5.3364	207.77
1750	0.6021	5.3818	217.42
1800	0.5461	5.4645	236.17
1850	0.5083	5.5058	246.11
1900	0.4538	5.6199	275.87
1950	0.4231	5.6271	277.86
2000	0.3904	5.7104	301.98
2050	0.3633	5.7174	304.12
2100	0.3419	5.7201	304.93
2150	0.3095	5.7976	329.49
2200	0.3073	5.8343	341.83
2250	0.2913	5.8768	356.66
2300	0.2779	5.8826	358.76
2350	0.2657	5.9157	370.83
2400	0.2585	5.9795	395.23
2450	0.2506	5.9247	374.16
2499	0.2399	6.0090	407.09

The raw values are in history.json and the complete run log is in log.txt.

Samples

Best checkpoint at step 750, temperature 0.8, top k 64, seed 7:

ROMEO: worthy father hath
That thrice which he spent to be made
To this barny bark dish revenge from the wife of hope,
Thereford thou hast not before thy joi'a mortal man.
But, Warwick, Clarence shall be the world;
Is it to him be at the Tower:
'Tis slight where he will, Taste he slain, which nothing more dyears
The night of those that he do it? Will it ty:
Henry Hold him honourable,
And marry Paris, sing me by Siam Paris;
And therefore he is not an Warwick messen?

PRINCE EDWARD:
A otunt a ma suitor
Either the benef and devil to him.

PRINCE EDWARD:

Best checkpoint, seed 11:

JULIET: this shall I cannot tell
ULIET:
At you my time to wife is to give my patience: say as much
I be admine own facess
Which love this present that soritor, I
For before the deeds the pursuity and o' the greater
Of what thought once to give yours.

MERCUTIO:
Nay, good doubt not a hand
Which, as day, beat is the greatest ta cop of body a ma bosk,
Three shall I do get my flull of dless,
But slecry 'Come on, to rarve you to your hands.

TYBALT:

KING RICHARD III:

Intended use

This model is for research, education, and local experiments. It generates Shakespeare style text from a short prompt.

Limitations

• It overfits after step 750. The best checkpoint invents words and breaks grammar, and the final checkpoint has lower training loss but higher validation loss.
• The corpus is about 1.1 million characters, so the model only learns the style of one small collection of plays.
• It is not instruction tuned and has no safety tuning. Do not use it in production.

License

MIT. The Tiny Shakespeare text is public domain.

Notes

This repository uses a custom implementation, not the transformers library, so load it with the included code rather than AutoModelForCausalLM.

Data credit: Tiny Shakespeare from Andrej Karpathy's char rnn repository. The play text is public domain.