I trained this Language Model from scratch to understand how the architecture works internally. It is a 124 million parameter model based on the GPT-2 architecture, trained on the TinyStories dataset.
I used Andrej Karpathy's nanoGPT repository as a base and trained it on a single RTX 5060 Ti using Vast.ai.
Model Details Architecture: GPT-2 style (Decoder-only Transformer)
Parameters: 124M
Layers: 12
Heads: 12
Embedding Dimension: 768
Dataset: TinyStories
Context Window: 1024 tokens
Usage This model is designed to generate short, coherent English stories with simple vocabulary. It is not a chat assistant, but a completion model.
Reflection I am Andre, a prompt engineer and AI Red Teamer. I wanted to see if I could build and train a functional LLM completely from zero. It took some trial and error, but it works.
If I was able to do this at my 14yo, you can do it too. It is never too late to learn how these things work behind the scenes.