⚡ MiniGPT-TinyStories v1.0
A compact decoder-only language model built from scratch in TensorFlow/Keras.
Small enough to inspect. Large enough to expose the machinery of a real language model.
MiniGPT-TinyStories is a compact autoregressive Transformer trained from scratch on TinyStories-style data.
Rather than wrapping an existing LLM, this project implements the core components directly:
Token Embeddings → RMSNorm → RoPE → Multi-Head Causal Self-Attention → SwiGLU → Residual Blocks → RMSNorm → Weight-Tied Language Model Head
The goal is not to compete with billion-parameter foundation models.
The goal is more interesting:
Build, train, inspect, debug and deploy a complete language model from first principles.
🚀 Model at a Glance
| Specification | MiniGPT-TinyStories v1.0 |
|---|---|
| Model type | Decoder-only autoregressive Transformer |
| Parameters | <UNKNOWN FILL> |
| Hidden dimension | 768 |
| Attention heads | 12 |
| Head dimension | 64 |
| Transformer blocks | 5 |
| FFN dimension | 3,072 |
| Vocabulary size | 50,257 |
| Training context | 256 tokens |
| RoPE implementation capacity | 2,048 positions |
| Training tokens | ~100M |
| Validation tokens | ~5M |
| Configured epochs | 3 |
| Maximum learning rate | 1e-4 |
| Minimum learning rate | 1e-5 |
| Warmup | 2,000 steps |
| Optimizer | AdamW |
| Weight decay | 0.001 |
| Gradient clipping | 1.0 |
| Precision | bfloat16 mixed precision |
| Framework | TensorFlow / Keras |
| Training dataset | TinyStories |
| Training objective | Causal next-token prediction |
| Generation | Temperature + Top-p sampling |
The architectural configuration above is taken directly from the training implementation.
🧬 Architecture
INPUT TOKENS
│
▼
┌──────────────────┐
│ TOKEN EMBEDDING │
│ 50,257 × 768 │
└────────┬─────────┘
│
▼
┌─────────────────────────┐
│ DECODER BLOCK │
│ │
│ RMSNorm │
│ │ │
│ ▼ │
│ Q / K / V Projection │
│ │ │
│ ▼ │
│ RoPE │
│ │ │
│ ▼ │
│ Causal Self Attention │
│ │ │
│ ▼ │
│ Residual Connection │
│ │ │
│ ▼ │
│ RMSNorm │
│ │ │
│ ▼ │
│ SwiGLU │
│ │ │
│ ▼ │
│ Residual Connection │
└──────────┬──────────────┘
│
│ × 5
▼
┌────────────────┐
│ RMSNorm │
└───────┬────────┘
│
▼
┌─────────────────────┐
│ LM HEAD │
│ │
│ Hidden × Embedding │
│ Weight Tied │
└──────────┬──────────┘
│
▼
LOGITS
│
▼
TEMPERATURE / TOP-P
│
▼
NEXT TOKEN
⚙️ Under the Hood
01 — Token Embeddings
Input token IDs are mapped into a continuous 768-dimensional representation.
token_id
│
▼
Embedding Table
50,257 × 768
│
▼
768-dimensional representation
The language model operates on these learned representations rather than directly on token IDs.
02 — Multi-Head Self-Attention
Each decoder block uses 12 attention heads.
D_MODEL = 768
HEADS = 12
768 / 12 = 64
HEAD_DIM = 64
The attention implementation explicitly constructs Q/K/V projections, splits them into heads, applies rotary position encoding, computes scaled dot-product attention, applies a causal mask, and merges the heads back together.
Conceptually:
X
│
├── Q ──┐
├── K ──┼──► RoPE ──► Causal Attention
└── V ──┘
│
▼
Output Projection
The causal mask ensures that token t cannot access information from tokens after t.
🌀 Rotary Positional Embeddings
MiniGPT uses RoPE — Rotary Positional Embeddings.
Instead of adding a learned positional vector to the token embedding, positional information is injected directly into the attention representation by rotating Q and K.
Q ──► RoPE ──┐
├──► Attention
K ──► RoPE ──┘
The implementation uses a base of 10,000 and a half-split rotation strategy.
Important distinction
The model was trained with a 256-token context.
The internal RoPE implementation is instantiated with a maximum sequence length of 2,048.
These are not the same thing.
Architectural positional capacity ≠ experimentally validated context length.
Long-context behavior beyond the trained 256-token window should therefore be considered unvalidated.
🧱 SwiGLU Feed-Forward Network
Each decoder block contains a gated feed-forward network:
Input
│
┌──────┴──────┐
▼ ▼
Gate Projection Up Projection
│ │
SiLU │
│ │
└──────┬──────┘
│
Elementwise ×
│
▼
Down Projection
│
▼
Output
Configuration:
d_model = 768
hidden_dim = 3072
The implementation uses separate gate, up and down projections with a SiLU activation on the gate branch.
📐 Decoder Block
Each Transformer block follows:
x
│
├── RMSNorm
│
├── Multi-Head Self-Attention
│
└── Residual Add
│
▼
RMSNorm
│
▼
SwiGLU
│
▼
Residual Add
Five such blocks are stacked sequentially.
This gives the model:
5 × Decoder Block
rather than an extremely deep architecture.
🔗 Weight-Tied Language Model Head
The output projection reuses the learned token embedding matrix.
Instead of maintaining an entirely separate vocabulary projection:
Hidden State
│
▼
Embedding Matrixᵀ
│
▼
Vocabulary Logits
The implementation explicitly uses the embedding weights to produce the logits.
This reduces the number of independent parameters and ties the input and output token representations together.
🧮 Parameter Breakdown
Model dimensions
Vocabulary : 50,257
Embedding dim : 768
Layers : 5
Attention heads : 12
Head dimension : 64
FFN dimension : 3,072
Estimated structural parameter budget
The architecture is approximately in the ~86M parameter class, assuming the embedding vocabulary and weight tying in the current implementation are exactly as configured.
Exact published parameter count:
<UNKNOWN FILL>
Recommended command before publishing:
print(model.count_params())
Do not use the checkpoint filename tinystories_50m_model.weights.h5 as the parameter count. The filename is not authoritative.
📚 Training Data
MiniGPT was trained on:
TinyStories
A synthetic children's-story dataset designed to provide a relatively simple environment for studying language-model learning.
Training configuration:
Training token budget ~ 100,000,000
Validation token budget ~ 5,000,000
Context length = 256
Batch size = 16
The implementation explicitly defines a ~100M training-token budget and ~5M validation-token budget.
Dataset identifier:
<UNKNOWN FILL>
Dataset preprocessing/tokenization details:
<UNKNOWN FILL>
🏋️ Training Recipe
TinyStories
│
▼
Tokenization
│
▼
100M-token budget
│
▼
256-token chunks
│
▼
Batch size 16
│
▼
┌─────────────────────┐
│ MiniGPT-Tiny │
│ │
│ 5 Transformer │
│ decoder blocks │
└──────────┬──────────┘
│
▼
Next-token loss
│
▼
AdamW
│
▼
Checkpoint
Optimizer
Optimizer AdamW
Weight decay 0.001
Gradient clip 1.0
β₁ 0.0
β₂ 0.95
The checkpoint-reload code currently reconstructs AdamW with a learning rate of 4e-5; this should be described as the resume/checkpoint state rather than confused with the configured maximum learning rate.
Learning-rate schedule
Maximum LR 1e-4
Minimum LR 1e-5
Warmup 2,000 steps
Configured max 80,000 steps
⚠️ Training Configuration Note
The training configuration currently contains:
100M tokens
16 batch
256 context
3 epochs
80,000 total steps
However:
100,000,000 / (16 × 256)
≈ 24,414 steps / epoch
24,414 × 3
≈ 73,242 steps
Therefore the published release should distinguish between:
configured epochs, token budget, and actual completed optimization steps.
Final training run:
Actual epochs completed : <UNKNOWN FILL>
Actual optimizer steps : <UNKNOWN FILL>
Actual tokens processed : <UNKNOWN FILL>
Final training loss : <UNKNOWN FILL>
Final validation loss : <UNKNOWN FILL>
Final validation perplexity : <UNKNOWN FILL>
This is intentionally called out rather than silently choosing one number.
🔢 Context Window
| Property | Value |
|---|---|
| Training sequence length | 256 tokens |
| Model configuration | 256 tokens |
| RoPE internal maximum | 2,048 positions |
| Validated inference context | <UNKNOWN FILL> |
For release purposes:
Officially validated context: 256 tokens
until longer contexts have actually been tested.
🎲 Generation
MiniGPT performs autoregressive generation:
Prompt
│
▼
Tokenize
│
▼
Transformer
│
▼
Logits
│
├── Temperature
│
└── Top-p / Nucleus Sampling
│
▼
Next Token
│
▼
Repeat
The current generation implementation uses nucleus sampling with configurable top_p and temperature. The example inference configuration uses:
temperature = 0.7
top_p = 0.9
max_new = 50
as demonstrated in the current inference wrapper.
✍️ Generation Examples
Example 01
Prompt
A wise crow
Model output
<UNKNOWN FILL>
Example 02
Prompt
Once upon a time there lived two little cats.
Model output
<UNKNOWN FILL>
Example 03
Prompt
The little girl opened the door and saw
Model output
<UNKNOWN FILL>
Generation examples should be copied from an actual fixed inference run rather than manually written for the release page.
📊 Benchmark
A serious model release needs numbers.
For MiniGPT, the benchmark section should therefore be populated with measured results, not fabricated claims.
| Metric | MiniGPT-TinyStories v1.0 |
|---|---|
| Validation loss | <UNKNOWN FILL> |
| Validation perplexity | <UNKNOWN FILL> |
| Training loss | <UNKNOWN FILL> |
| Tokens/sec | <UNKNOWN FILL> |
| Inference tokens/sec | <UNKNOWN FILL> |
| Time to first token | <UNKNOWN FILL> |
| Peak memory | <UNKNOWN FILL> |
| Model file size | <UNKNOWN FILL> |
Suggested evaluation suite
TinyStories validation perplexity
Held-out next-token loss
Generation coherence
Repetition rate
Average generation throughput
Peak memory usage
Optional external benchmarks:
WikiText-2 : <UNKNOWN FILL>
LAMBADA : <UNKNOWN FILL>
HellaSwag : <UNKNOWN FILL>
ARC-Easy : <UNKNOWN FILL>
These should only be added if the model is actually evaluated on them.
🧪 Qualitative Evaluation
Because this model is trained primarily on TinyStories-style data, qualitative evaluation should focus on:
Narrative continuation
Can the model continue simple stories?
Local coherence
Does each generated sentence remain connected to the preceding context?
Repetition
Does generation collapse into repeated phrases or tokens?
Grammar
Does the model learn basic grammatical structure?
Character consistency
Can the model maintain simple entities across several sentences?
Long-context degradation
How quickly does quality degrade beyond the 256-token training context?
Results:
Narrative coherence : <UNKNOWN FILL>
Grammar : <UNKNOWN FILL>
Repetition behaviour : <UNKNOWN FILL>
Character consistency : <UNKNOWN FILL>
Long-context behaviour : <UNKNOWN FILL>
💻 Training Compute
Hardware
Accelerator : <UNKNOWN FILL>
GPU/TPU : <UNKNOWN FILL>
VRAM/HBM : <UNKNOWN FILL>
CPU : <UNKNOWN FILL>
RAM : <UNKNOWN FILL>
Software
Python : <UNKNOWN FILL>
TensorFlow : <UNKNOWN FILL>
Keras : <UNKNOWN FILL>
Transformers : <UNKNOWN FILL>
Hugging Face Hub: <UNKNOWN FILL>
CUDA/ROCm : <UNKNOWN FILL>
Training duration
Total training time : <UNKNOWN FILL>
Total GPU/TPU hours : <UNKNOWN FILL>
Energy / carbon
Energy consumed : <UNKNOWN FILL>
CO₂ estimate : <UNKNOWN FILL>
Hugging Face explicitly recommends documenting compute infrastructure and, where possible, training footprint as part of technical specifications and environmental-impact reporting.
🧠 Why Build a Small LLM?
Modern language models can contain billions of parameters and require enormous distributed infrastructure.
MiniGPT takes the opposite approach.
BIG FOUNDATION MODELS
billions of parameters
│
▼
massive distributed systems
MiniGPT
thousands of lines of code
│
▼
understandable components
│
▼
trainable on accessible
infrastructure
The purpose is mechanistic understanding.
Every major component can be inspected:
Embedding
Attention
RoPE
RMSNorm
SwiGLU
Residual pathways
Causal masking
LM head
Sampling
Training loop
Checkpointing
Inference
🔬 What Was Built From Scratch?
The project implements the core model architecture directly in TensorFlow/Keras rather than importing a pretrained GPT/LLaMA model and fine-tuning it.
Core components
- Token embedding
- Multi-head attention
- Q/K/V projections
- Causal masking
- Rotary positional embeddings
- RMSNorm
- SwiGLU
- Residual connections
- Decoder blocks
- Weight-tied LM head
- Autoregressive generation
- Top-p sampling
- Temperature sampling
- Mixed-precision training
- Checkpoint restoration
- Hugging Face integration
The decoder, SwiGLU, RMSNorm, RoPE and LM head are all directly represented in the project implementation.
🧩 Model Design Philosophy
MiniGPT is LLaMA-inspired, but it should not be described as an official LLaMA implementation.
The architecture combines modern decoder-Transformer ideas:
GPT-style
│
├── decoder-only autoregression
│
└── causal language modeling
LLaMA-style ideas
│
├── RMSNorm
├── RoPE
└── SwiGLU
Custom implementation
│
├── TensorFlow/Keras
├── custom attention
├── custom generation
└── custom Hugging Face wrapper
MiniGPT is an independently implemented, LLaMA-inspired decoder-only Transformer.
🤗 Hugging Face Integration
The project includes a custom Hugging Face wrapper around the TensorFlow/Keras model.
The wrapper exposes the Keras model through a causal-language-model interface and implements a custom text-generation pipeline.
Example:
result = generator(
"A wise crow",
max_new_tokens=50,
top_p=0.9,
temperature=0.7
)
print(result[0]["generated_text"])
⚠️ Limitations
MiniGPT-TinyStories is a small experimental language model.
It should not be compared directly with modern billion-parameter instruction-tuned models.
Known limitations
- Small parameter count
- Small training corpus
- TinyStories-focused distribution
- Short validated context window
- No instruction tuning
- No RLHF
- No preference optimization
- No safety alignment
- No tool-use training
- No retrieval augmentation
- No multimodal capability
- No factuality guarantee
- Limited general-world knowledge
- Possible repetition
- Possible grammatical errors
- Possible hallucinations
This model should be treated as an experimental research/educational artifact rather than a production assistant.
🛠️ Intended Uses
MiniGPT is particularly suitable for:
Research
Experimenting with:
- Transformer architectures
- Attention
- positional encodings
- normalization
- sampling
- language-model scaling
Education
Understanding how an autoregressive language model works internally.
Engineering
Testing:
- TensorFlow inference
- custom model wrappers
- checkpoint restoration
- Hugging Face integration
- lightweight deployment
Experimentation
Useful for exploring:
"What happens if I change the attention?"
"What happens if I increase context?"
"What happens if I change FFN width?"
"What happens if I modify RoPE?"
"What happens if I change sampling?"
🚫 Out of Scope
MiniGPT should not be treated as:
- a general-purpose ChatGPT replacement
- an enterprise knowledge model
- a factual database
- a medical model
- a legal model
- an autonomous agent
- a production coding assistant
- a safety-critical reasoning system
📦 Repository Structure
miniGPT_Project/
│
├── model/
│ ├── model architecture
│ ├── custom layers
│ └── generation
│
├── checkpoint/
│ ├── tinystories_50m_model.weights.h5
│ └── optimizer_momentum.npz
│
├── tokenizer/
│
├── training/
│
├── inference/
│
├── notebooks/
│
├── README.md
│
└── <UNKNOWN FILL>
🔭 Roadmap
v1.0 — Foundation
- Custom decoder Transformer
- RoPE
- RMSNorm
- SwiGLU
- Causal attention
- Weight tying
- TensorFlow/Keras training
- Checkpoint restoration
- Hugging Face wrapper
- Top-p generation
v1.1 — Evaluation
- Formal validation perplexity
- Generation benchmark
- Throughput benchmark
- Memory benchmark
- Repetition analysis
- Long-context experiments
v2.0 — Scaling
- Larger parameter configuration
- Longer context
- Better training schedule
- Larger dataset
- More systematic evaluation
v3.0 — Systems
- Quantized inference
- Optimized KV-cache inference
- Streaming generation
- Local inference server
- Dockerized deployment
- Developer monitoring API
🏗️ From Model → System
The long-term direction of the project is not simply:
TRAIN MODEL
but:
┌─────────────────┐
│ MiniGPT Model │
└────────┬────────┘
│
▼
┌─────────────────┐
│ Inference Engine│
└────────┬────────┘
│
▼
┌─────────────────┐
│ Model Server │
└────────┬────────┘
│
┌───────────┴───────────┐
▼ ▼
Developer API User Interface
│ │
└───────────┬───────────┘
▼
Local AI Workbench
The model is therefore one component in a larger AI systems stack.
📜 Reproducibility
To reproduce the model:
1. Obtain the dataset
2. Prepare/tokenize the training corpus
3. Construct the MiniGPT architecture
4. Initialize the optimizer
5. Train using the published configuration
6. Save checkpoints
7. Evaluate on the held-out validation set
8. Run the generation suite
Random seed:
<UNKNOWN FILL>
Exact preprocessing pipeline:
<UNKNOWN FILL>
Exact training commit:
<UNKNOWN FILL>
📌 Release Status
MiniGPT-TinyStories v1.0
STATUS
Architecture ████████████████████ COMPLETE
Training ████████████████████ COMPLETE
Checkpoint ████████████████████ COMPLETE
Inference ████████████████████ COMPLETE
HF Integration ████████████████████ COMPLETE
Benchmarking ███████░░░░░░░░░░░░░ IN PROGRESS
Documentation ████████████████░░░░ IN PROGRESS
🧠 Final Note
MiniGPT-TinyStories is intentionally small.
That is the point.
A modern LLM can feel like an opaque artifact when all you ever do is call an API.
This project takes the opposite approach:
Build the machinery. Understand the machinery. Train the machinery. Break it. Fix it. Then deploy it.
MiniGPT is an experiment in understanding what actually happens between:
TOKEN
↓
EMBEDDING
↓
ATTENTION
↓
POSITION
↓
NONLINEARITY
↓
RESIDUAL STREAM
↓
LOGITS
↓
PROBABILITY
↓
NEXT TOKEN
And eventually:
DATA
↓
MODEL
↓
INFERENCE
↓
SYSTEM
↓
DEPLOYMENT
Built from scratch. Trained locally. Designed to be understood.
📎 Citation
If you use this model or repository in your work:
@misc{<UNKNOWN FILL>,
title = {MiniGPT-TinyStories v1.0},
author = {<UNKNOWN FILL>},
year = {2026},
url = {<UNKNOWN FILL>}
}
📬 Contact
Developer:
<UNKNOWN FILL>
GitHub:
<UNKNOWN FILL>
Hugging Face:
<UNKNOWN FILL>
Project repository:
<UNKNOWN FILL>
🏷️ Keywords
TensorFlow · Keras · Transformer · GPT · MiniGPT · TinyStories · RoPE · RMSNorm · SwiGLU · Causal Language Model · Language Modeling · Deep Learning · LLM · From Scratch · Generative AI
- Downloads last month
- -
Model tree for ARX1A07/miniGPT_Project
Base model
openai-community/gpt2