AI25-3
AI25-3 is a decoder-only Transformer language model built completely from scratch as an educational project. The primary objective of this model is to demonstrate how modern Large Language Models (LLMs) are implemented, trained, fine-tuned, and deployed without relying on existing LLM frameworks.
Core Design Philosophy
AI25-3 is a learning-first implementation. The code prioritizes readability, correctness, and educational value over maximum training efficiency or benchmark performance.
⚠️ Educational Model Notice
AI25-3 is an educational toy model created for learning and experimentation. It is not intended for production use, safety-critical applications, or as a replacement for larger instruction-tuned commercial models.
Model Details
The implementation follows the modern design philosophy adopted by many recent open-weight language models while keeping the codebase simple enough to study and modify.
| Feature / Component | Specification |
|---|---|
| Architecture | Decoder-only Transformer (GPT-style) |
| Framework | PyTorch |
| Tokenizer | Custom Byte-Pair Encoding (BPE) |
| Vocabulary Size | 49,152 |
| Positional Encoding | Rotary Positional Embeddings (RoPE) |
| Attention Mechanism | Grouped Query Attention (GQA) |
| Normalization | Pre-RMSNorm |
| Feed-Forward Network | SwiGLU |
| Optimizer | AdamW |
| Learning Rate Schedule | Warmup + Cosine Decay |
Purpose
The goal of AI25-3 is education, not benchmarking.
This project was built to understand every major component involved in training a modern language model, including:
- Tokenizer training
- Data preprocessing
- Decoder-only Transformer implementation
- Attention mechanisms
- Language model pretraining
- Supervised fine-tuning
- Text generation
- Hugging Face model integration
The 3-Stage Pipeline
The overall learning path follows the classic three-stage pipeline:
- Build the architecture
- Pretrain a foundation model
- Fine-tune for downstream tasks
This mirrors the educational workflow described in "Build a Large Language Model (From Scratch)".
Training
Pretraining
AI25-3 was pretrained on a mixture of publicly available datasets including educational web text and synthetic educational corpora.
Supervised Fine-Tuning
The released checkpoint is instruction-tuned for text summarization using publicly available summarization datasets including:
- CNN/DailyMail
- XSum
- SciTLDR
The objective is to produce concise, grammatically correct summaries.
Scope of Use
Intended Uses
AI25-3 is intended for:
- Learning how LLMs work internally
- Reading and understanding Transformer implementations
- Educational demonstrations
- Research prototypes
- Fine-tuning experiments
- Building custom GPT-style models from scratch
Out-of-Scope Uses
AI25-3 should not be used for:
- Production applications
- Medical advice
- Legal advice
- Financial advice
- Safety-critical systems
- Autonomous decision making
Outputs may be inaccurate, incomplete, or fabricated.
⚠️ Limitations
This is a relatively small educational language model. Compared with modern frontier models, it has:
- Far fewer parameters
- Much less training data
- Limited world knowledge
- Reduced reasoning capability
- Lower factual reliability
The focus of this project is understanding how modern LLMs are built, not achieving state-of-the-art benchmark performance.
Future Work
Planned future improvements include:
- Scaling the architecture to approximately 2B parameters
- Pretraining on significantly larger and more diverse datasets
- Improving instruction-following performance
- Extending context length
- Exploring more advanced architectures such as Mixture-of-Experts (MoE) and Multi-Head Latent Attention (MLA)
- More comprehensive evaluation across standard language model benchmarks
Acknowledgements
This project draws inspiration from:
- Attention Is All You Need
- Language Models are Unsupervised Multitask Learners
- Language Models are Few-Shot Learners
- GLU Variants Improve Transformer
- Build a Large Language Model (From Scratch) by Sebastian Raschka
Citation
If you use AI25-3 in your work, please cite this repository.
- Downloads last month
- 165