GitHub repository: s-chh/PyTorch-Scratch-LLM
Primary language: Python
Description: Simple and easy to understand PyTorch implementation of Large Language Model (LLM) GPT and LLAMA from scratch with detailed steps.  Implemented: Byte-Pair Tokenizer, Rotational Positional Embedding (RoPe), SwishGLU, RMSNorm, Mixture of Experts (MOE).  Tested on Taylor Swift song lyrics dataset.