Federalist-Char-LM (From Scratch)
Model Description
This repository contains a lightweight, character-level causal language model built entirely from scratch using pure Python and standard math libraries (no PyTorch, TensorFlow, or Transformers).
It trains on a subset of The Federalist Papers using a sliding context window to predict the next character autoregressively.
Model Details & Architecture
- Architecture: Linear Multi-Position N-gram Classifier (Custom 3D Weight Grid)
- Total Parameters: 27,744 ($N \times V \times V$ where $N=6$, $V=68$)
- Context Window ($N$): 6 characters
- Vocabulary Size ($V$): 68 unique characters (letters, punctuation, spaces)
- Optimizer: Stochastic Gradient Descent (SGD)
- Loss Function: Cross-Entropy Loss
- Training Corpus: First 20,000 characters of The Federalist Papers (
data.txt)
How to Use
Clone the repository and run the training/generation script:
python basic_llm3.py