Micro GPT
A trainable, decoder-only GPT for conversational instruction data. The default model has 19,960,704 parameters: 384 hidden dimensions, 6 layers, 6 heads, a 512-token context, and an 8,192-token tied BPE embedding. It uses RoPE, RMSNorm, SwiGLU, causal scaled dot-product attention, and no linear biases.
Requirements
Use Python 3.12 or newer in a virtual environment. Install PyTorch with a CUDA build compatible with your NVIDIA driver, then install the project dependencies. The main tokenizer/data scripts stream HuggingFaceH4/no_robots (train and test); examples are formatted as one User: / Assistant: exchange, optional System: context is retained, and the story-generation category is excluded. No Robots is licensed CC BY-NC 4.0, so its use is non-commercial. Token data is stored on disk as uint16 files and accessed through NumPy memmaps.
Run In Fish
Install dependencies once, then start training from the project directory:
python3 -m venv .venv
source .venv/bin/activate.fish
python -m pip install torch numpy tokenizers datasets tqdm
python train.py
python llm.py
python generate.py --checkpoint agent-checkpoints/best.pt --tokenizer agent-tokenizer.json --prompt "User: What is photosynthesis?\nAssistant:"
llm.py opens an interactive chat using agent-checkpoints/latest.pt. It is prompted to respond to the current goal directly rather than reciting chat history. If a reply contains common uncertainty phrases, the chat checks whether huggingface.co is reachable; only when connected does it load/cache the No Robots training split, retrieve up to two keyword-matched demonstrations, and retry once. The demonstrations are not factual sources; if offline or no good match is found, the original reply is kept. Enter /reload to load a newer checkpoint while training continues, /reset to clear the conversation, or /quit to exit. The model learns tool-call patterns from training data, but the chat runner does not execute external tools or actions.
train.py asks whether to start fresh (f) or continue from the latest checkpoint (s); choose s to keep training the current model. It automatically runs tokenizer_train.py and prepare_data.py when No Robots outputs are missing or the agent-data/dataset.json manifest does not match. It also prepares the optional tool-use data if needed. These data additions do not reset weights; select resume rather than --fresh. The first run downloads the HF datasets and requires network access. Agent artifacts use agent-tokenizer.json, agent-data/, and agent-checkpoints/, leaving existing story-model files untouched.
Training
The defaults use 10,000 optimizer steps, a micro-batch of 8, and 4 accumulation steps (effective batch 32), tuned to fit the project's 4 GB GTX 1650. Validation runs every 1,000 steps over 5 batches, with samples every 2,000 steps to reduce interruptions to training. It uses fp16 autocast, fused AdamW on CUDA, gradient clipping, decay on matrix parameters only, 500 warmup steps and cosine decay, best/latest checkpoints, throughput and GPU-memory reporting. Batch windows are gathered in vectorized NumPy operations, and loss reporting synchronizes with CUDA once per optimizer update. On normal training runs it automatically prepares a small Hermes function-calling corpus and mixes 20% of batch rows from it with No Robots; disable this with --no-agent-data or adjust --agent-data-probability.
Masked-token adaptation is opt-in and separate from normal causal training. Run python masked.py to load the existing agent-checkpoints/latest.pt, perform 500 masked reconstruction updates on the Hugging Face No Robots token data, and overwrite that same checkpoint. It does not initialize new weights or require --fresh; use python masked.py --steps 100 --learning-rate 1e-5 to adjust the run. This is a bidirectional denoising objective, distinct from CLM, and may affect causal generation quality, so the script uses a low learning rate by default.
If CUDA runs out of memory, lower the micro-batch and raise accumulation to keep a similar effective batch, for example:
python train.py --batch-size 4 --gradient-accumulation 8
Agent tool-use examples come from NousResearch/hermes-function-calling-v1 (func_calling_singleturn, Apache-2.0); No Robots remains the main dataset and is CC BY-NC 4.0. Checkpoints use fixed filenames: latest.pt is overwritten at each evaluation and on Ctrl+C, while best.pt is overwritten only when validation loss improves. No per-step checkpoint copies are created. The append-only agent-checkpoints/training-notes.md records run settings, evaluation losses, and completion or interruption status; use --notes PATH to choose another file. Pressing Ctrl+C saves the last completed training step, including optimizer state. Choose s on the next launch to continue from it. Use --fresh to skip the prompt and start over, or --resume PATH to skip the prompt and load a specific checkpoint.
To extend the training target while resuming:
python train.py --data-dir agent-data --tokenizer agent-tokenizer.json --save-dir agent-checkpoints --steps 20000
Use --device cpu to run without CUDA, or --device cuda to require the GPU. Training from scratch with this 20M-parameter model remains a small-scale experiment; the dataset is not a substitute for large-scale language-model pretraining.