transformer-from-scratch — trained checkpoints
Weights for github.com/adimunot21/transformer-from-scratch, a decoder-only Transformer built from first principles in PyTorch — attention, multi-head projections, positional encoding, blocks and training loop written by hand, plus a BPE tokenizer also implemented from scratch.
Two models, both trained 5000 steps on Tiny Shakespeare:
| Folder | Tokenizer | Vocab | Params | d_model | Heads | Layers | Context |
|---|---|---|---|---|---|---|---|
char/ |
character-level | 65 | 1.89 M | 128 | 4 | 4 | 256 |
bpe/ |
BPE (from scratch) | 768 | 5.37 M | 192 | 6 | 6 | 256 |
Each folder holds model.safetensors and a config.json with the exact architecture
and training hyperparameters.
Loading
Plain state_dicts for the model class in the source repo:
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
path = hf_hub_download("adimunot/transformer-from-scratch", "bpe/model.safetensors")
model.load_state_dict(load_file(path)) # model built from config.json
model.eval()
Notes
- The optimizer state present in the original training checkpoints has been stripped — these are inference weights only. The char model went from 11.3 MB to 4.4 MB as a result.
- Intermediate step checkpoints (1000/2000/3000/4000) were not published; only the final step-5000 weights are here.