dec-11M

This is a custom, character-level Autoregressive Transformer (GPT-style Decoder) model trained on Chip Huyen's "AI Engineering" book.

Model Details

• Parameters: ~10.84M
• Sequence Context Length (block_size): 256 characters
• Vocabulary Size: 134 unique characters
• Layers: 6 blocks
• Attention Heads: 6 heads
• Embedding Dimension: 384
• Framework: PyTorch

Intended Uses & Limitations

This model is designed for educational purposes to demonstrate text continuation on the domain of AI Engineering. Because it is a character-level model of 11M parameters, it may generate some non-existent words or exhibit limited factual reasoning, but it captures the style, vocabulary, and formatting of the dataset.

How to Use

To generate text from this model, recreate the architecture from your training script, load the pytorch_model.bin state dictionary, and run autoregressive generation with temperature/top-k sampling.