IAU-1

A ~209M parameter GPT trained completely from scratch. Custom architecture, not a transformers class, so the model code is included in model.py.

• 24 layers, 12 heads, 768 embedding dim, 1024 context, learned positional embeddings, pre-LN, GELU, no biases, tied input/output embeddings
• Tokenizer: tiktoken gpt2 (vocab padded to 50304)
• Trained for 4000 steps at ~65k tokens/step (about 0.26B tokens)

Training data

Streamed mix: OpenWebText 20% (stand-in for WebText2, which was never released), Wikipedia (20231101.en) 10%, BookCorpus 20%, FineWeb sample-10BT 50%.

Usage

from model import load_iau1, generate
m = load_iau1('model.safetensors')
print(generate(m, 'The history of the internet', n=100))

License

Custom research license, see LICENSE. Short version: free to use for research, learning and personal projects, no commercial use. The training data has mixed licensing (summary below), so the weights are not offered under a standard open license.

Dataset	License / terms
OpenWebText	CC0 for the compilation; the underlying web pages belong to their authors
Wikipedia	CC BY-SA 3.0 and GFDL
BookCorpus	No license stated; many of the books carry personal-use-only terms
FineWeb	ODC-By 1.0, plus Common Crawl terms of use

Limitations

Small model, trained on roughly 1.3B tokens at most, so expect fluent-ish but often wrong or rambling text. Base model only, no instruction tuning or safety tuning. It can reproduce biases and errors from its training data.