Mayuresh231/tiny-transformer-29m

Tiny Transformer 29M — TinyStories

A 29,545,472-parameter decoder-only Transformer trained from scratch on 600,000 unique TinyStories examples. It uses a custom 4,096-token lowercase ByteLevel BPE tokenizer and a 256-token context window.

Load and generate

This repository contains custom model code. Review it, then opt in with trust_remote_code=True:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "YOUR_USERNAME/tiny-transformer-29m"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
)

inputs = tokenizer("Once upon a time", return_tensors="pt")
torch.manual_seed(100)
output = model.generate(
    **inputs,
    max_new_tokens=200,
    do_sample=True,
    temperature=0.7,
    top_k=50,
    top_p=0.9,
    repetition_penalty=1.08,
    no_repeat_ngram_size=3,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Architecture

Setting	Value
Parameters	29,545,472
Vocabulary	4,096 ByteLevel BPE tokens
Context	256 tokens
Hidden size	512
Layers	8
Attention heads	8 × 64 dimensions
Feed-forward size	2,048
Dropout	0.1

The output projection is not tied to the token embedding. The implementation uses PyTorch scaled dot-product causal attention. Generation is compatible with the Transformers API but does not use a KV cache, so it recomputes the active context window for every new token.

Training results

Epoch	Train loss	Validation loss	Perplexity
1	2.1191	1.7118	5.54
2	1.6595	1.5911	4.91
3	1.5568	1.5243	4.59
4	1.4971	1.4977	4.47

Training processed 547,749,888 token positions on an NVIDIA RTX 3050 Laptop GPU with 6 GB VRAM. The completed epochs took 3h 9m 25s.

Intended use and limitations

This is an educational small language model for studying tokenization, attention, optimization, checkpointing, and generation. It is not an assistant or a factual knowledge model. Outputs may be repetitive, contradictory, or inherit encoding noise and other biases from TinyStories. Do not use it for high-stakes decisions.