TinyBuddy-80K

🏆 RECORD ATTEMPT: The smallest functional English-speaking language model on Hugging Face. 83,856 parameters — that's ~84K, beating the NaA-IA/Small-ever record by being both tiny AND coherent.

Mission: Prove that under 100K parameters, a language model can still learn English patterns and generate recognizable text. This is not just the smallest — it's the smallest that works.

---

Model Details

Property	Value
Parameters	83,856 (~84K)
Layers	1
Hidden size	48
Attention heads	4 (query) / 2 (key-value) = GQA
FF intermediate size	192
Context length	128
Vocabulary	1,024 tokens (BPE)
Architecture	Llama-style: RMSNorm, RoPE, SiLU/SwiGLU, tied embeddings
Precision	float32

Parameter Breakdown

Component	Parameters
Token Embedding (tied)	49,152
Attention (Q/K/V/O)	5,760
FeedForward (Gate/Up/Down)	27,648
LayerNorm (3× RMSNorm)	144
Total	83,856

---

Architecture

TinyBuddy-100K uses a single transformer block with:

• RMSNorm (pre-norm) — efficient normalization
• Grouped Query Attention — 4 query heads, 2 KV heads (saves params)
• RoPE (Rotary Position Embeddings) — relative position encoding
• SwiGLU (SiLU-gated MLP) — modern activation
• Tied embeddings — input and output share weights (saves ~49K params!)

Input → Embedding → [RMSNorm → GQA Attention → +] → [RMSNorm → SwiGLU FFN → +] → RMSNorm → LM Head → Output

---

Training

• Dataset: TinyStories (~5,000 stories)
• Tokenizer: Byte-level BPE, 1,024 vocabulary (trained from scratch)
• Optimizer: AdamW (lr=5e-3, weight_decay=0.1)
• Schedule: Warmup (50 steps) + Cosine decay
• Steps: 1,000 on CPU
• Hardware: Single CPU core (the challenge!)

---

Usage

import torch
from model import create_model

# Load config
import json
with open("config.json") as f:
    config = json.load(f)

# Create model
model = create_model(config)
model.load_state_dict(torch.load("output/model.pt", map_location="cpu"))
model.eval()

# Generate
from tokenizers import Tokenizer
tokenizer = Tokenizer.from_file("data/tokenizer.json")

prompt = "Once upon a time,"
encoded = tokenizer.encode(prompt)
ids = [1] + encoded.ids  # Add BOS
input_ids = torch.tensor([ids], dtype=torch.long)

output_ids = model.generate(input_ids, max_new_tokens=60, temperature=0.8, top_k=40)
print(tokenizer.decode(output_ids[0].tolist(), skip_special_tokens=True))

---

Limitations

This model is extremely small — it has fewer parameters than a 28×28 grayscale image.

What works:

• Basic word patterns and short phrases
• Recognizable English-like structure
• Story-like opening sentences

What's broken:

• Very limited coherence (1–2 sentences max)
• High repetition
• No factual knowledge or reasoning
• Limited vocabulary diversity

This model exists purely to explore the lower bounds of language modeling. It proves that even at 84K parameters, a neural network can capture statistical patterns in English text.

---

The Record

Model	Parameters	Speaks English?
NaA-IA/Small-ever	112	❌ No
TinyBuddy-80K	83,856	✅ YES

TinyBuddy-100K may not be the absolute smallest model ever, but it's the smallest that actually generates recognizable English text. That's the real achievement.

---

Citation

@misc{tinybuddy100k,
  title  = {TinyBuddy-100K: An 84K parameter Llama-style model that speaks English},
  year   = {2026},
  note   = {Record attempt: smallest functional English text generator.}
}

LONG LIVE TINYBUDDY-80K 🚀