ANTRU-Nano

A 79.5M-parameter causal decoder-only transformer trained 100% from scratch (random initialization, no pre-trained base) on a consumer laptop: an Intel i7-8550U CPU and a GeForce MX150 (2GB) GPU, in one ~40-minute GPU session at ~761 tokens/sec.

Built as part of the ANTRU project — a family of fully local, self-trained models.

Model details

Architecture	Decoder-only transformer (pre-norm LayerNorm, GELU FFN, learned positional embeddings, tied input/output embeddings)
Parameters	79,538,688
Hidden dim	768 · 12 layers · 12 heads (head_dim 64) · FFN 2048
Vocabulary	16,384 word-level BPE-free tokens (frequency-built, byte/char fallback — zero unknown-token failures)
Context	1,024-token attention window + up to 200K-token application-level RAM retrieval buffer
Checkpoint	fp32 + optimizer state (~911MB)

Training

• Corpus (~1.79M tokens): English Wikipedia (70%) + a curated ANTRU curriculum (identity & persona statements, ~25 verified Python/coding explanations, step-by-step math reasoning) + local knowledge-base facts + 30k synthetic arithmetic drills (repeated 6×, 4× respectively so the small model could actually learn them).
• Recipe: AdamW (lr 3e-4, cosine decay, 100-step warmup), grad clip 1.0, batch 1 × seq 512, 1 epoch, loss 9.7 → ~3.3.
• Hardware: MX150 2GB — weights + gradients + optimizer fit entirely in VRAM (~1.8GB at 99% utilization).

What it can do (honest)

✅ Generates English-like token structure, learned word ordering, answers simple arithmetic from its drills ("25 + 17" patterns), reproduces its identity curriculum ("I am ANTRU...") when prompted in-style, never crashes on unknown characters (byte fallback).

⚠️ This is an educational artifact, not a chat assistant. At ~22 training tokens per parameter it is heavily under-trained (Chinchilla-optimal would be ~20× more data). Expect loosely-coherent local text rather than fluent answers. Best results: low temperature (0.2–0.3), top_k ≤ 5, short prompts in its curriculum style.

Usage

The model uses a custom (transformers-independent) PyTorch class. Full code ships in the code/ folder:

import sys; sys.path.insert(0, "code")
from antru_model_torch import AntruTorchModel
from antru_tokenizer import AntruTokenizer
from antru_config import AntruConfig
import torch

tok = AntruTokenizer.load("vocab.json")
cfg = AntruConfig(vocab_size=tok.vocab_size)
model = AntruTorchModel.load_checkpoint("antru_nano.pt")  # 911MB fp32+optimizer
ids = tok.encode("What is 25 + 17?", add_special_tokens=False)
out = model.generate(ids, max_new_tokens=24, temperature=0.25, top_k=3)
print(tok.decode(out[len(ids):]))

License & attribution

Apache-2.0. 100% of the weights were trained by the ANTRU project's own hardware — no pre-trained base was used. Corpus text from English Wikipedia (CC BY-SA 3.0).