ayushhh1010/nanoserve-27m

Nanoserve 27M

A 26,747,392-parameter decoder-only transformer trained from scratch on TinyStories V2. The architecture, the tokenizer and the training loop were all written by hand -- no AutoModel, no tokenizers library.

It writes short children's stories. It is not a general-purpose assistant and cannot answer questions, follow instructions, or reason. That is the specification, not a shortfall.

Results

Parameters	26,747,392
Training tokens	536,174,127
Validation loss	1.1485 nats/token
Perplexity	3.153
Hardware	single RTX 3050 Laptop (4 GB)
Wall clock	3.27 hours

Architecture

d_model / layers / heads	512 / 8 / 8
KV heads	2 (grouped-query attention)
d_ff	1408 (SwiGLU)
Norm / position	RMSNorm pre-norm / RoPE
Vocabulary / context	8,192 / 1,024
Embeddings	tied

Parameter names mirror HuggingFace's Llama, so the weights load into a LlamaForCausalLM of matching shape.

Usage

from model.tokenizer import BPETokenizer
from model.generate import load_model, generate, SamplingParams

tok = BPETokenizer.load("tokenizer.json")
model, _ = load_model("model.safetensors", device="cuda")

print(generate(
    model, tok,
    "Once upon a time, there was a little girl named Lily who",
    SamplingParams(max_new_tokens=200, temperature=0.8),
))

Training data

TinyStories V2 (GPT-4 generated). 536,174,127 tokens under a purpose-built 8,192-token byte-level BPE vocabulary, which compresses this corpus to 3.965 bytes/token with 98.1% of words encoding to a single token.

Token count was chosen to sit at the Chinchilla-optimal ratio for the parameter count (~20 tokens per parameter).

Limitations

• Vocabulary and world model are limited to the simple English of TinyStories.
• 1,024-token context.
• No instruction tuning, no alignment, no safety filtering of any kind.
• Will confidently produce factually wrong statements; it models story-shaped text, not truth.

Source

Trained as Phase 1 of Nanoserve, a project that trains a small language model and then builds the distributed system that serves it.