bytes-per-token: small LMs that run at ~90 tokens/s on an $8 ESP32-S3

Weights for github.com/alikup-ai/bytes-per-token. The models were designed for one number: how many bytes of weights the microcontroller reads from PSRAM per generated token. They are trained with 4-bit quantization-aware training and run on the board in int4.

model	stored params	weights read per token (int4)	bits/byte, TinyStories valid	speed on ESP32-S3-N16R8
varC_long_s0	8.6M	1.16 MB	0.6253	~90 tok/s at the start of the context, ~81 mean over 256 tokens
combo_s0	3.4M	0.90 MB	0.7322	~113 tok/s at the start, ~88 mean

For reference, slvDev/esp32-ai (28.9M params) scores 0.863 bits/byte on the same text with the same procedure.

Bits per byte: first 5,000 stories of TinyStoriesV2-GPT4-valid.txt (skipping the first, partial one), joined by <|endoftext|>, scored on non-overlapping 256-token windows, summed loss in bits divided by UTF-8 bytes. Reproduce with python bench_pc.py from the GitHub repo.

Architecture

Llama-style decoder, 6 layers, 4 heads, context 256, RoPE, RMSNorm.

• ReLU FFN instead of SwiGLU: ~79% of FFN neurons are zero per token, and the engine skips their rows.
• Per-layer embeddings (PLE), the idea from Gemma 3n: a per-layer token table kept in flash, read only for the current token.
• 4-bit weights (group-wise QAT); embeddings and output layer tied.

model	d	ffn	vocab	PLE dim
varC_long_s0	128	256	8192	128
combo_s0	128	320	4096	64

Files (per model folder)

• model.safetensors: PyTorch weights for train/model.py in the GitHub repo (LM(Config(**config["cfg"]))).
• config.json: shape, training run, measured validation bits/byte.
• tokenizer.json: byte-level BPE (Hugging Face tokenizers), <|endoftext|> between stories.
• model.pack: the int4 pack the ESP32-S3 engine reads, flashed at 0x90000 with the demo firmware. Ready-made full images are in the GitHub release.

Training

Data: the first 300 MiB of TinyStoriesV2-GPT4-train.txt from roneneldan/TinyStories. varC_long_s0: 70M tokens (0.9 epoch), combo_s0: 20M tokens. AdamW, lr 3e-3, batch 32 x 256, one seed, laptop CPU (2.6 h for varC_long_s0).

Limitations

• Trained only on synthetic children's stories: it continues simple English stories and does not follow instructions or answer general questions.
• Context is 256 tokens.
• One training seed per model; small differences between models are within seed noise (~0.003 bits/byte).

License: MIT.