AtoMinima

A 420-parameter character-level GPT. Trained on 370,000,000 characters of FineWeb-edu (sample-10BT).

Benchmarks

Zero-shot, scored as mean log-likelihood per byte over each answer choice, argmax of the choices. Full splits.

Benchmark	AtoMinima	Chance
HellaSwag	25.53%	25.00%
ARC-Easy	26.98%	25.00%
ARC-Challenge	26.02%	25.00%
PIQA	49.13%	50.00%
ArithMark-3.0	30.70%	25.00%
Open SLM index	+1.62	0.00

Held-out validation loss: 3.8371 bits/byte (1M characters of FineWeb-edu held out from training).

Model details

Component	Details
Architecture	decoder-only transformer, pre-norm, causal
Layers	1
Width	d_model 4, 1 head, head_dim 4
Feed-forward	4 -> 16 -> 4, GELU
Normalization	LayerNorm
Embeddings	tied input/output, 42 ids
Positions	fixed sinusoidal
Dropout	none
Parameters	420 trainable
Context	256 characters
Tokenizer	character-level: a-z 0-9 space . , ' \n + unknown
Training tokens	370,000,000
Training data	FineWeb-edu sample-10BT, 10M-character slice
Checkpoint size	11 KB

Usage

pip install torch huggingface_hub
python example.py

import atominima

model = atominima.from_pretrained("arthurblg1802/AtoMinima")   # needs atominima.py in the path
print(model.generate("the water cycle begins when", max_new_tokens=80))

atominima.py in this repo holds the whole model and tokenizer: 90 lines, torch only, no other dependency.

Limitations

This is a character-level model with 420 parameters: it has no usable language ability, and every benchmark score above sits within noise of its baseline except ArithMark-3.0, which is below the 50.80% floor reachable by always answering the second-smallest number. Generated text is not English. It exists as a lower bound, not as a tool.

Licence

Apache-2.0.