CLOD-149M (v1)
A 148.9M-parameter decoder-only language model trained from scratch on a single
RTX 5050 8 GB Laptop GPU in 8 days. 20B tokens: ~18.2B of ClimbMix-400B
(single epoch, no repeats) followed by a ~10% terminal annealing phase of
procedurally generated arithmetic word problems (55%) mixed with held-out
ClimbMix text (45%), scheduled to coincide with LR warmdown.
Specs
|
|
| Parameters |
148,898,078 (sub-150M class) |
| Architecture |
10 layers, d=512, 4 heads, full attention, RoPE, value embeddings |
| Optimizer |
Muon (matrices) + AdamW (embeddings/head), nanochat defaults |
| Tokenizer |
custom 32,768-vocab BPE (rustbpe), single-digit number splitting |
| Tokens |
20.0B (415:1 tokens-per-scaling-param) |
| Batch |
524,288 tokens/step, seq len 2048 |
| Final val_bpb |
0.831965 |
Evaluation
| Task |
Score |
| HellaSwag |
34.86 |
| ARC-Easy / ARC-Challenge |
45.37 / 26.88 |
| PIQA |
67.08 |
| ArithMark-3 |
36.00 |
| Intelligence Index (Open SLM formula) |
19.64 |
Known limitations
- Confidently believes Michelangelo painted the Mona Lisa and that quartz is
the hardest natural substance - majority-vote errors absorbed from web text.
- Arithmetic is completion-format trained; multiple-choice transfer is partial
(ArithMark 36%). v2 will anneal in MC format.
- Greedy samples loop; use temperature ≥ 0.8.
