Prochlo-190M-Base

The smallest and first model from CyanoAI — a 190.5M-parameter dense English base LM. Named after Prochlorococcus, the tiniest and most oxygen-producing photosynthetic organism on Earth: small, dense, and outproducing everything its size.

Model family

Repo	Stage	Status
CyanoAI/Prochlo-190M-Base	Pre-trained base	this repo
CyanoAI/Prochlo-190M-SFT	Supervised fine-tune	available
CyanoAI/Prochlo-190M	Preference-aligned (DPO + RLAIF)	in training

All variants share the same architecture, tokenizer, and vocabulary, so they are drop-in interchangeable.

Model at a glance

Parameters	190.5M dense (untied embeddings)
Architecture	LLaMA-style decoder: RMSNorm, RoPE (θ=1e5), SwiGLU, no biases, per-head QK RMSNorm
Shape	12 layers × dim 768, 12 heads, hidden 3072
Vocab	50,261 (GPT-2 BPE + 4 control tokens)
Context	1024
Precision	FP16
Format	Safetensors, loads with Qwen3ForCausalLM (architecture-compatible; trained from scratch by CyanoAI, not derived from Qwen)

Pretraining

Tokens	~7.7B
Steps	300,000
Optimizer	Muon (2D matrices) + AdamW
Schedule	lr 1e-3, warmup 64, linear decay
Final train loss	1.7947 (ppl ≈ 6.0)
Final eval loss	1.6205 (ppl ≈ 5.06, still descending)

Training corpus: a curated ~826GB English mixture (web / books / code / reference), stream-packed with the tokenizer jointly shipped in this repo.

Intended use

This is a base model — it speaks fluent Wikipedia-style prose but has not been aligned. For chat or assistant behaviour, start from the SFT or aligned variants.

Load

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("CyanoAI/Prochlo-190M-Base")
model = AutoModelForCausalLM.from_pretrained("CyanoAI/Prochlo-190M-Base", dtype="float16")

Evaluation

Zero-shot on wikitext (JSON subset, ~500k tokens): ppl ≈ 109.7. Not directly comparable to GPT-2-series numbers — different training distribution.

License

Apache 2.0