Prochlo-190M-SFT

Supervised fine-tuned version of Prochlo-190M-Base — a 190.5M-parameter dense English SLM trained from scratch.

Model family

Repo	Stage
CyanoAI/Prochlo-190M-Base	Pre-trained base
CyanoAI/Prochlo-190M-SFT	Supervised fine-tune (this repo)
CyanoAI/Prochlo-190M	Preference-aligned (DPO + RLAIF) — in training

All variants share architecture, tokenizer, and vocabulary.

Training

Base	CyanoAI/Prochlo-190M-Base (~7.7B pre-training tokens, 12 layers, vocab 50,261)
Data	HuggingFaceTB/smol-smoltalk, ~460k English conversations, chat-formatted with <|user|> / <|assistant|>
Method	Full-parameter SFT, 2 epochs, lr 3e-4 cosine, batch 216, max_len 1024, fp16, loss masked to assistant tokens
Steps	96,135
Final loss	1.9308 (train), ~1.94 (eval)
Duration	27 hours

Evaluation

26-item behavioral rubric (format adherence, anti-sycophancy, code, concision): 61.5% pass.

Strong on code instructions and factual QA. Weak on concise replies, long multi-turn coherence, and arithmetic. English only — Chinese input produces garbage by design.

Usage

The chat format is plain text with control tokens, not a chat template:

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("CyanoAI/Prochlo-190M-SFT")
model = AutoModelForCausalLM.from_pretrained("CyanoAI/Prochlo-190M-SFT", dtype="float16").cuda()

prompt = "<|user|>Explain what a closure is.<|end|><|assistant|>"
ids = tok(prompt, return_tensors="pt").input_ids.cuda()
out = model.generate(ids, max_new_tokens=256, do_sample=False,
                     eos_token_id=tok.convert_tokens_to_ids("<|end|>"))
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))

Note: weights are sharded across 32 safetensors files with a model.safetensors.index.json; from_pretrained handles this transparently.

License

Apache-2.0