Summer-0.5B-Chat

Summer-0.5B-Chat is a from-scratch bilingual (Chinese/English) chat model: 524,336,128 parameters, trained from random initialization on 14.6B + 625M chat tokens with a self-trained 81,903-piece tokenizer.

It is not a fine-tune or a tokenizer-swap of any existing model. The architecture follows Qwen/Qwen3-0.6B-Base (28 layers / hidden 1024 / GQA 16:8 / head_dim 128 / tied embeddings / RoPE theta 1e6), but every weight starts from N(0, 0.02).

Stage

Chat (v9) — single-stage post-training from Summer-0.5B-S0 (not from Summer-0.5B-S1; the parallel-data anneal is a separate branch for the translation downstream, unrelated to becoming a general chat model).

This follows nanochat's current recipe (chat_sft.py, commit 1ddaad1 onward): no separate mid-training stage. An earlier version of this project's chat line used a two-stage midtrain-then-SFT design copied from an older nanochat snapshot; that stage was dropped once we confirmed nanochat itself had merged it into a single SFT pass, relying on a large enough data mixture to teach the special tokens (<user>/<assistant>/<end>) instead of a dedicated full-loss warm-up stage.

• Data: chat_v9 mixture, 305,187 rows / 625M tokens — the same chat_full mixture as the previous (v8) release (SmolTalk full 460K, MMLU-aux x1, ARC-Easy/Challenge, GSM8K x4, spelling tasks, Chinese data scaled to keep row-share constant), plus three targeted additions made after manual testing of v8 surfaced specific, reproducible failures:
  • Identity 2x1,000 → 2x5,000 (same hand-written QA pool, just seen more times during training) — v8 showed inconsistent self-identification across turns; the identity signal was only 0.23% of the mix by row count.
  • WildChat-1M Chinese subset, 8,000 rows (real user↔ChatGPT conversations, filtered for toxicity and for any turn where the assistant self-identifies as ChatGPT/GPT/OpenAI, which would otherwise teach the wrong identity) — v8's data was 100% synthetic QA/MC-style tasks with no coverage of casual, non-question follow-ups ("continue", a passing remark), which v8 handled by drifting to an unrelated topic.
  • ALMA parallel corpus reformatted as bidirectional translation instructions, 6,000 rows — v8 was trained entirely on Summer-0.5B-S0 (monolingual), so the chat model had never seen a translation instruction; asking it to translate produced fabricated, off-topic text rather than an attempt.
• Packing: BOS-aligned best-fit, padding (not cropping) at row boundaries so no conversation is ever cut mid-turn. Loss is computed on assistant turns only.
• Learning rate: single linear decay from 0.8x the pretraining peak LR to 0, matching nanochat's init_lr_frac=0.8 — identical hyperparameters to v8, since only the data changed (single-variable comparison).
• 4,768 steps.

Every input must start with <bos>, same as Summer-0.5B-S0.

What to expect

This is a 14.6B + 625M chat-token model. For scale: Qwen3-0.6B-Base saw 36T tokens — about 2,700x more. Treat the numbers below as what that budget buys, not as a competitive result.

Letter multiple-choice (nanochat's primary format)

	nanochat d20	v8 (previous release)	v9 (this release)
ARC-Easy	0.4033	0.4545	0.4440
ARC-Challenge	—	0.3532	0.3584
MMLU	0.3232	0.3328	0.3296
C-Eval	— (d20 has no Chinese)	0.3682	0.3646
Format-following (free_argmax_is_letter)	~1.0	0.99-1.00	0.99-1.00

Flat vs v8 — every difference is within the metric's own sampling noise (e.g. MMLU's 14,042-item standard error is ~0.004, the observed 0.0032 gap is smaller than that). Expected: v9's three data additions target conversational behavior (identity, casual follow-ups, translation) that this benchmark doesn't measure, not the multiple-choice skill itself. Still beats d20 on every comparable metric, same as v8.

Stop rate / long-form generation (repetition_penalty=1.15, not greedy)

Greedy decoding (temperature=0) is only the scoring protocol used above — it is not what this model is served with. repetition_penalty=1.15 was found empirically to be the deployment sweet spot (see this project's prepare/stoprate.py for the full derivation); nanochat's own interactive chat_cli.py also does not use greedy decoding by default.

	en natural-stop	en long-form success	zh natural-stop	zh long-form success
v8	99%	64%	79%	24%
v9	100%	68%	84%	25%

Small, consistently-positive movement on all four numbers (n=100 per language per release, standard error ~4-5 points each) — not a decisive win on its own, but a useful cross-check: nothing regressed. "Long-form success" = >=150 tokens, repetition ratio <0.2, stopped naturally. Chinese open-ended creative writing (poems, stories) is still the weakest spot — it is more prone to repetition collapse than factual Q&A, a known, unresolved limitation (root cause not fully identified; greedy decoding is the primary suspect, not the training recipe, per this project's own ablations — see docs/POSTTRAIN.md).

The three targeted fixes, checked by hand against v8's actual failures

Benchmarks above don't measure any of what v9 was built to fix, so these were checked directly, replaying the same prompts that failed on v8:

Scenario	v8	v9
"Who are you?"	inconsistent across turns	consistently answers "Summer-0.5B"
"Translate to English: 我是中国人"	ignored the instruction, produced an unrelated fabricated paragraph	"I am Chinese." — correct, on-topic
Asked for investment advice, then "go on"	non-sequitur ("I'm an AI assistant, I can't express myself like a human")	stayed on-topic, offered more investment-related points (not a literal continuation of the same list, but topically correct)
Given a bubble-sort explanation, then told "this looks like a pretty good model" (a remark, not a question)	drifted into an unrelated rambling "improved bubble sort" tangent	still fails — invented an unrelated "time-series analysis model" tangent, a different failure of the same underlying kind

Three out of four fixed or improved; the fourth (responding appropriately to a passing remark rather than a question) is not solved. 8,000 rows of WildChat data was evidently not enough coverage for this specific input shape. Reported here rather than glossed over — the goal is an accurate account of what changed, not a clean narrative.

What this model is / is not good at (from manual testing, not benchmarked)

This is a 524M-parameter model trained on 14.6B pretraining + 625M post-training tokens — for scale, Qwen3-0.6B-Base saw 36T tokens, about 2,700x more. Do not expect factual reliability. Manual testing surfaces confident-sounding but fabricated numbers on things like population figures, country counts, or basic arithmetic ("how many days in a week"). This tracks with what the letter-MC benchmark measures (does it follow instructions, pick a reasonable-sounding answer) rather than what it does not measure (is the answer actually correct) — the two are different questions, and this model was optimized against the former. Self-identification is now consistent (see above), but responding sensibly to a remark that isn't phrased as a question is still unreliable.

Tokenizer

The tokenizer is a compiled C++ extension, not loadable by AutoTokenizer. The release ships tokenizer.py and example_load.py:

pip install git+https://github.com/Ismantic/PieceTokenizer
python example_load.py

The model code (model.py, checkpoint.py) is bundled too — the package depends only on torch plus the tokenizer extension, not on transformers.

Training

Full pipeline, data mixes and every design decision (including the mistakes) are documented in https://github.com/Ismantic/Summer. Notably docs/WHY.md records why fp32 master weights are mandatory, why the learning rate schedule is WSD rather than cosine, and what the vocabulary swap cost.

License

Apache-2.0. Training corpora are public datasets (FineWeb-Edu, Cosmopedia, CCI3-HQ, SkyPile, WMT19, OPUS-100 and others; see data/source.py upstream). Please observe their respective licenses.