Summer-0.5B-S1

Summer-0.5B-S1 is a from-scratch bilingual (Chinese/English) base model: 524,336,128 parameters, trained from random initialization on 14.6B + 1.34B anneal tokens with a self-trained 81,903-piece tokenizer.

It is not a fine-tune or a tokenizer-swap of any existing model. The architecture follows Qwen/Qwen3-0.6B-Base (28 layers / hidden 1024 / GQA 16:8 / head_dim 128 / tied embeddings / RoPE theta 1e6), but every weight starts from N(0, 0.02).

Stage

S1 — S0 plus a parallel-data anneal, same nanochat-aligned recipe as Summer-0.5B-S0 (BOS-aligned best-fit packing, seq_len 2048). Branched from S0's final checkpoint (optimizer state reset — verified by A/B test to behave identically to resuming from a checkpoint with saved optimizer momentum) and trained for 5,103 steps on 1.34B tokens containing ~30% Chinese-English parallel text, packed the same BOS-aligned way as S0 (the anneal data must match the pretraining packing convention, or the model sees an out-of-distribution input shift mid-training).

Every input must start with <bos>, same as Summer-0.5B-S0 — see that model's card for why.

What to expect

This is a 14.6B + 1.34B anneal-token model. For scale: Qwen3-0.6B-Base saw 36T tokens — about 2,700x more. Treat the numbers below as what that budget buys, not as a competitive result.

WMT22 5-shot	BLEU	COMET
zh->en	8.99	0.6883
en->zh	28.36	0.7736

Against this model's own pre-anneal state (S0, 0.29 / 2.78 BLEU) this is qualitative: S0 ignores the in-context examples and produces off-topic continuations, S1 actually translates. Against the previous Summer-0.5B-S1 release (8.99 / 27.29 BLEU, COMET 0.6855 / 0.7743) this new release performs at parity — the nanochat-aligned data recipe did not cost any translation quality while improving the base model on every other tracked metric.

Tokenizer

The tokenizer is a compiled C++ extension, not loadable by AutoTokenizer. The release ships tokenizer.py and example_load.py:

pip install git+https://github.com/Ismantic/PieceTokenizer
python example_load.py

The model code (model.py, checkpoint.py) is bundled too — the package depends only on torch plus the tokenizer extension, not on transformers.

Training

Full pipeline, data mixes and every design decision (including the mistakes) are documented in https://github.com/Ismantic/Summer. Notably docs/WHY.md records why fp32 master weights are mandatory, why the learning rate schedule is WSD rather than cosine, and what the vocabulary swap cost.

License

Apache-2.0. Training corpora are public datasets (FineWeb-Edu, Cosmopedia, CCI3-HQ, SkyPile, WMT19, OPUS-100 and others; see data/source.py upstream). Please observe their respective licenses.