Summer-0.5B-S0

Summer-0.5B-S0 is a from-scratch bilingual (Chinese/English) base model: 524,336,128 parameters, trained from random initialization on 14.6B tokens with a self-trained 81,903-piece tokenizer.

It is not a fine-tune or a tokenizer-swap of any existing model. The architecture follows Qwen/Qwen3-0.6B-Base (28 layers / hidden 1024 / GQA 16:8 / head_dim 128 / tied embeddings / RoPE theta 1e6), but every weight starts from N(0, 0.02).

Stage

S0 — monolingual only, from-scratch pretraining. 14.6B tokens, English 70% / Chinese 30%, 55,694 steps.

This is a retrain of the original Summer-0.5B-S0, redesigned to align with nanochat's actual training recipe:

• English is a single source (FineWebEdu, HuggingFaceFW/fineweb-edu), matching nanochat's karpathy/fineweb-edu-100b-shuffle — not the five-source blend the original release used.
• Chinese is still multiple sources, but screened first: two sources found to be 51.8% / 55.7% Traditional-Chinese-dominant by sampled character frequency (against a downstream instruction mix that is 100% Simplified) were dropped rather than mixed in.
• Packing is BOS-aligned best-fit (bos_bestfit): every training row starts with <bos> and packs whole documents greedily by best fit, cropping only the one document (if any) that doesn't fit the remainder — the same algorithm as nanochat's tokenizing_distributed_data_loader_with_state_bos_bestfit, verified line-by-line against nanochat's source. The original release used a continuous stream with no BOS token at all.
• Sequence length is 2048, matching nanochat (max_seq_len=2048 since its first commit, used unchanged at every depth from d4 to d26+).

Every input to this model must start with <bos>. It has never seen a sequence that doesn't. example_load.py / example_vllm.py in this repo do this for you — if you tokenize text yourself without prepending <bos>, the model will produce degenerate repetitive output regardless of prompt.

No parallel or instruction data at any point.

What to expect

This is a 14.6B-token model. For scale: Qwen3-0.6B-Base saw 36T tokens — about 2,700x more. Treat the numbers below as what that budget buys, not as a competitive result.

Letter multiple-choice (nanochat's primary format — render the question, model answers a single letter)

	previous release	this release
ARC-Easy	0.2563	0.2668
ARC-Challenge	0.2338	0.2654
MMLU	0.2306	0.2551
C-Eval	0.2166	0.2527

This release scores above the random baseline (0.25) on all four tasks; the previous release was below it on three of four.

Likelihood-scoring protocol (lm-evaluation-harness convention)

	previous release	this release
ARC-Easy (acc_norm)	0.4949	0.5391
ARC-Challenge (acc_norm)	0.2671	0.3003
MMLU (acc)	0.2520	0.2465
C-Eval (acc_norm)	0.2363	0.2348
GSM8K 8-shot (flexible-extract)	0.0121	0.0167

WMT22 5-shot translation

	BLEU	COMET
zh->en	0.29	0.4354
en->zh	2.78	0.5505

Few-shot translation is still essentially zero — expected, this checkpoint has never seen parallel text. The model produces fluent but off-topic continuations rather than translations; it does not yet follow the in-context examples. See Summer-0.5B-S1 for the version where in-context translation appears. Its value is as (a) a from-scratch bilingual base model in its own right, and (b) a starting point for annealing / SFT.

Tokenizer

The tokenizer is a compiled C++ extension, not loadable by AutoTokenizer. The release ships tokenizer.py and example_load.py:

pip install git+https://github.com/Ismantic/PieceTokenizer
python example_load.py

The model code (model.py, checkpoint.py) is bundled too — the package depends only on torch plus the tokenizer extension, not on transformers.

Training

Full pipeline, data mixes and every design decision (including the mistakes) are documented in https://github.com/Ismantic/Summer. Notably docs/WHY.md records why fp32 master weights are mandatory, why the learning rate schedule is WSD rather than cosine, and what the vocabulary swap cost.

License

Apache-2.0. Training corpora are public datasets (FineWeb-Edu, Cosmopedia, CCI3-HQ, SkyPile, WMT19, OPUS-100 and others; see data/source.py upstream). Please observe their respective licenses.