MNN-Pocket-0.3B-RU (work in progress)

Status: still training. No weights in this repository yet. This card documents the recipe and the measurements taken so far; checkpoints live in the private repo Rahspide/mnn-pocket-ru-run (ckpt/state.pt). Sections marked (pending) will be filled when training finishes.

A continued-pretraining run of taobao-mnn/MNN-Pocket-0.3B-Base that teaches it Russian while keeping English and Chinese intact. It is a small multilingual continuation model first, and a keyboard engine second: the keyboard is the use case that set the size and latency budget, not the only thing it can do.

The goal is not instruction following. The model is trained as a plain causal LM on raw Russian text, so it continues a typed prefix instead of answering it. English and Chinese are kept alive with a 25% replay mix.

Base	taobao-mnn/MNN-Pocket-0.3B-Base (Qwen3 architecture, 303M non-embedding / 361M total params)
Tokenizer	65,536 tokens, byte-level BPE, `<
Training tokens	1.2B (0.9B ru + 0.204B en + 0.096B zh)
Method	full fine-tune, 1 epoch, no LoRA
Hardware	single Colab T4 16 GB, 4.8k tok/s, ~69 h total
Progress (2026-11-09)	step 1,340 / 36,621, 44M tokens

Why the recipe looks like this: Russian is expensive in this tokenizer

The base tokenizer is a distilled 65k vocabulary with almost no Cyrillic merges. Russian comes out close to character level — Спасибо becomes 6 tokens:

"Спасибо" -> ['С', 'п', 'аÑģ', 'и', 'б', 'о']

Measured on prose samples with the real tokenizer:

Tokenizer	RU chars/token	RU tokens per word	EN chars/token
MNN-Pocket / Qwen3-distilled (65,536)	1.75–1.85	≈3.3	3.78
ruGPT3-small (50k, Russian BPE)	4.2	≈1.3	—

Consequences that shaped the plan:

• the same 1.2B-token budget buys 2.2B Russian characters (340M words), roughly 2.2× less text than a Russian-native tokenizer would fit;
• on a phone, one suggested Russian word costs ~3 decode steps instead of ~1, which matters for a keyboard latency budget;
• vocabulary surgery (adding a few thousand Cyrillic merges plus an embedding resize) is a separate experiment, not part of v1. v1 stays tokenizer-compatible with the base model so it drops into the existing MNN / GGUF toolchain.

Training recipe

• Full fine-tune from MNN-Pocket-0.3B-Base, fp32 master weights with fp16 autocast (T4 has no usable bf16).
• Sequence length 1024, random windows over a packed uint16 token stream, <|endoftext|> between documents.
• 32,768 tokens per optimizer step (micro-batch 4 × 1024, gradient accumulation 8).
• AdamW, β = (0.9, 0.95), weight decay 0.1, gradient clip 1.0, lr 2e-4 with 1% linear warmup and cosine decay.
• One epoch over 1.2B tokens = 36,621 steps.
• Checkpoint every 500 steps, uploaded to a private HF repo; training auto-resumes from the last checkpoint after a session restart.

Data

All sources are public Hugging Face datasets. * marks the validation split: the first 100k tokens of each stream are held out before training starts, so validation text is never seen.

Language	Dataset	Config	Tokens	Share
ru	HuggingFaceFW/fineweb-2	rus_Cyrl	0.90B (100k held out*)	75%
en	HuggingFaceFW/fineweb	sample-100BT	0.204B (100k held out*)	17%
zh	HuggingFaceFW/fineweb-2	cmn_Hani	0.096B (100k held out*)	8%

Licences: FineWeb / FineWeb-2 are ODC-BY 1.0. No scraped Telegram dumps or other unlicensed chat corpora were used, so the resulting weights stay publishable.

Evaluation

Per language: perplexity, bits per character, and top-1 / top-5 next-token accuracy. Bits/char is the metric that stays comparable across tokenizers, which perplexity is not.

Baseline, the untouched base model (measured 2026-11-09, 100k held-out tokens per language):

Lang	ppl	bits/char	top-1	top-5	chars/token
ru	10.42	1.625	46.2%	69.3%	2.08
en	21.85	1.016	39.9%	64.1%	4.40
zh	23.95	3.274	41.9%	62.0%	1.42

Fine-tuned numbers: (pending). The acceptance gate is a clear drop in Russian bits/char with at most a ~10% regression on English and Chinese bits/char.

Two models, two jobs

This repository is the continuation model. Its sister, Rahspide/MNN-Pocket-0.3B-RU-Chat, is the assistant — same base, same Russian, different job.

	This model — MNN-Pocket-0.3B-RU	Sister — MNN-Pocket-0.3B-RU-Chat
Job	continue text	follow instructions
Input	a prefix, often mid-sentence	a system + user message
Output	the plausible continuation	an answer
Trained with	continued pretraining on 1.2B tokens of raw text	instruction tuning on top of this model
Choose it for	keyboard suggestions, autocomplete, prefix completion	chat, Q&A, short rewrites
Do not choose it for	assistant behaviour: it will continue your question instead of answering it	autocomplete: it will answer instead of continuing

The split exists because the two tasks reward opposite behaviour. Training a single checkpoint on both makes it mediocre at each: a continuation model that starts answering, and an assistant that leaks into completion mode.

Intended use

• On-device next-word and short-phrase suggestion in a keyboard or IME, Russian first, English and Chinese from the same checkpoint (a keyboard can pick the language from the current word or the layout).
• Text completion, prefix continuation and autocomplete for Russian anywhere a ~200 MB quantised model has to run locally: chat composers, note apps, search fields, terminal prompts.
• Any pipeline that needs a small causal LM fluent in Russian that can still read English and Chinese — for example tagging, short rewriting suggestions, or generating Russian seed text.

Not intended for: instruction following, factual question answering, code, long-form reasoning, or translation. This is a 0.36B model trained for 1.2B tokens; its knowledge is narrow by construction. The language mix is a deliberate trade: Russian is the target, English and Chinese are kept from degrading, neither is improved.

Roadmap

This checkpoint is the first of several variants, ordered by cost:

Variant	What	Status
v1	Russian continuation model — this run	training
v2	Russian assistant: instruction tuning (SFT on ChatML) on top of v1, not a separate pretraining run — see MNN-Pocket-0.3B-RU-Chat	planned
v3	Tokenizer extension: ~4–8k extra Cyrillic BPE merges plus an embedding resize, warm-started from v1, cutting Russian tokens per word by roughly half	optional, after v1
v4	Quantised builds for devices: GGUF Q8_0 / Q4_K_M and MNN INT4, with device-side tok/s measurements	after v1

The continuation model comes first because it is the cheapest way to put Russian into the base; an assistant is then a short SFT on top of it rather than another round of pretraining.

Limitations

• Work in progress. No quality claims until the run finishes and the table above is filled.
• Russian is tokenised near character level (see above): slower decoding and a larger effective cost per Russian word than the same model pays for English.
• Web text only. Register is mostly written/informal blog Russian; no dialogue corpora, no post-training.
• Capacity is 0.36B. Expect weak facts, weak arithmetic, and drift on long generations.
• English and Chinese are expected to regress somewhat; the 25% replay mix limits but does not remove this.
• The ChatML template from the base model is inherited but untrained here — do not use this checkpoint in chat mode.

Licence and attribution

Weights: Apache-2.0, inherited from the base model. Data: ODC-BY 1.0 (FineWeb, FineWeb-2). If you use this model, credit the base release:

@misc{mnn_pocket_03b,
  title  = {MNN-Pocket-0.3B},
  author = {Taobao MNN team},
  year   = {2026},
  url    = {https://huggingface.co/taobao-mnn/MNN-Pocket-0.3B}
}