nano-kyrgyz

A 5.7M-parameter character-level Kyrgyz language model, written from primitives in PyTorch and trained on a laptop.

Not a fine-tune. Not a distillation. Every component — RMSNorm, RoPE, grouped-query attention, SwiGLU, QK-normalisation, the Muon optimiser, the INT4 quantiser, the KV cache — is implemented from scratch in the source repository, and each one is ablated rather than assumed.

Two models ship here:

File	What it is	Size
nano-kyrgyz-q8_0.gguf	base — a text continuator	5.8 MB
nano-kyrgyz-chat-q8_0.gguf	chat — SFT'd for turn format and stopping	5.8 MB
nano-kyrgyz-f32.gguf, nano-kyrgyz-chat-f32.gguf	unquantised originals	22 MB each

Read the limitations section before you judge the outputs. This model has 5.7 million parameters — roughly 0.0007× a 8B model. It holds format and orthography, not facts. Asking it the capital of France produces fluent, confident nonsense, and that is expected behaviour, not a bug.

---

Why this exists

Kyrgyz is a low-resource Turkic language: ~5M speakers, no vision-language corpus, and three Cyrillic letters (ң ө ү) that mainstream OCR and tokenisation stacks systematically corrupt. This model is the language half of a document-understanding stack for Kyrgyz, built to demonstrate that every layer — data hygiene, architecture, training, quantisation, deployment — was understood rather than imported.

The measurable claim is not "this model is good". It is: every number below was produced on an M5 Pro MacBook, in about 70 minutes of total training, for $0, and every one of them is reproducible.

---

Results

Metric	Value
Parameters	5,678,848
Validation loss	1.204
Bits per character	1.737 (uniform-random baseline: 5.017)
Training data	Kyrgyz Wikipedia (Leipzig corpus), ~10M characters
Training time	~21 minutes, MPS, bf16
Vocabulary	151 tokens — character-level, full Kyrgyz alphabet asserted
Context	256 tokens
Chat SFT validation loss	0.451

Architecture (deliberately component-identical to Qwen3, see Deployment below):

hidden / layers	256 / 8
query / KV heads	8 / 2 (GQA)
head dim	32
FFN hidden	704 (SwiGLU)
normalisation	RMSNorm + per-head QK-norm before RoPE
position	RoPE, θ = 10000
embeddings	tied
KV cache	2.0 KiB/token

---

Usage

LM Studio

Search DJKG/nano-kyrgyz in the Discover tab, or download the GGUF and drop it in ~/.lmstudio/models/DJKG/nano-kyrgyz/.

Both models are self-configuring — no prompt-template setup required. The chat template ships inside the GGUF, and the base model carries a deliberately empty template so it behaves as a pure continuator in the chat UI.

• Chat model — just talk to it in Kyrgyz. It stops on its own (finish_reason: stop).
• Base model — type the beginning of a sentence, not a question. It continues text.

Set context length to 256 (its trained maximum). Turn off "Chat with Files" — RAG injection instantly overflows a 256-token window.

llama.cpp

# base — continuation
llama-completion -m nano-kyrgyz-q8_0.gguf \
  -p "Кыргыз Республикасынын" -n 60 --temp 0 --no-conversation
# → Кыргыз Республикасынын Кол жазмалар фондусунда калыптанган.

# chat — needs --jinja to use the embedded template
llama-cli -m nano-kyrgyz-chat-q8_0.gguf --jinja -c 256

Homebrew's llama-cli is an interactive TUI and will hang in a script. Use llama-completion for non-interactive runs; the flag is --no-conversation.

Ollama

cat > Modelfile <<'EOF'
FROM ./nano-kyrgyz-chat-q8_0.gguf
TEMPLATE """Колдонуучу: {{ .Prompt }}
Жооп: {{ .Response }}<eos>
"""
PARAMETER stop "Колдонуучу:"
PARAMETER stop "<eos>"
PARAMETER temperature 0.5
PARAMETER num_ctx 256
PARAMETER num_predict 200
EOF
ollama create nano-kyrgyz-chat -f Modelfile
ollama run nano-kyrgyz-chat "Салам!"

Roughly 1,900 tokens/second on an M5 Pro.

OpenAI-compatible API (LM Studio server)

curl http://localhost:1234/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"nano-kyrgyz-chat",
       "messages":[{"role":"user","content":"Салам!"}],
       "temperature":0}'
# → "Салам! Кандай жардам керек?"

Download programmatically

from huggingface_hub import hf_hub_download

path = hf_hub_download("DJKG/nano-kyrgyz", "nano-kyrgyz-chat-q8_0.gguf")

---

What the chat model can and cannot do

The SFT mix is deliberately grounded — every answer is contained in, or derivable from, the prompt. Teaching a 5.7M-parameter model to answer knowledge questions teaches it to hallucinate confidently, which is the exact failure mode the wider project exists to prevent.

Trained tasks:

Prompt	Response
Салам!	Салам! Кандай жардам керек?
Сен кимсиң?	Мен нано-кыргыз — кичинекей кыргызча тил модели.
Баш тамга менен жаз: бишкек	БИШКЕК
Кайталап жаз: Кыргызстан тоолуу өлкө.	Кыргызстан тоолуу өлкө.
Уланта жаз: Кыргыз Республикасы…	(continues the text)

Not trained, will produce nonsense: factual questions, arithmetic, reasoning, translation, anything in Russian or English.

---

Deployment: shipped as architecture qwen3, and why that is honest

This GGUF declares general.architecture = "qwen3". It shares no weights with any Qwen model and is not a fine-tune of one.

A bespoke architecture string requires landing an upstream llama.cpp PR before anyone can run the model. Instead, the model was built from the same components as Qwen3 — RMSNorm, NEOX-style RoPE, GQA, per-head QK-norm before RoPE, SwiGLU, tied embeddings — so llama.cpp's existing Qwen3 graph executes these weights natively and correctly.

The bar for calling that legitimate was token parity, not "it runs": at --temp 0, llama.cpp produces character-identical output to the reference PyTorch generate() across multiple prompts, ң included through the byte-level merges.

The character-level tokenizer is dressed as GPT-2 byte-level BPE. Each character token's string is its UTF-8 bytes mapped through the GPT-2 bytes→unicode table (к = 0xD0 0xBA → к); merges are one chain per multi-byte character; unused base-byte symbols pad the vocabulary to 341 so any input byte tokenises to something. Padding rows are zero vectors in the tied embedding, so their logits are exactly 0 and greedy decoding never selects them.

---

Three bugs worth reading about

The engineering value of this project is in what measurement caught. Three that shaped these files:

1. A frequency cutoff silently deleted a letter of the alphabet. Building the vocabulary with a count >= 200 threshold dropped capital Ң, which occurs 5 times in 10 million characters. Not a data error — Turkic phonotactics: /ŋ/ never begins a word, so the capital form appears only in all-caps headings. Which is exactly what official Kyrgyz forms look like. Nothing would have crashed; every future Ң would silently become <unk>. Fix: assert the alphabet, never infer it from statistics.

2. The corpus was contaminated with homoglyphs. Sampling at high temperature produced кѳйчщер — ѳ (U+0473 FITA), an archaic Russian letter abolished in 1918, not ө (U+04E9 BARRED O). The audit found 482 of them, plus 3,156 Latin o and 4,141 Latin e inside Cyrillic words. Total repaired: 1,010 characters, 0.0102% of the corpus — which is why nobody finds this by reading samples.

3. add_bos_token was wrong for the base model. The pretraining stream is a raw character river; token id 1 never appears as an input. Setting add_bos_token = true placed a never-seen embedding at position 0 — the position every later token attends to — and the model degenerated:

add_bos_token = true   →  "Кыргыз Республикасынын Кыргыз Республикасынын Кыргыз Респуб"
add_bos_token = false  →  "Кыргыз Республикасынын Кол жазмалар фондусунда калыптанган."

A repetition loop on the project's own headline example. The chat model, whose SFT does render every conversation starting with <bos>, requires the opposite setting. Both are now correct per-file. The general rule this project kept relearning: serve the format that was trained, byte-for-byte.

---

Limitations

Stated plainly, because a model card that hides these is worth less than one that names them.

• 5.7M parameters hold format, not knowledge. Factual questions produce fluent, confident, wrong answers. This is inherent to the scale, not a training defect.
• 256-token context. Chat quality degrades after roughly six turns as the window fills.
• Kyrgyz only. Trained on Kyrgyz Wikipedia. Russian and English input produce garbage, and much of the Latin alphabet is outside the 151-token vocabulary.
• Wikipedia register. The base model continues text in encyclopedic prose because that is all it has ever read.
• The base model has no EOS. It was never trained to stop; cap output length. (The chat model stops correctly.)
• Character-level. No subword structure, so it generalises orthographically but has no lexical prior.
• High-temperature sampling can reach padding tokens. Their logits are 0 and greedy decoding never selects them, but temperatures above ~1.5 can. A documented consequence of the tokenizer trick, not a surprise.
• Synthetic evaluation only for the wider stack. The vision components of the parent project are evaluated on generated documents; real-photograph numbers will be worse, and collecting them is the current work.

---

Training

Data	Kyrgyz Wikipedia, Leipzig Corpora kir_wikipedia_2021_100K
Preprocessing	NFC normalisation, homoglyph repair, forced-alphabet vocabulary
Precision	bf16 with gradient accumulation, EMA, WSD schedule
Optimiser	Muon (Newton–Schulz orthogonalisation) on hidden matrices + AdamW on embeddings, head and norms
Steps	3,000 base / 900 SFT (best-validation checkpoint)
Hardware	Apple M5 Pro, MPS backend

Measured in the parent project: Muon beat AdamW by −0.126 validation loss at equal steps; QK-norm earned its cost (+0.034 without it); z-loss slightly hurt at this scale; and MQA matched GQA at half the KV cache. Two of those cut against received wisdom, which is the argument for running the ablation instead of copying a config.

---

Licence and attribution

Apache 2.0. Training data is Kyrgyz Wikipedia via the Leipzig Corpora Collection (CC BY-NC 4.0 for the corpus distribution — the model weights are original work).

The architecture is component-compatible with Qwen3 by deliberate design; no Qwen weights, data, or code were used.

Built by Bakyt Djumabaev.