Qazwen-2B

Kazakh-focused instruction tuning. Compact adaptation. Open research.

Қазақ тілі · Нұсқауларды орындау · Шағын тілдік модель

Model · Run locally · Evidence · Data · Limitations

2.23B total parameters · 16.82M LoRA parameters · Research release v0.1

---

Qazwen-2B brings a Kazakh-focused instruction-tuning layer to Qwen3.5-2B. Published in the AbaiUniversity model collection, this compact adaptation investigates short Kazakh instructions, everyday messages, contextual questions, arithmetic and selected cultural topics.

The training recipe combines a focused Kazakh corpus with a 16.82-million-parameter LoRA adapter, preserving the frozen multilingual foundation. In the internal diagnostic comparison, the final adapter corrected selected arithmetic, reading and literature responses. Complete evaluation evidence accompanies the release.

This release includes the trained LoRA adapter, native tokenizer assets, an explicit inference loader, training settings and the complete responses from two small diagnostic comparisons. Base weights must be downloaded separately. Release status: research preview. Task-specific progress and generalization are reported separately in Evaluation, with the intended scope described under Limitations.

Қазақша

Qazwen-2B — Qwen3.5-2B негізінде қазақ тіліне бейімделген зерттеу моделі. Жоба қазақша нұсқауларды орындау, қысқа хабарламалар жазу және мәтіндегі ақпаратпен жұмыс істеу бағыттарын қамтиды. Ішкі тексерудің кейбір есептерінде, мәтінді түсіну және әдебиет тапсырмаларында ілгерілеу байқалды. Толық жауаптар, бағалау әдісі және модельдің шектеулері төменде берілген.

Model at a glance

Attribute	Specification
Release	Qazwen-2B v0.1 · final adapter, step 938
Foundation	Qwen/Qwen3.5-2B, official post-trained model
Pinned base revision	15852e8c16360a2fea060d615a32b45270f8a8fc
Native class	Qwen3_5ForConditionalGeneration
Loaded base parameters	2,213,241,664 unique parameters, with tied weights counted once
LoRA parameters	16,819,200 trainable parameters, stored in float32
Base + active adapter	2,230,060,864 unique parameters
Adapter download	67,332,688 bytes, excluding tokenizer/documentation
Adaptation scope	Text language-model linear layers; vision and embeddings frozen
Method	Fresh LoRA, rank 16, alpha 32, dropout 0.05
Training coverage	3,750 records × 2 epochs = 7,500 presentations; 938 optimizer updates
Evaluated generation	Text only; native non-thinking template; greedy decoding
License	Apache-2.0; upstream attribution retained

The parameter counts above come from the loaded native model in the recorded environment, rather than an estimate from the rounded “2B” name. They include the native wrapper's loaded parameters. This adapter does not establish improved image understanding; multimodal inputs were not evaluated.

Quick start

Use the native Qwen3.5 wrapper and attach the LoRA adapter explicitly. The adapter alone cannot generate text. Do not substitute an unrelated Qwen text architecture.

pip install -r requirements.txt
python infer.py "Мен бүгін шай емес, айран ішкім келеді. Мен не ішкім келеді? Бір сөзбен жауап бер."

Run these commands inside a downloaded copy of this repository. The loader uses the exact base revision above and the local adapter by default. For a Hub adapter:

python infer.py "Қысқа хабарлама жаз." --adapter AbaiUniversity/Qazwen-2B

Pin --adapter-revision to a release commit for reproducibility. infer.py defaults to float32, the evaluation precision. Base float32 weights alone require roughly 8.85 GB in decimal bytes, plus the adapter, activations and runtime overhead. Lower precision can be selected explicitly, but its numerical behavior has not been evaluated for this release. The packaged CLI is derived from the completed evaluator and syntax-checked; a separate GPU smoke test of this packaging script has not been run.

The tokenizer's declared maximum context is inherited from Qwen. Training examples in this run were at most 472 serialized tokens, under a 768-token admission cap. Long-context Kazakh performance is not established.

Training design

The objective was completion-only causal cross-entropy: prompt tokens were masked, the native end-of-message token was supervised, and example losses were equally weighted. The base model, vision encoder and embeddings were frozen. Only a fresh text LoRA adapter was optimized; previous project adapters were not loaded.

Setting	Value
Optimizer	AdamW
Initial learning rate	3 × 10⁻⁵
Schedule	5% warmup, cosine decay to 20% of initial learning rate
Effective batch	8; microbatch 1, gradient accumulation
Weight decay / gradient clipping	0.01 / 1.0
Precision	Float32 frozen base and adapters; float16 autocast with gradient scaling
Memory controls	SDPA; non-reentrant gradient checkpointing; supervised-position logits
Random seed	20261024

See training configuration, layer targets and data provenance. The admitted corpus contains 2,870 conversation/task records, 380 mathematics records, 200 culture/language records and 300 retention-replay records. It includes synthetic examples and externally sourced material; it is not an exclusively original, fully human-reviewed dataset. The raw corpus is not part of this release.

Evaluation and observed behavior

Internal 16-scenario diagnostic

The original base, the first-epoch adapter (step 469), and final Qazwen (step 938) were compared on the same 16 scenarios: 12 Kazakh and four Russian/English tasks. All literal answers were manually reviewed by an agent. This is a small internal DEV set, now inspected, not an independent public benchmark or native-speaker certification.

Variant	Completion cross-entropy on 16 validation examples ↓
Original Qwen3.5-2B	1.2316
Epoch 1 · step 469	0.6610
Qazwen · step 938	0.6238

These values are teacher-forced losses, not accuracy percentages or intelligence scores. The final adapter corrected the discount/change calculation and the author of «Көксерек» on particular diagnostic items. Both epochs improved selected meeting-update and reading tasks. Some instructions were already answered correctly by the original model. There were also regressions: an unsupported weekday was added, a translation changed “address” to “name”, and a requested question became an answer.

Five later Kazakh everyday prompts

Qazwen was compared with the pinned original 2B model and official Qwen3.5-4B on five new simple requests. The five prompts were sampled from a fixed 12-prompt pool with seed 20261024 before reviewing answers; exact TRAIN prompt matches were zero. This does not prove semantic independence or absence from base pretraining.

All three models correctly selected айран over tea. Qazwen failed the practical intent of the rain, lost-pen and out-of-stock bread requests, and repeated a phrase instead of offering natural support. The 4B control was relatively better on the bread request, but also had failures and truncated answers. This check did not confirm robust general conversation gains for Qazwen.

Both comparisons used float32, greedy decoding, enable_thinking=False, a 192-new-token cap and a 75-second per-answer limit. A capped answer is not treated as a complete response. All responses are preserved, without selecting retries:

• All 48 internal responses and review
• All 15 later responses and review
• Machine-readable internal responses
• Machine-readable later responses
• Comparison models and exact revisions

Examples: successes and failures

The following are literal final-adapter outputs, including errors. These examples illustrate observed behavior; they are not a balanced capability score.

A short message

Q: Әріптесім Динаға менің атымнан қысқа хабарлама жаз: бүгін файлды жібере алмаймын, себебі интернетім істемей тұр. Ертең сағат 11:00-ге дейін жіберемін. Екі сөйлем болсын.

A — Qazwen, verbatim:

Бүгін файлды жібере алмаймын, интернетім істемей тұр. Ертең сағат 11:00-ге дейін жіберемін.

Review: The base already answered this request correctly; this is retained behavior.

A worked arithmetic answer

Q: Бір билет 1200 теңге. Төрт билетке жалпы 25% жеңілдік берілді. 5000 теңге төлесем, қанша теңге қайтарылады? Есебін қысқа көрсет.

A — Qazwen, verbatim:

Билеттердің жалпы бағасы: 1200 × 4 = 4800 теңге. Жеңілдіктен кейін төлейміз: 4800 × 75 ÷ 100 = 3600 теңге. Қайтарылатын ақша: 5000 − 3600 = 1400 теңге.

Review: The final adapter corrected the answer on this internal item. This is not evidence of general mathematics superiority.

A literature fact

Q: «Көксерек» шығармасының авторы кім? Бір сөйлеммен жауап бер.

A — Qazwen, verbatim:

Мұхтар Әуезов.

Review: The author is correct on this item; broader literature competence has not been established.

A culture failure

Q: Наурыз көже деген не? Қысқа әрі түсінікті айт.

A — Qazwen, verbatim:

Наурыз — қыстағы ең қызық көже. Ол қыста өсіріледі, қыста жемісіп, қыста сатылады.

Review: This answer is factually wrong. It is included to make the release auditable.

Limitations and intended use

Qazwen is intended for reproducible research into Kazakh language adaptation, instruction tuning and compact multilingual models. It can serve as a baseline for improved data curation and independent evaluation.

Kazakh fluency, practical dialogue, clarification, translation fidelity and cultural facts remain unreliable. Shorter answers and reduced validation loss did not ensure better understanding on the later check. Broad mathematics, literature and multimodal competence have not been demonstrated. The model can hallucinate, confuse roles or intentions, repeat text and produce incomplete answers. Use human review for educational content and verify factual claims. It has not been validated for high-stakes decisions or autonomous actions.

Reproducibility and attribution

• Adapter SHA256: ab611b8d96b85e36cd94295fd885eba20dceba496a1397b0d2e9041a00771ffc
• Corpus and configuration fingerprints: release manifest.
• The publication copy replaces a temporary base-cache path with the public base ID and pins its revision; adapter tensor bytes are unchanged.
• Qwen3.5-2B was developed by the Qwen team. This project provides the additional Kazakh text adapter and its experimental documentation; it does not claim to have pretrained the foundation model.
• See NOTICE, LICENSE and data sources.

По-русски

Qazwen — экспериментальная казахская адаптация Qwen3.5-2B: LoRA после 938 шагов на 3 750 учебных примерах. На части внутренних заданий есть улучшения, однако новая короткая проверка не подтвердила надёжный свободный диалог. Публикуются адаптер, настройки, код запуска и реальные ответы с ошибками, чтобы результаты можно было проверить и использовать в дальнейших исследованиях.