Gemma-4-E4B-it β Tajik adaptation (CPT β SFT β tool-calling)
Saidzoda Lab. One model line, trained 2026-09-22 β 2026-09-25 on 6ΓH100.
Base: google/gemma-4-E4B-it. Adapters are LoRA; only the final tool-calling stage is
published as a full merged model.
Contents
| path | what | size | format |
|---|---|---|---|
cpt-lora/ |
stage 1 β Tajik continual pretraining | 3.33 GB | LoRA adapter (r64, Ξ±64, rslora; embed_tokens+lm_head trained) |
sft-lora/ |
stage 2 β Tajik supervised fine-tune | 325 MB | LoRA adapter (r32) |
tool-full/ |
stage 3 β tool-calling, full merged model | 15 GB | safetensors |
Each LoRA applies to the output of the previous stage, not to the stock base:
base β cpt-lora β (merge) β sft-lora β (merge) β tool-full.
So sft-lora is not loadable on stock E4B β it expects the CPT-merged model.
Measured results
Frozen benches, method = log-likelihood, identical across every row so the deltas are comparable. Full sets (no subsampling).
| stage | belebele-tgk (900) | tajik-curated (657) |
|---|---|---|
stock gemma-4-E4B-it |
37.8% (340) | 28.3% (186) |
| + CPT | 63.8% (574) | 44.9% (295) |
| + SFT | 72.1% (649) | 53.4% (351) |
| + tool (this repo's full model) | 72.6% (653) | 53.7% (353) |
belebele 37.8 β 72.6 (+34.8 pp), curated 28.3 β 53.7 (+25.4 pp). CPT alone contributes +26.0 / +16.6 β the largest single step, because stock E4B is not competent in Tajik (37.8% against a 25% chance floor on a 4-choice benchmark).
Read before using the numbers
- These are log-likelihood scores. They are not comparable to chat/generative-mode scores, and LL generally understates an instruction-tuned model's practical performance.
- The CPT stage ran 1.5 of 2 planned epochs (stopped at step 26,500/35,332 by owner decision, mid-cosine at 16% of peak LR). It is not an annealed endpoint. Loss was still improving when stopped.
- The tool-calling stage adds +0.5/+0.3 on these MCQ benchmarks, which is inside noise β it targets tool-call formatting, not knowledge, so these benches are not the right measure of it.
- No GRPO stage. GRPO fails on E4B with
ValueError: Target module Gemma4ClippableLinear(...) is not supportedβ PEFT cannot wrap E4B's non-nn.Linearwrapper modules. Unresolved at publication.
Training
| stage | data | steps | seq | LR | schedule |
|---|---|---|---|---|---|
| CPT | Tajik text corpus | 26,500 / 35,332 | 2048 packed | 5e-5 | cosine |
| SFT | Tajik instruction set (1,112,888 rows) | 37,097 (2 ep) | 1024 | 1e-5 | cosine |
| tool | tool-call set (176,522 rows) | 2,705 (1 ep) | 1024 | 2e-5 | cosine |
Global batch 60 on all stages (6-GPU DDP). bf16, max_grad_norm 0.5.
Gated (manual) β access on request.