Psycodem/gemma-4-e4b-lora-diacritization

Gemma 4 E4B-it + LoRA for Arabic Diacritisation

LoRA adapters that restore Arabic diacritics (tashkeel) on top of google/gemma-4-E4B-it. Three adapters are included, trained on 10%, 30% and 50% of the corpus, so the effect of training data volume can be read directly.

Part of DiacriticS, a contamination-controlled study of open-weights models on this task — project site · code.

Which subfolder to use

Each fraction is a subfolder of this repo. 50pct is the strongest.

Subfolder	Training rows	DER (CE)	DER (no CE)	WER (CE)	WER (no CE)
10pct	104,270	3.15	2.68	7.28	4.47
30pct	312,809	2.98	2.54	6.82	4.12
50pct	521,349	2.81	2.38	6.54	3.96

Scored on the full 1,200-paragraph SadeedDiac-25 benchmark with Evaluation_Functions_Corrected.py. Percentages; lower is better. CE = sentence-final case ending (i'rab).

All three splits

DER / WER with case endings.

Subfolder	Train sample (500)	Tashkeela test (2,485)	SadeedDiac-25 (1,200)
10pct	1.23 / 3.24	12.96 / 16.2	3.15 / 7.28
30pct	1.81 / 3.54	12.73 / 15.73	2.98 / 6.82
50pct	1.76 / 3.19	12.57 / 15.41	2.81 / 6.54

The Tashkeela test split scores worse than the benchmark because it inherits the residual annotation noise of the source corpus, not because the model does worse on it. SadeedDiac-25 is expert-reviewed and balanced 50/50 between Modern Standard and Classical Arabic.

Usage

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

BASE     = "google/gemma-4-E4B-it"
ADAPTER  = "Psycodem/gemma-4-e4b-lora-diacritization"
FRACTION = "50pct"          # "10pct" | "30pct" | "50pct"

tok = AutoTokenizer.from_pretrained(BASE, trust_remote_code=True)
if tok.pad_token is None:
    tok.pad_token = tok.eos_token

model = AutoModelForCausalLM.from_pretrained(
    BASE, dtype=torch.bfloat16, device_map="auto", trust_remote_code=True)
model = PeftModel.from_pretrained(model, ADAPTER, subfolder=FRACTION)
model = model.merge_and_unload()
model.eval()

SYSTEM_PROMPT = (
    "أنت نظام متخصص في التشكيل الآلي للنصوص العربية. "
    "مهمتك إضافة الحركات (التشكيل) الصحيحة إلى النص العربي المُدخل دون تغيير الكلمات أو ترتيبها، "
    "مع مراعاة السياق النحوي والصرفي الكامل للجملة."
)

def diacritize(text):
    msgs = [{"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": text}]
    try:
        prompt = tok.apply_chat_template(msgs, tokenize=False,
                                         add_generation_prompt=True, enable_thinking=False)
    except TypeError:
        prompt = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
    enc = tok(prompt, return_tensors="pt").to(model.device)
    with torch.no_grad():
        out = model.generate(**enc, max_new_tokens=512, do_sample=False,
                             pad_token_id=tok.pad_token_id)
    return tok.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True).strip()

print(diacritize("ذهب الطالب إلى المدرسة"))

The prompt must match the one above — it is what the adapter was trained against, and a different instruction degrades output in ways that look like a bad model rather than a harness mistake. Decode greedily (do_sample=False); the reported numbers assume it. If you batch, set tok.padding_side = "left".

Training

Trained on nested subsets of Misraj/Sadeed_Tashkeela, drawn with a fixed shuffle seed so the 10% subset is contained in the 30%, and that in the 50%. The corpus has a measured 0.4% overlap with the benchmark, which is what makes the benchmark number meaningful.

Method	LoRA, base weights frozen in bf16 (not QLoRA)
Rank / alpha / dropout	16 / 32 / 0.05
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Adapted matrices	258
Effective batch	96 (4 x 8 x 3 GPUs)
Learning rate	2e-4, cosine
Warmup	5% of total steps
Epochs	1
Max sequence length	1024
Seed	42
Hardware	3× A100 80GB, DDP via torchrun

Both base models in the study share this configuration exactly, so differences between them reflect architecture and data volume rather than tuning. The warmup ratio of every run was verified after the fact from its logged learning-rate curve.

Limitations

Sentence-final case endings (i'rab) and Classical Arabic remain the dominant error sources, as they are for every system in the study. These adapters were trained for one epoch on a subset of a single corpus and are not expected to transfer to dialectal Arabic or to Quranic orthography with its additional annotation marks.

Citation

If you use these adapters, please cite the benchmark and corpus they build on:

@article{sadeed2025,
  title  = {Sadeed: Advancing Arabic Diacritization Through Small Language Model},
  author = {Aldallal, Z. and Chrouf, S. and Hennara, K. and Hamed, M. M. and
            Hreden, M. and AlModhayan, S.},
  journal = {arXiv preprint arXiv:2504.21635},
  year   = {2025}
}