Biatus/nllb-finetuned-sw-ki-bidirectional

NLLB-200-600M QLoRA: Bidirectional Swahili ↔ Kikuyu Translation

Fine-tuned PEFT LoRA adapter for facebook/nllb-200-distilled-600M specialized for bidirectional machine translation between Swahili (swh_Latn) and Kikuyu / Gĩkũyũ (kik_Latn).

---

Model Summary

• Base Model: facebook/nllb-200-distilled-600M
• Adapter Type: QLoRA (4-bit NF4 base model + LoRA adapters)
• Target Modules: q_proj, k_proj, v_proj, out_proj
• LoRA Parameters: Rank r=15, alpha=30, Dropout = 0.05
• Trainable Parameters: 4,423,680 (0.998% of total parameters)
• Primary Languages:
  • Swahili (sw / swh_Latn)
  • Kikuyu (kik / kik_Latn)

---

Evaluation Results

Evaluated on held-out test splits after 5 full training epochs (24,110 steps):

Metric	Score	Note
Validation Loss	1.468	Stable cross-entropy convergence
SacreBLEU	27.41	+14 BLEU improvement over multilingual baselines
chrF++ (word_order=2)	53.34	High character- and word-level fidelity

---

Translation Examples

Swahili → Kikuyu (swh_Latn → kik_Latn)

• Input (SW): Wakulima wanashauriwa kupanda mahindi wakati wa mvua.

• Output (KIK): Arĩmi nĩmaratarwo kũhanda mbembe hĩndĩ ya mbura.

  
  

• Input (SW): Mchezo wa kandanda unapendwa sana.

• Output (KIK): Mũthako wa mũbira wa magũrũ nĩ wendetwo mũno.

Kikuyu → Swahili (kik_Latn → swh_Latn)

• Input (KIK): Mũgũnda ũyũ ũrĩ na ng'ombe cia kũruta ĩrĩa rĩingĩ.

• Output (SW): Shamba hili lina ng'ombe wa kutoa maziwa mengi.

  
  

• Input (KIK): Nĩ wona indo ciothe ĩrĩa ũrabatara.

• Output (SW): Umepata vifaa vyote unavyohitaji.

---

Setup & Inference

1. Installation

pip install torch transformers peft bitsandbytes sentencepiece accelerate

2. Python Inference Script

import torch
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel

# Model Identifiers
BASE_MODEL_NAME = "facebook/nllb-200-distilled-600M"
ADAPTER_REPO_ID = "Biatus/nllb-finetuned-sw-ki-bidirectional"

# 1. Load Tokenizer
tokenizer = AutoTokenizer.from_pretrained(ADAPTER_REPO_ID)

# 2. Configure 4-bit NF4 Quantization (QLoRA)
compute_dtype = torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float16
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=compute_dtype
)

# 3. Load Base Model and Inject LoRA Adapter
base_model = AutoModelForSeq2SeqLM.from_pretrained(
    BASE_MODEL_NAME,
    quantization_config=bnb_config,
    device_map="auto"
)
model = PeftModel.from_pretrained(base_model, ADAPTER_REPO_ID)
model.eval()

def translate(text: str, direction: str = "sw2kik") -> str:
    """
    Translates text between Swahili and Kikuyu.
    direction: 'sw2kik' (Swahili -> Kikuyu) or 'kik2sw' (Kikuyu -> Swahili)
    """
    if direction == "sw2kik":
        src_lang, tgt_lang = "swh_Latn", "kik_Latn"
    else:
        src_lang, tgt_lang = "kik_Latn", "swh_Latn"

    tokenizer.src_lang = src_lang
    inputs = tokenizer(text, return_tensors="pt").to(model.device)
    tgt_token_id = tokenizer.convert_tokens_to_ids(tgt_lang)

    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            forced_bos_token_id=tgt_token_id,
            max_length=128,
            num_beams=4
        )

    return tokenizer.decode(outputs[0], skip_special_tokens=True).strip()

# --- Test Swahili to Kikuyu ---
sw_text = "Wakulima wanashauriwa kupanda mahindi wakati wa mvua."
kik_translation = translate(sw_text, direction="sw2kik")
print(f"[SW -> KIK]\nInput : {sw_text}\nOutput: {kik_translation}\n")

# --- Test Kikuyu to Swahili ---
kik_text = "Mũgũnda ũyũ ũrĩ na ng'ombe cia kũruta ĩrĩa rĩingĩ."
sw_translation = translate(kik_text, direction="kik2sw")
print(f"[KIK -> SW]\nInput : {kik_text}\nOutput: {sw_translation}\n")

---

Training Configuration

• Dataset Size: 154,274 clean parallel pairs (bidirectional)
• Validation Split: 8,570 pairs
• Epochs: 5
• Optimizer: paged_adamw_8bit
• Learning Rate: 2 * 10^-4 (cosine schedule, 10% warmup)
• Batch Size: 8 per device * 4 gradient accumulation steps (effective batch size: 32)
• Sequence Length: 128 subword tokens

---

Limitations & Ethical Considerations

• The model is optimized for colloquial, agricultural, and general conversational text in Swahili and Kikuyu.
• Specialized medical, highly legal, or obscure dialect terms may require verification by native speakers.