AmplifiedAccess/Swahili-nllb-600m

Swahili NLLB 600M

A LoRA fine-tune of facebook/nllb-200-distilled-600M for English → Swahili translation — the smaller sibling of Swahili-nllb-1.3b, using NLLB's purpose-built translation architecture (encoder-decoder, SentencePiece) rather than a general-purpose chat LLM.

Highlights

• BLEU 35.97 on English→Swahili FLORES devtest
• chrF++ 61.26
• Beats the CraneAILabs Swahili-Gemma-1B reference (27.6 BLEU / 56.8 chrF++) on both metrics, at under half the parameters
• LoRA adapter only 8.65M trainable parameters (~1.4% of the 615M base model)

Results

FLORES devtest, English→Swahili:

Model	BLEU	chrF++
CraneAILabs Swahili-Gemma-1B (reference)	27.6	56.8
This model	35.97	61.26
Swahili-nllb-1.3b (larger sibling)	38.74	63.37

Usage

This repo holds a LoRA adapter, not merged weights — load the base NLLB model first, then apply the adapter.

import torch
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
from peft import PeftModel

base_model_id = "facebook/nllb-200-distilled-600M"
adapter_id = "AmplifiedAccess/Swahili-nllb-600m"

tokenizer = AutoTokenizer.from_pretrained(base_model_id, src_lang="eng_Latn")
model = AutoModelForSeq2SeqLM.from_pretrained(base_model_id)
model = PeftModel.from_pretrained(model, adapter_id)
model.eval()

text = "A community officer can help residents complete forms, check eligibility, and even process applications offline."
inputs = tokenizer(text, return_tensors="pt")
target_lang_id = tokenizer.convert_tokens_to_ids("swh_Latn")

with torch.no_grad():
    outputs = model.generate(**inputs, forced_bos_token_id=target_lang_id, max_new_tokens=128)

print(tokenizer.batch_decode(outputs, skip_special_tokens=True)[0])

Training details

Data

Same curated parallel-corpus mixture and caps used for the 1.3B sibling:

Source	Notes
Rogendo/English-Swahili-Sentence-Pairs	Capped at 60k
MAFAND-MT (en-swa)	Human-translated news domain (Masakhane), uncapped
Sunbird SALT (text-all)	Full/uncapped
Svngoku/Inkuba-Swahili-MMT	Capped at 30k

~146K parallel pairs total after filtering.

Configuration

Parameter	Value
Base model	facebook/nllb-200-distilled-600M
Method	LoRA (no quantization — full-precision base)
LoRA rank / alpha	16 / 32
Language pair	eng_Latn → swh_Latn (one direction)
Epochs	3
Effective batch size	32 (4 per device × 8 gradient accumulation)
Learning rate	3e-5 (cosine schedule, 3% warmup)
Max sequence length	128
Optimizer	AdamW (8-bit)
Training steps	13,660 / 13,662
Final training loss	11.81

Trained on a single Kaggle T4 GPU. Evaluated on FLORES devtest via sacrebleu after training (BLEU/chrF++ above); the reported training loss reflects raw seq2seq cross-entropy and is not directly comparable to the causal-LM losses reported for our Gemma-based models — treat the FLORES BLEU/chrF++ scores above as the quality signal for this model.

Limitations

• Trained and evaluated only for English→Swahili; the reverse direction is untested and not claimed.
• Evaluated on FLORES devtest only; performance outside that domain (news, casual text, dialectal Swahili) is unverified.
• LoRA adapter requires the base NLLB-600M model at inference time (~2.4 GB combined).

Intended use

• English → Swahili machine translation on more constrained hardware than the 1.3B variant
• Research on low-resource African language NLP with NLLB-family models

Acknowledgments

• Masakhane for the MAFAND-MT parallel corpus
• Sunbird AI for the SALT dataset
• Meta AI for the NLLB-200 model family

Framework versions

• PEFT 0.19.1
• Transformers 5.0.0
• PyTorch 2.10.0