Omballa/nllb-200-swa-dholuo

Model Card for Model ID

A fine-tuned, vocabulary-trimmed variant of NLLB-200-distilled-600M for Swahili↔Dholuo (Luo) translation. This checkpoint is optimized for natural, conversational Dholuo rather than formal/news-register text, and is the strongest of several experimental variants on in-domain, everyday-language evaluation — while underperforming on the more formal FLORES-style benchmark.

Model Details

Model Description

This model is part of a low-resource machine translation effort targeting Dholuo, an East African language with limited parallel data. Starting from facebook/nllb-200-distilled-600M, the vocabulary was trimmed from ~256k to ~38.5k tokens to reduce model size and better fit the Swahili–Dholuo language pair, followed by denoising pretraining on monolingual Dholuo text and supervised fine-tuning on gold parallel data.

This particular checkpoint reflects a direct gold-data fine-tune (as opposed to staged gold+silver training variants also tested in the project). It performs best on natural, conversational Dholuo — the register most relevant to real-world use — but its performance drops on the FLORES+ evaluation set, which is written in a more formal, literary register that is underrepresented in the gold training data.

• Developed by: William Brandon Omballa (independent researcher, Kenya)
• Funded by : Self-funded / independent research
• Shared by : William Brandon Omballa
• Model type: Sequence-to-sequence neural machine translation (Transformer, NLLB architecture, vocabulary-trimmed)
• Language(s) (NLP): Swahili (sw), Dholuo/Luo (luo)
• License: Apache 2.0
• Finetuned from model : facebook/nllb-200-distilled-600M

Uses

Direct Use

Swahili↔Dholuo translation for conversational, everyday text — chat messages, spoken-style transcripts, informal communication. This is the primary intended use case and where the model performs best.

Downstream Use [optional]

Can serve as a backbone for downstream Dholuo NLP applications (e.g., voice assistants, chatbots, community communication tools) where conversational fluency matters more than formal register accuracy.

Out-of-Scope Use

Not recommended for formal, literary, or news-register translation (e.g., FLORES-style text), where evaluation shows a measurable performance gap. Not intended for legal, medical, or other high-stakes translation without human review, given the low-resource nature of Dholuo data.

Bias, Risks, and Limitations

• Register mismatch: The model excels on conversational/custom benchmark data but underperforms on the FLORES dataset, indicating a gap in formal-register generalization.
• Low-resource language risk: Dholuo is a low-resource language; training data (gold, silver/back-translated, and monolingual) may not fully represent dialectal variation.
• Custom benchmark caveat: Strong results are partly measured against a personal, native-speaker-curated 80-sentence benchmark, which — while a meaningful conversational-fluency signal — is small and may not generalize as broadly as a larger standardized test set.

Recommendations

Users should evaluate this model on data representative of their actual use case before deployment, particularly if working with formal or written registers where FLORES-style performance is more representative. Human review is recommended for any high-stakes use.

How to Get Started with the Model

Use the code below to get started with the model.

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model_id = "Omballa/nllb-200-swa-dholuo"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id)

# Swahili -> Dholuo
text = "Habari, unaendeleaje?"
tokenizer.src_lang = "swh_Latn"
inputs = tokenizer(text, return_tensors="pt")

translated_tokens = model.generate(
    **inputs,
    forced_bos_token_id=tokenizer.convert_tokens_to_ids("luo_Latn"),
    max_length=128,
)
print(tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0])

Training Details

Training Data

Training Data

• ~365k monolingual Dholuo sentences (denoising pretraining)
• ~200k parallel Swahili–Dholuo sentences (OPUS)
• ~106,079 back-translated (silver) parallel sentences
• ~40k gold parallel sentences (KenCorpus)
• This checkpoint was fine-tuned directly on the gold parallel set (as opposed to staged gold+silver variants also tested)

Training Procedure

1. Vocabulary trimming of NLLB-200-distilled-600M from ~256k to ~38.5k tokens
2. Denoising pretraining on monolingual Dholuo data
3. Supervised fine-tuning on gold parallel corpus
4. Evaluation against Google Translate and FLORES+ benchmarks, plus a custom native-speaker benchmark

Preprocessing [optional]

[More Information Needed]

Training Hyperparameters

• Training regime: fp16 mixed precision
• Bidirectional training (Swahili→Dholuo and Dholuo→Swahili)
• Early stopping tracked via chrF

Speeds, Sizes, Times [optional]

[More Information Needed]

Evaluation

Testing Data, Factors & Metrics

Testing Data

• In-domain test set (held-out from training pipeline)
• Custom 80-sentence native-speaker conversational benchmark
• FLORES+ benchmark (formal/literary register)

Factors

Register/domain (conversational vs. formal/literary text) is the key disaggregating factor — performance differs substantially between the two.

Metrics

BLEU and chrF, with chrF used as the primary tracked metric during training and evaluation.

Results

• Foundational pipeline: chrF improved from ~29 (base model) to ~50.7 (fine-tuned) on the in-domain test set
• Custom native-speaker benchmark: this direct gold fine-tune scored chrF 56.13, outperforming staged gold+silver variants and far ahead of Google Translate (chrF 43.26)
• FLORES+ evaluation: underperforms relative to its conversational-benchmark results — the model's strength is conversational Dholuo, not formal-register text

Summary

This model is the strongest variant tested for natural, conversational Dholuo translation, substantially outperforming Google Translate in that domain. It should not be used as a proxy for formal-register translation quality, where FLORES+ results are weaker.

Model Examination [optional]

[More Information Needed]

Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

• Hardware Type: [More Information Needed]
• Hours used: [More Information Needed]
• Cloud Provider: [More Information Needed]
• Compute Region: [More Information Needed]
• Carbon Emitted: [More Information Needed]

Technical Specifications [optional]

Model Architecture and Objective

NLLB-200-distilled-600M architecture with a trimmed vocabulary (~256k → ~38.5k tokens), fine-tuned for Swahili↔Dholuo translation.

Compute Infrastructure

[More Information Needed]

Hardware

[More Information Needed]

Software

Google Colab; Hugging Face Transformers

Citation [optional]

BibTeX:

[More Information Needed]

APA:

[More Information Needed]

Glossary [optional]

• chrF: Character n-gram F-score, a machine translation evaluation metric well-suited to morphologically rich, low-resource languages.
• Silver data: Back-translated or automatically generated parallel data, as opposed to human-curated "gold" data.
• FLORES / FLORES+: A standardized multilingual evaluation benchmark, generally written in a formal/literary register.

More Information [optional]

[More Information Needed]

Model Card Authors [optional]

William Brandon Omballa

Model Card Contact

https://www.linkedin.com/in/brandon-omballa/