AfriBERTa-siSwati

Fine-tuned AfriBERTa for the siSwati language of the Kingdom of Eswatini. Developed by AI Eswatini.

Model Description

This model adapts AfriBERTa (Ogueji et al., 2021) to siSwati through continued pre-training on the SADILAR EN-SS Bilingual Corpus using Masked Language Modelling (MLM).

Training Data

• Dataset: SADILAR EN-SS Bilingual Corpus v1.0.3
• Sentences: 114,839 parallel English-siSwati pairs
• siSwati words: 1,423,414
• Licence: CC BY 4.0

Training Configuration

• Base model: castorini/afriberta_large
• Epochs: 10
• Batch size: 32
• Learning rate: 2e-5
• Hardware: Kaggle Tesla T4 (15.6 GB)
• Training time: ~5 hours

Results

Model	Perplexity
mBERT (baseline)	47.83
AfriBERTa (no fine-tuning)	35.59
AfriBERTa-siSwati (this work)	7.27

• Perplexity reduction: 84.8% vs mBERT baseline
• Loss reduction: 42.9%
• No overfitting detected (max train-val gap: 0.365)

Usage

from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("KhanyiTapiwa00/afriberta-siswati")
model = AutoModelForMaskedLM.from_pretrained("KhanyiTapiwa00/afriberta-siswati")

Citation

@misc{AIEswatini2026siswati,
  title     = {AfriBERTa-siSwati: Fine-tuning AfriBERTa for siSwati},
  author    = {Magagula, Tapiwa and [Partner Name]},
  year      = {2026},
  publisher = {HuggingFace},
  url       = {https://huggingface.co/KhanyiTapiwa00/afriberta-siswati}
}

Acknowledgements

• SADiLaR, CTexT (North-West University), Department of Sport Arts and Culture
• Ogueji et al. for AfriBERTa