Ikinyamakuru: Kinyarwanda news section classifier

This model files a Kinyarwanda news article under one of the 14 KinNews sections. It is a character n-gram TF-IDF vectorizer followed by logistic regression with balanced class weights, trained on the deduplicated KinNews corpus for a Machine Learning Techniques I summative at the African Leadership University.

The live desk runs this model in the browser at huggingface.co/spaces/Gkea/Ikinyamakuru. The data preparation, the experiments, and the full write-up are at github.com/Keaane/Ikinyamakuru.

Sections

Label	Kinyarwanda	Covers
politics	politiki	Government, elections, and public affairs.
sport	imikino	Matches, clubs, and competitions.
economy	ubukungu	Business, markets, and livelihoods.
health	ubuzima	Hospitals, disease, and public health.
entertainment	imyidagaduro	Music, film, and celebrity.
history	amateka	The past, memory, and commemoration.
technology	ikoranabuhanga	Devices, telecoms, and digital services.
tourism	ubukerarugendo	Travel, places, and hospitality.
culture	umuco	Arts, language, and custom.
fashion	imideri	Clothes, style, and beauty.
religion	iyobokamana	Faith, churches, and religious life.
environment	ibidukikije	Land, climate, and conservation.
education	uburezi	Schools, exams, and learning.
relationship	urukundo	Family, marriage, and personal life.

How to use

import json
import joblib
from huggingface_hub import hf_hub_download

repo = "Gkea/ikinyamakuru"
model = joblib.load(hf_hub_download(repo, "char_balanced.joblib"))
config = json.load(open(hf_hub_download(repo, "config.json")))

article = "Amavubi yatsinze Uganda ibitego bibiri ku busa mu mukino wa gicuti i Kigali"
scores = model.predict_proba([article])[0]
best = scores.argmax()
section = model.classes_[best]
filed = scores[best] >= config["threshold"]
print(section, round(float(scores[best]), 3), "filed" if filed else "left for an editor")

The pipeline needs scikit-learn 1.9.1 and NumPy 2.x, the versions it was saved with. A joblib file is a pickle and can run code when it loads, so load it only from a source you trust.

The weights/ folder holds the same model as plain arrays with no pickle: the vocabulary, the IDF weights, the coefficients (little-endian float64, one row per section in the order of weights/meta.json, each row as long as the vocabulary), and the intercepts. The browser desk (web/ngram.js) reads these files and gives the same answers as the pipeline on every development and test article.

Below a top score of 0.5743, the desk does not file the article and leaves it for an editor. config.json holds that cutoff with the section names.

Training data

KinNews (Niyongabo, Qu, Kreutzer, and Huang, 2020) has 21,268 Kinyarwanda articles from Rwandan news sites and newspapers, in 14 sections. This model uses the raw release. Exact duplicates were removed inside each official file (9,384 training rows and 790 test rows), and the 3 training articles that also appear in the test file were dropped. That leaves 6,864 training articles, 763 development articles (a stratified 10 percent of training, seed 42), and 3,464 test articles. The model text is the title, a line break, and the body, with stopwords and punctuation kept.

Training procedure

The vectorizer uses character 2 to 4 grams inside word boundaries (char_wb), lowercased, with min_df 2, at most 80,000 features, sublinear tf, and l2 normalisation. The classifier is saga logistic regression with C = 16, chosen from {0.25, 1, 4, 16} on development macro-F1, and class_weight="balanced". Development macro-F1 selected this system over six others: a majority baseline, word TF-IDF, the unbalanced character model, and three AfroXLMR-mini runs.

Evaluation

Development macro-F1 is 0.692. The test split was scored once, after the selection: accuracy 0.734, macro precision 0.648, macro recall 0.594, and macro-F1 0.583. With the cutoff, the model files 73.8 percent of test articles, and 84.7 percent of those are correct.

Section	Precision	Recall	F1	Test articles
politics	0.496	0.825	0.620	315
sport	0.961	0.918	0.939	744
economy	0.543	0.724	0.621	174
health	0.554	0.621	0.586	272
entertainment	0.819	0.797	0.808	774
history	0.219	0.636	0.326	11
technology	0.735	0.699	0.716	103
tourism	0.731	0.250	0.373	76
culture	0.840	0.304	0.447	138
fashion	0.581	0.549	0.565	91
religion	0.833	0.645	0.727	256
environment	0.719	0.460	0.561	100
education	0.273	0.043	0.074	70
relationship	0.762	0.838	0.798	340

Limitations

Some labels are noisy. 156 repeated training articles carry more than one section label, mostly entertainment and sport, and some test articles are labelled against their content, such as football stories filed under entertainment.

Tourism, fashion, and environment have 31 to 34 training articles each. Education is the weakest section (F1 0.07), and most of its misses are filed as relationship.

The test file has a different mix of sections from training, so scores on new sources may differ again. The scores are softmax outputs and are not calibrated probabilities.

The model learned from whole edited news articles. A headline alone gets a weaker score, and it has not been tested on social media posts or transcribed speech. The saga solver reached its iteration cap (max_iter 220) before reporting convergence.

Licence

The model is released under the MIT licence. The KinNews articles remain the copyright of their publishers. The model files hold character fragments of 2 to 4 letters and their weights, with no article text.

Citation

Niyongabo, R. A., Qu, H., Kreutzer, J., and Huang, L. (2020). KINNEWS and KIRNEWS: Benchmarking cross-lingual text classification for Kinyarwanda and Kirundi. Proceedings of COLING 2020, 5507-5521.