Hailay/VEXMLM-TIGQA

VEXMLM — TIGQA

Tigrinya question answering fine-tuned from Hailay/VEXMLM, the vocabulary-extended XLM-R for Ge'ez-script languages.

Official implementation: https://github.com/hailaykidu/VEXMLM

Task	question-answering
Dataset	TIGQA
Language	Tigrinya
Architecture	XLMRobertaForQuestionAnswering
Base model	Hailay/VEXMLM
Vocabulary	280,002
Seeds published	42, 43, 44, 45, 46

This checkpoint performs poorly in absolute terms. Exact Match of 2.39 is near zero. TIGQA's test split contains only 67 questions, so the metric is both very hard and very high-variance. It is published for completeness and reproducibility of the paper's evaluation, not as a usable Tigrinya QA system. For Tigrinya extractive QA, see Hailay/VEXMLM-TiQuAD.

Five-seed benchmark evaluation

Fine-tuned independently under seeds 42–46 with one configuration (hash ce27cc194946) on an A100-PCIE-40GB. Reported as mean ± standard deviation over the five runs, on the dataset's test split.

Metric	Score
Exact Match	2.39 ± 0.82
F1	9.76 ± 0.97

These are the paper's verified results. They come from the five-seed evaluation described above — not from interactive use.

Interactive inference vs. benchmark

Benchmark evaluation is the five-seed measurement on the held-out test split, shown in the table above.

Interactive inference is what the usage example below performs: Supply a Tigrinya context and question; the model returns an extracted span. Predictions on arbitrary user input are demonstrations only and do not produce or reproduce the benchmark score.

Repository layout

Five independently fine-tuned checkpoints, one per seed. The reported benchmark score is the mean ± standard deviation over all five; no single seed is the "five-seed model."

seed-42/  seed-43/  seed-44/  seed-45/  seed-46/

Load a specific seed with the subfolder argument, as in the example below.

Fine-tuning

Fine-tuned from Hailay/VEXMLM, a vocabulary-extended XLM-R (280,002 subwords, 30,000 Ge'ez tokens merged into the SentencePiece model) after continued MLM pretraining.

Hyperparameter	Value
Max sequence length	256
Batch size	32
Epochs	4
Learning rate	2e-5
LR schedule	Linear decay, 10% warmup
Weight decay	0.01
Gradient clipping	1.0
Optimizer	AdamW (β₁ 0.9, β₂ 0.999, ε 1e-8)
Precision	bf16
Trainable parameters	All
Hardware	1× NVIDIA A100

Runs are bit-reproducible: enable_full_determinism, CUBLAS_WORKSPACE_CONFIG=:4096:8, dataloader_num_workers=0.

Usage

from transformers import AutoTokenizer, AutoModelForQuestionAnswering
import torch

repo = "Hailay/VEXMLM-TIGQA"
tokenizer = AutoTokenizer.from_pretrained(repo, subfolder="seed-42")
model = AutoModelForQuestionAnswering.from_pretrained(repo, subfolder="seed-42")
model.eval()

question = "ኤርትራ ኣብ ኣየናይ ክፍለ ዓለም ትርከብ?"
context  = "ኤርትራ ኣብ አፍሪቃ ክፍለ ዓለም እትርከብ ሃገር እያ።"
enc = tokenizer(question, context, return_tensors="pt", truncation=True, max_length=256)

with torch.no_grad():
    out = model(**enc)

start = out.start_logits.argmax()
end = out.end_logits.argmax()
print(tokenizer.decode(enc.input_ids[0][start:end + 1], skip_special_tokens=True))

Limitations

• Fine-tuned for Tigrinya on TIGQA only; performance on other languages, domains or label schemes is not characterised.
• The base model covers Amharic and Tigrinya; other Ge'ez-script languages were not part of pretraining.
• Corpora are drawn largely from religious and news domains, and the model may reflect those distributions and any biases in them.
• Single-configuration study: no hyperparameter search was performed, and baseline comparisons in the paper are single-seed.

Reproducibility

The fine-tuning launcher, evaluation code and per-run result records are in the official repository: https://github.com/hailaykidu/VEXMLM

sbatch scripts/slurm_stage2_spm_seeds.sh    # 6 tasks × 5 seeds
python3 evaluation/export_spm_results.py    # regenerates the metrics table

Citation

@inproceedings{teklehaymanot2026vexmlm,
  title     = {Expanding the Lexicon of Ge'ez Based African Languages:
               A Comparative Study of Amharic and Tigrinya},
  author    = {Teklehaymanot, Hailay Kidu and Yadeta, Gebregziabihier and
               Nejdl, Wolfgang},
  booktitle = {Proceedings of the Workshop on Language Models for
               Underserved Communities (LM4UC) at IJCAI},
  year      = {2026}
}

Accepted at the LM4UC Workshop, IJCAI 2026.

License

Apache 2.0, following xlm-roberta-base.