DariusTheGeek/mhqa-itu-adapters

MHQA · ITU · Zindi Challenge — Generation Adapters

DariusTheGeek/mhqa-itu-adapters · the trained-weight component of the winning solution to the Multilingual Health Question Answering in Low-Resource African Languages Challenge (ITU), hosted on Zindi. Data by the HASH consortium (Hub for AI in Maternal, Sexual & Reproductive Health).

This repo hosts 3 LoRA adapters (the generation members). The full retrieval + generation + judge system, and one-command reproduction, live in the code repo: Multilingual-Health-QA-ITU-Zindi-Challenge on GitHub.

⚠️ Not medical advice. This is a research artifact built for a benchmark competition. It is not a medical device, is not for clinical use, and its outputs must not be used to diagnose, treat, or make health decisions. Sexual/reproductive-health content may be inaccurate, incomplete, or unsafe.

---

Model summary

Reproduces	submission sub_v40 · public LB 0.728509
Metric	0.37·ROUGE-1 + 0.37·ROUGE-L + 0.26·LLM-judge (micro-averaged, unicode-aware)
Languages	English, Kiswahili, Luganda, Akan/Twi, Amharic (8 country×language subsets)
What's here	3 LoRA adapters (4.7 GB) — the generation component
Adapter type	PEFT/LoRA on Gemma-4-31B-it and MedGemma-27B-text-it

The three adapters

adapter	base model	role	LoRA config	trained on
gen7454	google/gemma-4-31B-it	primary generator	r64 / α128 / dropout 0 · all linear projections · 2 epochs	all 8 subsets
raft_r1	merged gen7454	RAFT self-distillation member	r32 / α64 · lr 5e-5 · 1 epoch	Aka_Gha, Eng_Gha
mg_2226	google/medgemma-27b-text-it	MedGemma member	r64 / α128 · lr 1e-4 · 2 epochs	Aka_Gha, Eng_Gha

raft_r1 was trained on top of the merged gen7454 model (RAFT round-1), not on the raw base.

---

Architecture & approach

The competition is scored 0.37·ROUGE-1 + 0.37·ROUGE-L + 0.26·judge, so ~74 % of the score is lexical overlap with a hidden gold answer. The test set splits into two regimes:

• Retrieval subsets (~60 %: Eng_Uga, Eng_Ken, Eng_Eth, Swa_Ken, Lug_Uga) — the gold answer is almost always a verbatim reuse of a Train+Val answer-bank entry, already in the candidate pool. The job is selecting the canonical twin, done by LightGBM selectors that use a fine-tuned cross-encoder as a feature (not an argmax chooser). These are trained inline and deterministic — not weight files.
• Generation subsets (~37 %: Aka_Gha, Eng_Gha) — golds are original/local, so answers are generated. The 3 LoRA adapters in this repo each produce a K36 sample pool; a cross-model ROUGE-MBR medoid picks the consensus answer.

A final LLM-judge pass (gemma-4-31B-it) reverts off-topic picks (27 cached decisions).

What is a weight vs. deterministic code:

component	form	where
3 generation adapters	trained LoRA weights	this repo
cross-encoder reranker (CE-ft)	trained inline per fold, discarded; frozen feature outputs shipped	dataset repo (mhqa-itu-artifacts)
3 LightGBM selectors	trained inline each run, deterministic	code (GitHub)
LLM judge	27 cached revert decisions, replayed	dataset repo

---

Intended use

• In scope: reproducing / validating the competition result; research on retrieval-vs-generation for low-resource multilingual health QA; a baseline for the 8 named country×language subsets.
• Out of scope: any clinical, diagnostic, or patient-facing use; generating medical advice; deployment as a health chatbot; languages/domains outside the 8 competition subsets (behaviour is untested and unsafe).

Training data

All data is from the Zindi competition (HASH consortium sexual/reproductive-health Q&A across Uganda, Kenya, Ghana, Ethiopia). No external or hand-authored data; no preprocessing outside the code. Raw Train.csv / Val.csv are Zindi competition data and are not redistributed — download them from the competition page. The adapters are trained on this data but do not contain it.

Evaluation

submission	Public LB	ROUGE-1	ROUGE-L	Judge
sub_v40	0.728509	0.7187	0.6514	0.8522

The exact 0.728509 requires the hidden Test gold + Zindi's hosted judge. common/eval.py computes the exact 0.37/0.37/0.26 blend given a gold file (and per-row judge scores); reproduce.sh proves the shipped output is byte-identical to the scored sub_v40.

Known limitations

• Coverage: only the 8 competition subsets/languages; no guarantees elsewhere.
• ROUGE-optimized: the system is tuned to a lexical-overlap metric; high ROUGE ≠ clinical correctness.
• Answer-bank reliance: retrieval subsets depend heavily on verbatim reuse of Train/Val bank answers, so generalization to genuinely novel questions is weak.
• Amharic: ROUGE is effectively 0 for Ge'ez script under the metric; Amh_Eth passes through generation.
• Generation is stochastic: the shipped pool uses temperature>0 sampling (reproducible up to sampling; a greedy demo path is deterministic). Not a source of clinical reliability.
• Health-safety: outputs may be inaccurate/unsafe; see the disclaimer above.

How to use

The adapters are components of a pipeline — use the GitHub repo, which pulls these weights automatically:

git clone https://github.com/DariusTheGeek/Multilingual-Health-QA-ITU-Zindi-Challenge.git
cd Multilingual-Health-QA-ITU-Zindi-Challenge
bash env/install.sh
bash stages/0_setup/fetch_artifacts.sh              # ~250 MB data/features (Path A, CPU)
bash reproduce.sh                                    # -> sub_v40, BYTE-IDENTICAL, minutes, no GPU

# Run the actual trained model (GPU ~48 GB):
bash stages/0_setup/fetch_artifacts.sh --adapters    # pulls these 3 adapters (4.7 GB)
python stages/1_generation/demo_generate.py --adapter models/adapters/gen7454 --base google/gemma-4-31B-it --n 2

Standalone load (PEFT):

from peft import PeftModel
from transformers import AutoModelForCausalLM
base = AutoModelForCausalLM.from_pretrained("google/gemma-4-31B-it", torch_dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(base, "DariusTheGeek/mhqa-itu-adapters", subfolder="gen7454")

Dependencies

Python 3.11. CPU reproduction: lightgbm, pandas, pyarrow, regex, rouge_score, scikit-learn (requirements.txt). GPU generation: unsloth / vllm + transformers/peft (env/requirements-serve.txt, env/requirements-train.txt).

License

The adapters are LoRA derivatives of gated Google base models; base weights are referenced, not re-hosted. Use is subject to the base licenses — gemma-4-31B-it: Gemma Terms of Use and medgemma-27b-text-it: Health AI Developer Foundations terms. Accept each on the Hub before downloading the bases. Competition data is governed by the Zindi competition rules.

Citation

Solution to the ITU Multilingual Health Question Answering in Low-Resource African Languages challenge (Zindi / HASH consortium). Author: DariusTheGeek. Companion dataset repo: DariusTheGeek/mhqa-itu-artifacts.