whisper-small-afrispeech

Fine-tune of openai/whisper-small on AfriSpeech-200 for African-accented English speech recognition, spanning both general and clinical/medical domains.

• Base model: openai/whisper-small (244M)
• Training data: Professor/afrispeech-200-16khz — a clean 16 kHz mono build of AfriSpeech-200 (Intron Health): ~200 h, 120 accents, 13 countries, clinical + general.
• Language: English (African accents)

Results (test split, 6,178 clips)

Fine-tuned vs. zero-shot baseline

Zero-shot = openai/whisper-small evaluated on the same test set with no fine-tuning.

Metric	Zero-shot (baseline)	Fine-tuned (this model)	Improvement
Overall WER	49.15%	22.94%	−26.2 pts (−53%)
Clinical WER	56.28%	29.84%	−26.4 pts (−47%)
General WER	42.21%	16.23%	−26.0 pts (−62%)

Fine-tuning roughly halves WER across the board — a 26-point absolute reduction overall — and cuts clinical error from 56% to 30%.

WER by domain

Domain	WER	n
General	16.23%	2,670
Clinical	29.84%	3,508
Overall	22.94%	6,178

Clinical speech is ~1.8× harder than general — medical terminology (drug names, conditions, dosages) drives most of the error, so domain matters when reporting WER.

WER by accent (70 accents with n ≥ 25)

Performance varies widely across accents (~8% to ~62%):

Best accents	WER		Hardest accents	WER
okirika	8.2%		agatu	61.6%
brass	9.4%		mwaghavul	58.1%
afrikaans	11.7%		efik	57.8%
twi	11.9%		bini	51.0%
ikwere	12.1%		idah	47.7%

The hardest accents are mostly smaller, under-represented Nigerian accents — an important coverage/equity consideration. Full per-accent numbers: eval_wer_breakdown.csv.

Usage

import torch
from transformers import WhisperForConditionalGeneration, WhisperProcessor

model = WhisperForConditionalGeneration.from_pretrained("Professor/whisper-small-afrispeech")
processor = WhisperProcessor.from_pretrained("Professor/whisper-small-afrispeech")

# audio: a 16 kHz mono waveform (numpy array)
inputs = processor(audio, sampling_rate=16000, return_tensors="pt")
ids = model.generate(inputs.input_features, language="english", task="transcribe")
print(processor.batch_decode(ids, skip_special_tokens=True)[0])

Training

• Objective: full fine-tune (all parameters), 3 epochs
• Precision / hardware: bf16 on a single NVIDIA A40 (48 GB)
• Batch: 32 · LR: 1e-5 (500 warmup steps, linear decay) · best model by eval WER
• Data handling: on-the-fly log-Mel feature extraction; clips > 30 s filtered out (Whisper's fixed 30 s window); labels truncated to Whisper's 448-token decoder cap.
• Runtime: ~83 min.

Limitations & biases

• Clinical error rate is high (~30%) — verify medical terms; not for unsupervised clinical use.
• Large accent disparity (~8–62% WER) — much weaker on under-represented accents.
• ≤ 30 s audio — longer inputs are truncated to 30 s by Whisper's feature extractor.
• English only; inherits Whisper's general biases.

Dataset, license & attribution

Trained on AfriSpeech-200, created by Intron Health and released under CC-BY-NC-SA-4.0 (non-commercial, share-alike, attribution). This derivative model carries the same license. Please cite AfriSpeech-200:

@article{olatunji2023afrispeech,
  title={AfriSpeech-200: Pan-African Accented Speech Dataset for Clinical and General Domain ASR},
  author={Olatunji, Tobi and others},
  journal={Transactions of the Association for Computational Linguistics},
  year={2023}
}