Afri2Universal: Multilingual African Speech Recognition (314 Languages) on Whisper Large-v3-Turbo

Afri2Universal adapts OpenAI's state-of-the-art Whisper Large-v3-Turbo (809M parameters) across 314 African languages and dialects (~950 hours of audio).

The model is conditioned on the English transcription prompt (<|startoftranscript|> <|en|> <|transcribe|> <|notimestamps|>) to leverage Whisper's vast phonetic and subword priors, mapping spoken audio from all 314 indigenous African languages directly into regularized Latin orthography with preserved word boundaries (spaces) and zero punctuation.

---

Model & Training Architecture

• Foundation Model: openai/whisper-large-v3-turbo (32-layer encoder, 4-layer fast decoder, 128 mel frequency bins)
• Decoder Prompt Conditioning: <|startoftranscript|> <|en|> <|transcribe|> <|notimestamps|> (English transcription mode)
• Adaptation Strategy: Parameter-Efficient Fine-Tuning with LoRA ($r=32, \alpha=64$, dropout $0.05$) applied to query and value attention projections (q_proj, v_proj).
• Trainable Parameters: 6,553,600 out of 815,431,680 (0.80% parameter footprint).
• Target Vocabulary: Universalized Latin script (a-z) with single spaces separating words, lowercase, no punctuation.
• Batch Size: Effective batch size of 64 audio samples per optimizer step (per-device batch size 16 with gradient accumulation 4).
• Precision: Pure bfloat16 with Hopper TensorFloat-32 (TF32) tensor cores on NVIDIA H200 (140 GB VRAM).
• Evaluation: Evaluates real Word Error Rate (WER) and Character Error Rate (CER) on held-out test splits.

---

Dataset

Trained on the Afri2Universal corpus: AfriSpeech/afri2universal spanning 314 African languages balanced to ~8.39 hours per language.

---

Quickstart & Inference

import torch
from transformers import WhisperProcessor, WhisperForConditionalGeneration
from peft import PeftModel

MODEL_ID = "openai/whisper-large-v3-turbo"
LORA_ID = "AfriSpeech/afri2universal"

processor = WhisperProcessor.from_pretrained(MODEL_ID, language="en", task="transcribe")
base_model = WhisperForConditionalGeneration.from_pretrained(
    MODEL_ID, torch_dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(base_model, LORA_ID)

# Transcribe audio array
inputs = processor(audio_array, sampling_rate=16000, return_tensors="pt").input_features.to("cuda", dtype=torch.bfloat16)

with torch.no_grad():
    generated_ids = model.generate(inputs, max_new_tokens=225)
    
transcription = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
print("Transcription:", transcription)