PrinceAlhassanNasamu/tekyerema-audio-native

Tɛkyerɛma-1 audio-native (arm 3)

Author: Prince Nasamu Alhassan

Overview

Qwen2-Audio-7B-Instruct + LoRA. Takes SPEECH and returns a tool call, with no recogniser in between: nothing can be lost in a transcript that is never produced.

Measured against the cascade it is meant to replace. Same 60 held-out commands, same synthesised audio, same gold, scored on the 50 rows both paths could sit (Dagbani has no recogniser, so the cascade cannot reach 10 of them):

path	correct
audio → ASR → tekyerema-1-native-tool	5/52 · 9.6%
audio → this model (1,200 clips)	23/52 · 44.2%
audio → tekyerema-audio-native-4k (4,000 clips)	45/52 · 86.5%

These replace an earlier 30.0% / 38.0% pair, which was scored on a leaking split. The corpus has 8,488 rows and 2,943 distinct utterances, so holding out by row index put verbatim copies of the training text into the test set. The numbers above skip every row whose utterance appears in training — 139 of them.

Train on more clips. The 4,000-clip run in tekyerema-audio-native-4k roughly doubles this model's accuracy on the same unseen rows. Prefer it.

Read that by its rows, not its total. The eight points are four items. Per language, both paths score 11/11 on Ghanaian English and close to nothing elsewhere: Ewe 0/9 against 3/9, Kusaal 0/9 against 2/9, Twi 0/9 against 1/9, Hausa 4/12 against 2/12, Dagbani 0/10 for this model and unreachable for the cascade.

The comparison is BIASED TOWARDS THIS MODEL, and the project wrote that down before the test was run. EVAL20/README.md says scoring an audio-native arm on MMS-VITS speech is close to self-evaluation, because it is the same synthetic voice family it trained on. That is exactly what this measurement did: this model was trained on MMS-VITS audio and tested on MMS-VITS audio, while the recognisers in the cascade had never heard that voice family in their lives. Read the eight-point lead as an upper bound that flatters this model, not as a finding.

Both numbers are about the test as much as the models. No recordings of people speaking agent commands in these languages exist, so the audio is synthesised, and synthetic speech degrades these recognisers four- to eightfold. This model was trained on synthetic audio and still reaches only 38%.

Trained on 1,200 clips for 3 epochs; training loss moved from 4.65 to about 4.3 and then flattened, which is not the curve of a model that has learned the task. Treat this as evidence that the approach is worth more data, not as a finished router.

Use it

import torch, soundfile as sf
from peft import PeftModel
from transformers import AutoProcessor, Qwen2AudioForConditionalGeneration

BASE = "Qwen/Qwen2-Audio-7B-Instruct"
proc  = AutoProcessor.from_pretrained("PrinceAlhassanNasamu/tekyerema-audio-native")
model = PeftModel.from_pretrained(
    Qwen2AudioForConditionalGeneration.from_pretrained(
        BASE, dtype=torch.bfloat16, device_map="auto"),
    "PrinceAlhassanNasamu/tekyerema-audio-native").eval()

convo = [{"role": "system",
          "content": "Convert the spoken command into one tool call as JSON."},
         {"role": "user",
          "content": [{"type": "audio", "audio_url": "x.wav"}]}]
prompt = proc.apply_chat_template(convo, add_generation_prompt=True,
                                  tokenize=False)

wav, sr = sf.read("command.wav", dtype="float32")
inp = proc(text=prompt, audio=[wav], sampling_rate=sr,
           return_tensors="pt").to(model.device)
with torch.no_grad():
    out = model.generate(**inp, max_new_tokens=48, do_sample=False)
print(proc.batch_decode(out[:, inp["input_ids"].shape[1]:],
                        skip_special_tokens=True)[0])

Training data

Trained on the Ghana Speech dataset and related Ghanaian corpora, licensed CC BY-NC 4.0.

Intended use & license

Non-commercial use only (CC BY-NC 4.0). This is inherited from the training data and required by the terms under which the compute was granted: models trained in that window are non-commercial by condition of access, not by inference.

Limitations, stated plainly

• Dagbani had no recogniser of its own for this whole project, and the reason given for that was wrong. Every card here said "one fine-tuning session on 74 validation rows would not change that". Those 74 rows are the eng-dag machine-translation validation split. The Dagbani speech data in this same account is waxal_dag: 13,228 training rows, 1,750 validation rows, ~71 hours, 1,041 speakers with the largest at 1% — more data and better speaker diversity than Ewe, which produced a working 42.19 WER recogniser. A number was carried across from a translation table into a speech claim, and then repeated on every model card on the account. It is training now, on 2026-08-31. Until it is scored, the honest statement is that Dagbani's best available recogniser scores 86.6 WER and nobody had tried fine-tuning on the data already in hand.
• Evaluation is on read and machine-translated text. No recordings of people speaking agent commands in these languages exist. Numbers measured this way are optimistic about phrasing and pessimistic about code-switching, and should not be read as field performance.
• Research work from a hackathon entry, not a supported product.

The rest of the family

Recognisers

• whisper-large-v3-turbo-tekyerema-eng-foundation — Ghanaian English ASR — course 1 (foundation)
• kusaal-whisper-small-lora — Kusaal ASR (Whisper-small LoRA, superseded)
• kasa42-asr — KASA-42 (Kusaal, third-party export)
• tekyerema-asr-ctc — Twi ASR (w2v-BERT CTC)
• tekyerema-asr-mms-ewe — Ewe ASR (MMS adapter)
• tekyerema-asr-mms-dag — Dagbani ASR (MMS adapter)
• tekyerema-asr-mms-hau — Hausa ASR (MMS adapter)
• tekyerema-asr-mms-kus — Kusaal ASR (MMS adapter)
• whisper-large-v3-turbo-tekyerema-eng — Ghanaian English ASR (Whisper large-v3-turbo)

Voices

• tekyerema-tts-twi — Twi TTS (VITS)
• tekyerema-tts-kus — Kusaal TTS (VITS)
• tekyerema-tts-ewe — Ewe TTS (VITS)
• tekyerema-tts-hau — Hausa TTS (VITS)
• tekyerema-tts-eng — Ghanaian English TTS (VITS)

Agent models

• tekyerema-1-reply — Tɛkyerɛma-1 reply adapter (arm ①)
• tekyerema-1-native-reply — Tɛkyerɛma-1 reply adapter (arm ②)
• tekyerema-1-tool — Tɛkyerɛma-1 tool adapter (arm 1)
• tekyerema-audio-native — Tɛkyerɛma-1 audio-native (arm 3)
• tekyerema-audio-native-4k — Tɛkyerɛma-1 audio-native, 4,000 clips (arm 3 v2)
• tekyerema-1-native-tool — Tɛkyerɛma-1 tool adapter (arm 2)

Translation

• tekyerema-nllb600m-v1 — Tɛkyerɛma MT v1 (NLLB-600M)
• kusaal-nllb-600M — Kusaal MT specialist (NLLB-600M)

Routing

• tekyerema-intent-afroxlmr — Intent classifier (AfroXLMR)

Acknowledgements

Compute resources provided by AI Skills and Compute Africa (AISCA). Trained on the Ghana NLP H200 GPU. Please keep derivatives non-commercial and share improvements back with the Ghana NLP community (ghananlpcommunity).