Dynamic Accent Duplex (experimental)
An experimental cascaded full-duplex voice agent for Nigerian code-switched speech (English, Nigerian Pidgin, Igbo, Yoruba, Hausa) by Modular. You talk however you naturally mix languages; it answers in the same mix. Mostly English with an Igbo greeting gets a reply in mostly English with an Igbo greeting. Pidgin gets Pidgin.
This is not a native speech-to-speech model (no Moshi-style joint model). It is a pipeline of permissively licensed components with a language-mix controller in the middle:
mic ─► Silero VAD ─► Dynamic ASR (CTC, streaming partials) ─► per-word language ID + mix estimate
│ end-of-turn, barge-in, backchannels │ {eng: 0.8, ibo: 0.2, pattern: greeting:ibo}
▼ ▼
stop playback ◄── barge-in Qwen3-30B-A3B reply (3 candidates, inline <eng>/<ibo>/... tags)
▼
validator: mix distance, tags, greeting, length, loops, echo ─► regenerate / fallback
▼
speaker ◄── Dynamic Accent Flash-DPO TTS (chunk streaming, code-switching voice) ◄──┘
Status: research demo. Igbo, Yoruba and Hausa beyond greetings and simple phrases are weak (the reply LLM's limit, not the voice's). Read the limitations before using it for anything.
Components and licences
| stage | component | licence |
|---|---|---|
| voice activity / turn-taking | Silero VAD v5 | MIT |
| speech recognition | Dynamic ASR real (Meta omniASR_CTC_1B_v2 fine-tuned on licensed Nigerian speech; language-agnostic CTC + Latin-script mask). Fallback: stock Meta Omnilingual ASR omniASR_CTC_1B |
weights CC-BY-SA-4.0 (base Apache-2.0) / Apache-2.0 |
| language ID / mix | Dynamic Accent tagger.py (rules) + GlotLID v3 sliding windows |
Apache-2.0 / Apache-2.0 |
| reply LLM | Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 served with vLLM | Apache-2.0 / Apache-2.0 |
| text-to-speech | Dynamic Accent Flash-DPO (Modularcomputing/dynamic-accent-pro-v1.1, checkpoints/flash) on Fun-CosyVoice3-0.5B-2512 |
weights CC-BY-SA-4.0 (base Apache-2.0) |
| diacritic restoration (only when a Yoruba/Igbo span has no tone marks) | Dynamic Accent ByT5 restorer (base google/byt5-small) |
CC-BY-SA-4.0 (base Apache-2.0) |
| this repo's code | orchestration, mix controller, validator, servers | Apache-2.0 |
Evaluation-only models (not part of the agent): Whisper large-v3-turbo (MIT), omniASR CTC-300M / LLM-1B / LLM-3B (Apache-2.0), Mistral-Small-3.2-24B-Instruct-2506 (Apache-2.0), MADLAD-400-3B-MT (Apache-2.0, back-translation). Evaluation data: WAXAL test splits (Media Trust ibo/yor/hau/pcm subsets, CC-BY-SA-4.0), FLEURS validation (CC-BY-4.0). Not used, on licence grounds: African Voices / WaZoBia data, NaijaVoices (non-commercial), N-ATLaS (1,000-user cap, share-alike). Gemma 3 was not tested (gated).
This repo ships code and config only; each model is downloaded from its own repository under its own licence. Audio samples are made with the CC-BY-SA-4.0 TTS and are CC-BY-SA-4.0.
How the language mix works
- Per-word language ID. The release tagger labels every word (
eng/pcm/ibo/yor/hau) from diacritics, lexicons and Pidgin markers; ASR text has no punctuation or tone marks, so GlotLID scores 4-word sliding windows and confident windows override the rule labels. A sentence-level pass turns English-looking sentences with Pidgin markers into Pidgin. - Mix estimate. Word shares plus patterns:
greeting:<lang>,closing:<lang>,discourse:<lang>, intra- vs inter-sentential switching, blended 30 % with the conversation so far. ASR-misspelt greetings ("nde wo", "sanno", "dalo") are matched against a variant list. - Constrained reply. The LLM gets a natural-language target ("English about 80 %, Igbo about 20 %; open with a short Igbo greeting"), a phrasebook of correct greetings/thanks/closings, four examples, and must tag every span (
<eng>…</eng> <ibo>…</ibo>), which is exactly the input format of the Dynamic Accent TTS. - Validator. Each of 3 candidates is scored: tag format, total-variation distance between target and reply mix (declared tags averaged with re-measured LID), every target language ≥ 15 % present, greeting kept, 2–40 words, no loops, no echo of the user's words. The best passing candidate wins; otherwise one stricter round of 3 more; otherwise a fallback (best candidate, or for an Igbo/Yoruba/Hausa/Pidgin-led turn a correct phrasebook opener in that language plus a short clarification request).
Duplex behaviour
- 32 ms Silero frames; speech start after 96 ms; end of turn after 600 ms of silence.
- Speculative start: ASR → LLM → first TTS chunk begin at 256 ms of silence; audio is released at end of turn, and dropped if the user resumes.
- Barge-in: 192 ms of user speech while the agent speaks (or has committed to speak) stops playback at once (client queue flushed); the interrupted reply is kept in history as what was actually played +
[interrupted]. - Backchannels: during a long user turn (≥ 4.5 s), at a 288–448 ms pause, at most every 10 s: "Mm-hmm" (English) / "Ehen" (Pidgin, Igbo, Yoruba) / "To" (Hausa).
- Streaming TTS: reply cut into a short first chunk (≤ 6–10 words, at a comma/greeting/conjunction), then sentences ≤ 22 words, each synthesised while the previous one plays.
Run it
Tested on one H100 80 GB (vLLM 0.32, torch 2.x). Set up the Dynamic Accent TTS from Modularcomputing/dynamic-accent-pro-v1.1 as its README describes, then:
pip install -r requirements.txt # duplex server env (the TTS env)
pip install "omnilingual-asr>=0.2.0" huggingface_hub fastapi uvicorn # ASR worker env (can be a separate venv)
# ASR choice: ASR_MODEL=dynamic-asr (default) | dynamic-asr-cs (experimental code-switch) | omni (stock omniASR_CTC_1B)
pip install vllm # LLM server env
DA_REL=/path/to/dynamic-accent-pro-v1.1 ./launch.sh # -> http://localhost:7860 (use headphones)
Offline scripted conversations (same components, simulated real-time clock; latency = end-of-turn wait + measured compute):
cd code && python offline.py convo scenarios.json out/ # demo conversations -> wav + stereo (user L / agent R) + json log
python offline.py eval list.jsonl out/ 40 # batch latency / mix / barge-in eval on audio clips
Evaluation
All numbers from one H100 80 GB with the three processes sharing the GPU. Latency = end of the user's speech (VAD) to the first agent audio sample, from the offline driver: the end-of-turn wait is simulated frame by frame and every compute stage is timed on the real GPU (network and audio-device buffers excluded; the live WebSocket path measured 1.9-2.5 s over the public internet on two test turns). Held-out data only; whether Meta's ASR saw WAXAL/FLEURS in training is unknown.
ASR (229 held-out clips; WER / CER; CER without diacritics in brackets)
| model | eng | pcm | ibo | yor | hau | cs | latency p50 / p90 per clip | CS mix TVD after ASR (rules / +GlotLID windows) |
|---|---|---|---|---|---|---|---|---|
| omniASR_CTC_1B | 0.14 / 0.07 (0.07) | 0.38 / 0.17 (0.17) | 0.63 / 0.20 (0.16) | 0.49 / 0.19 (0.12) | 0.30 / 0.09 (0.09) | 0.33 / 0.13 (0.10) | 44 / 50 ms | 0.402 / 0.277 |
| omniASR_CTC_300M | 0.13 / 0.05 (0.05) | 0.58 / 0.23 (0.23) | 0.70 / 0.21 (0.17) | 0.59 / 0.21 (0.14) | 0.40 / 0.13 (0.13) | 0.40 / 0.14 (0.12) | 27 / 30 ms | 0.381 / 0.381 |
| omniASR_LLM_1B | 0.07 / 0.03 (0.03) | 0.33 / 0.15 (0.15) | 0.54 / 0.17 (0.12) | 0.40 / 0.16 (0.11) | 0.30 / 0.09 (0.09) | 0.32 / 0.14 (0.11) | 1037 / 1802 ms | 0.4 / 0.4 |
| omniASR_LLM_3B | 0.06 / 0.03 (0.03) | 0.43 / 0.18 (0.18) | 0.63 / 0.21 (0.15) | 0.35 / 0.14 (0.10) | 0.30 / 0.09 (0.09) | 0.32 / 0.15 (0.13) | 848 / 1760 ms | 0.352 / 0.352 |
| whisper_turbo | 0.03 / 0.01 (0.01) | 0.42 / 0.21 (0.21) | 1.06 / 0.41 (0.34) | 1.05 / 0.52 (0.29) | 0.90 / 0.33 (0.32) | 0.81 / 0.62 (0.61) | 119 / 208 ms | 0.51 / 0.51 |
| omniASR_CTC_1B_latin | 0.14 / 0.07 (0.07) | 0.38 / 0.17 (0.17) | 0.63 / 0.20 (0.16) | 0.49 / 0.19 (0.12) | 0.30 / 0.09 (0.09) | 0.33 / 0.13 (0.10) | 45 / 54 ms | 0.402 / 0.277 |
| omniASR_CTC_1B_v2 (stock base) | 0.10 / 0.02 (0.02) | 0.34 / 0.12 (0.12) | 0.46 / 0.13 (0.11) | 0.40 / 0.13 (0.10) | 0.32 / 0.11 (0.11) | 0.29 / 0.10 (0.09) | same CTC-1B arch, not re-timed (≈ CTC_1B) | not measured |
| Dynamic ASR real (default) | 0.10 / 0.03 (0.03) | 0.19 / 0.07 (0.07) | 0.32 / 0.09 (0.08) | 0.35 / 0.12 (0.08) | 0.25 / 0.08 (0.08) | 0.24 / 0.08 (0.06) | same CTC-1B arch, not re-timed (≈ CTC_1B) | not measured |
| Dynamic ASR code-switch (experimental) | 0.09 / 0.02 (0.02) | 0.19 / 0.08 (0.08) | 0.32 / 0.10 (0.08) | 0.35 / 0.12 (0.08) | 0.25 / 0.08 (0.08) | 0.22 / 0.07 (0.06) | same CTC-1B arch, not re-timed (≈ CTC_1B) | not measured |
Note: the end-to-end latency tables below (and the per-clip ASR latency column) were measured with omniASR_CTC_1B as the ASR, before Dynamic ASR existed. Dynamic ASR is the same size and architecture (1B wav2vec2 CTC, greedy decoding), so latency is expected to be similar but was not re-measured. Dynamic ASR rows use the same 229 clips and scorer; see the Dynamic ASR card for the code-switch tests.
LID on reference CS text (upper bound): word accuracy 0.9, mix TVD 0.133.
Reply LLM (28 prompts: 16 mixed/greeting/Pidgin/English written prompts + 12 native WAXAL sentences ibo/yor/hau)
| model | subset | first candidate passes | regen rate | fallback | mean mix TVD | GlotLID agrees with span tag | judge 1-5 (back-translated) | LLM s (n=3) |
|---|---|---|---|---|---|---|---|---|
| mistral-small-3.2-24b | mixed | 0.93 | 0.07 | 0.0 | 0.027 | 1.0 | 4 | 0.433 |
| mistral-small-3.2-24b | native | 0.92 | 0.08 | 0.0 | 0.008 | 1.0 | 1 | 0.584 |
| mistral-small-3.2-24b | all | 0.89 | 0.07 | 0.0 | 0.017 | 1.0 | 2.79 | 0.497 |
| qwen3-30b-a3b-2507 | mixed | 0.86 | 0.07 | 0.0 | 0.018 | 1.0 | 4.43 | 0.277 |
| qwen3-30b-a3b-2507 | native | 0.92 | 0.0 | 0.0 | 0.029 | 1.0 | 1.58 | 0.41 |
| qwen3-30b-a3b-2507 | all | 0.89 | 0.04 | 0.0 | 0.021 | 1.0 | 3.25 | 0.317 |
Latency, batch eval (80 turns; seconds, p50 / p90)
| stage | p50 | p90 |
|---|---|---|
| end-of-turn wait (VAD silence) | 0.576 | 0.576 |
| ASR final | 0.036 | 0.042 |
| LID + mix | 0.013 | 0.03 |
| LLM + validator rounds | 0.389 | 0.791 |
| TTS first chunk | 0.863 | 1.69 |
| compute total (ASR->first chunk) | 1.381 | 2.211 |
| end of user speech -> first agent audio | 1.637 | 2.467 |
Mix adherence (batch_eval): mean TVD(target, reply) 0.124 (p90 0.451); first candidate passes 0.588; regenerated 0.037; fallback 0.15 (safe phrasebook 0.113); 39/80 turns had a multi-language target.
Latency, demos (19 turns; seconds, p50 / p90)
| stage | p50 | p90 |
|---|---|---|
| end-of-turn wait (VAD silence) | 0.576 | 0.576 |
| ASR final | 0.057 | 0.069 |
| LID + mix | 0.005 | 0.016 |
| LLM + validator rounds | 0.246 | 0.537 |
| TTS first chunk | 0.795 | 1.168 |
| compute total (ASR->first chunk) | 1.139 | 1.488 |
| end of user speech -> first agent audio | 1.395 | 1.744 |
Mix adherence (demos): mean TVD(target, reply) 0.084 (p90 0.162); first candidate passes 0.789; regenerated 0.0; fallback 0.053 (safe phrasebook 0.053); 11/19 turns had a multi-language target.
Barge-in (batch eval, n=40): user speech onset -> agent audio stopped p50 0.235 s, p90 0.3 s (VAD detection p50 0.195 s + 40 ms output flush).
Demos
samples/ holds four scripted conversations made with the offline driver (agent: Flash-DPO male anchor). User turns in these samples are TTS-synthesised (Flash female voices), not real people. Each has a JSON log with the ASR text, the target mix, the reply, the validator outcome and per-stage latency.
Limitations (read before use)
- Igbo, Yoruba and Hausa content is weak beyond greetings, thanks and short phrases. The reply LLM (and every permissive LLM we tried) parrots or garbles longer native-language replies; a back-translation judge scored native-language replies 1.6/5 versus 4.4/5 for mixed/greeting/Pidgin prompts. The validator catches echoing and drift and falls back to an honest short reply, which means native-led turns often get "Sannu! Sorry, I did not catch all of that…".
- ASR on accented or synthetic speech misspells Igbo/Yoruba greetings (and loses tone marks), which can hide the user's language from the mix estimate; a variant list covers common greetings only.
- Pidgin vs English is the hardest LID boundary; Pidgin detection uses marker words.
- Latency p90 is driven by the TTS first chunk (autoregressive 0.5B LM, ~17 ms per audio token, ~0.4-0.6 x real time on a shared H100).
- Barge-in uses VAD only: without headphones or echo cancellation the agent's own voice can interrupt it. Backchannels are rule-based and can be too frequent on long read speech.
- No safety filtering beyond the LLM's own; replies can be wrong. Research demo, not a product.
Citation / credits
Built by Modular on Meta Omnilingual ASR, Silero VAD, GlotLID, Qwen3, vLLM, CosyVoice3 and the Dynamic Accent TTS (trained on licensed data only: WAXAL, FLEURS, Open.Bible, NaijaS2ST, IgboAPI, IgboSynCorp, BibleTTS, OpenSLR).
- Downloads last month
- -