MBZUAI

/

speecht5_tts_clartts_ar

like 32

Follow

Mohamed Bin Zayed University of Artificial Intelligence 864

Thanks for releasing this model — the notes below are a limitation report from
an external benchmark, not a defect claim. ArTST is fine-tuned on ClArTTS, a Classical
Arabic corpus, and the model card nowhere claims numeral coverage. The reason it seemed
worth writing up is that the failure is total and silent: a pipeline containing a
date or a price ships audio with the number missing and nothing raises an error.

Measured on a 45-utterance set (15 sentences x 3 numeral forms), speecht5_tts_clartts_ar
recovers 0 of 30 numerals written in digits — 0/15 Western (2026), 0/15 Arabic-Indic
(٢٠٢٦). Spelled out in Arabic words the same numbers score 4/15.

The two digit failures look identical in the output and are not the same bug.

1. Arabic-Indic digits are not in the vocabulary

The tokenizer is character-level SentencePiece with 87 tokens. All ten Western digits are
present. None of ٠١٢٣٤٥٦٧٨٩ are:

from transformers import SpeechT5Processor
tk = SpeechT5Processor.from_pretrained("MBZUAI/speecht5_tts_clartts_ar").tokenizer
v = tk.get_vocab()
[d for d in "٠١٢٣٤٥٦٧٨٩" if d in v]   # -> []
[d for d in "0123456789"  if d in v]   # -> all ten

Because it is character-level, a run of unknown characters collapses to a single
<unk>, so the whole year is gone before synthesis starts:

'في عام ٢٠٢٦ ارتفعت'
  -> [..., '▁', '<unk>', '▁', ...]
  -> decodes back as 'في عام  ارتفعت'

Reproducible in three lines, no audio and no GPU needed.

2. Western digits reach the model and still produce no audio

Western digits tokenise cleanly (15/15) and reach the decoder. They are still not spoken.
Measured without ASR in the loop, by synthesising each sentence twice — as written, and
with the numeral deleted — and comparing durations:

ArTST's own spelled-out renderings of the same 15 numbers run +1.41 s over its
Western-digit renderings, so the digit run yields roughly an eighth of the audio the
same model produces when it actually says the figure. Transcripts show it directly —
the sentence survives with a number-shaped hole in it:

Re-scoring the same audio with whisper-medium instead of whisper-small moves the
spelled baseline from 4/15 to 6/15 and leaves both digit forms at 0/15 — floored
regardless of listener capacity, which is what you expect when there is nothing in the
audio to hear.

3. The honest baseline

ArTST scores 4/15 (27%) on the spelled form, where every character is in-vocab. So part
of the 0% is general intelligibility on this set, not numerals — the sentences are Modern
Standard Arabic news-register copy and this is a Classical Arabic model being run outside
its domain. The numeral-specific effect is the drop from 27% to 0%, not the whole 0%.
Quoting 0/30 without that qualifier would overstate it.

4. Two practical suggestions

1. A one-line vocabulary/preprocessing fix mapping ٠-٩ to 0-9 would convert the
   Arabic-Indic failure into the Western-digit failure. It fixes neither, but it stops
   the tokenizer silently deleting input, which is the part that hurts most in
   production.
2. A line on the model card saying numerals must be spelled out upstream would save
   integrators the debugging. Western-digit normalisation alone is not enough here —
   unlike other engines, ArTST fails on both digit forms.

Happy to share the full 45-sentence set, the per-utterance transcripts and the duration
control script if useful. Data and method: https://huggingface.co/datasets/syamjithnk/arnum-tts

— Syamjith NK