Fine-Grained Control
Thank you for your efforts in creating a Persian-capable model. Huge fan of your Shenava.
One of the features I really like about OmniVoice is its fine-grained pronunciation control. I think this is especially important for languages like Persian, where short vowels are usually omitted in writing and the same written form can therefore have different pronunciations.
For example, مرد can be pronounced mard (مَرد, “man”) or mord (مُرد, “died”). There are also proper nouns and less common words whose pronunciation can be difficult to infer from the written form alone, such as شهر خوی (“the city of Khoy”).
In the OmniVoice demo, this is presented under Fine-Grained Control, where English pronunciation can be explicitly controlled using CMU pronunciation dictionary phonemes, for example:
"He plays the [B EY1 S] guitar while catching a [B AE1 S] fish."
I tried fine-tuning OmniVoice on my own Persian dataset, but my dataset only contains pairs of (audio -> plain text) with no phoneme or paralinguistic annotations. I noticed that the provided fine-tuning pipeline also appears to use ordinary text/audio pairs. So I was wondering how fine-grained controls should be introduced when fine-tuning for a new language such as Persian.
For paralinguistic tags such as [laughter], I assume they could potentially be generated automatically using audio event detection or similar classifiers. However, pronunciation control is much more important for Persian. Is there a recommended way to add Persian phoneme-level control similar to the CMU phoneme representation used for English?
For example, could we introduce something like:
[M AE R D] → مَرد
[M O R D] → مُرد
or use IPA/a Persian G2P system and include those representations in the fine-tuning data? Would this require modifying the tokenizer or the original training pipeline, or can OmniVoice learn new phonetic control tokens during fine-tuning?
Again, thank you for all the work you've done.