AlexTheSun/omnivoice-komi

OmniVoice Komi (Komi-Zyrian text-to-speech)

A fine-tune of k2-fsa/OmniVoice for Komi-Zyrian (коми кыв, ISO 639-3 kpv). It reads Komi text with Komi pronunciation in a voice cloned from a short reference clip. The base model's other languages still work.

This is the model behind the Komi TTS demo at majbyr.com/tts.

• model.safetensors and the rest of the root folder: the merged model, ready to use.
• lora/: the same fine-tune as a LoRA adapter (rank 32) on k2-fsa/OmniVoice at revision c5fdb5cc.

Usage

Install OmniVoice. OmniVoice silently drops language codes it does not know, and Komi is not among its built-in languages, although this model was trained with the kpv tag. Register it before generating:

import soundfile as sf
import torch
import omnivoice.utils.lang_map as lang_map
from omnivoice.models.omnivoice import OmniVoice

if isinstance(lang_map.LANG_IDS, set):
    lang_map.LANG_IDS.add("kpv")
else:
    lang_map.LANG_IDS = list(lang_map.LANG_IDS) + ["kpv"]

model = OmniVoice.from_pretrained("AlexTheSun/omnivoice-komi", device_map="cuda", dtype=torch.float16)

audio = model.generate(
    text="тасянь неылын эм чудь гу.",
    ref_audio="komi_reference.wav",       # 3-15 s of the voice
    ref_text="transcript of the reference clip",
    language="kpv",
)[0]
sf.write("out.wav", audio, model.sampling_rate)

Best results: a Komi "anchor" reference

Cloning straight from a reference in another language (say English) carries that language's accent into Komi, especially in short words. The fix that worked best in listening tests:

1. Read one fixed Komi sentence in the target voice, using the original clip as the reference:

   ANCHOR_TEXT = ("Менам сьӧлӧм кылӧ тэнсьыд олан рутӧ. "
                  "Сёрниясыс, паныдасьлӧмъясыс да кӧсйысьӧмъясыс ньӧти оз збыльмӧдчыны.")
   anchor = model.generate(text=ANCHOR_TEXT, ref_audio="voice_any_language.wav",
                           ref_text="its transcript", language="kpv")[0]
   sf.write("anchor.wav", anchor, model.sampling_rate)
   

2. Use that Komi clip as the reference from then on, with ANCHOR_TEXT as its transcript and language="kpv".

The anchor alone already beats the language tag alone, and anchor + kpv is clearly best. The voice is kept well.

Input text

• Write in lowercase, keeping only short abbreviations (СССР, США) in capitals. The training text had capitals almost only at sentence starts, so capitals get misread: Ӧ as О, softened consonants (Сійӧ read as «Сийӧ»), odd stress. The majbyr.com service lowercases Komi input before synthesis.
• Spell out numbers in Komi words; digits are not read reliably.
• Use the Komi letters ӧ and і, not Latin ö and i, which look the same.
• Split long texts into sentence groups of about 15 seconds of speech.

Training

• Data: about 2.9 hours of read Komi sentences:
  • recordings by FU-Lab;
  • recordings made for this project by Aleksei Ivanov.
• Method: LoRA (r = 32, alpha 64, dropout 0.05) on the attention and MLP projections of the language model.
  • Audio embeddings and audio heads were kept frozen.
  • Learning rate 5e-5, 1,500 steps on one A100 80 GB.
  • Half of the samples were tagged kpv, half untagged.
• Why this recipe: an earlier run also trained on word lists from many low-quality readers and fine-tuned the audio layers. With that run, outside voices drifted towards the training speakers. This recipe fixed it: drift on single words dropped from 56 of 96 to 20 of 96 test words, the same level as the base model.

License and credits

• Model weights: CC BY-NC 4.0, inherited from the base model. OmniVoice's weights are licensed CC BY-NC because of their training data; its code is Apache 2.0.
• Audio tokenizer: audio_tokenizer/ is Boson AI's Higgs Audio 2 tokenizer, as shipped with OmniVoice, under its own licence (audio_tokenizer/LICENSE).
• Base model: k2-fsa/OmniVoice.
• Komi recordings: FU-Lab and Aleksei Ivanov.