Vyvo-Multilingual-JA-FT-v0.1

A Japanese voice-cloning TTS model — Vyvo/Vyvo-Multilingual-v0.1 fine-tuned for a single conversational Japanese speaker. Give a short reference clip (+ its transcript) and a target text; the model speaks the text in that voice by pure language-model continuation.

• Backbone: Qwen3-0.6B + kyutai/mimi audio codec (32 codebooks, 24 kHz)
• Base model: Vyvo/Vyvo-Multilingual-v0.1
• Goal: improve Japanese intelligibility for this voice over the multilingual base

Results

Japanese Character Error Rate (CER — the standard metric for Japanese) on 100 evaluation texts, transcribed with Qwen3-ASR-1.7B (lower = better):

Model	Decoding	CER ↓	WER ↓	Accuracy (1-CER)
Base (Vyvo-Multilingual-v0.1)	greedy	0.395	0.426	60.5 %
This model	greedy	0.338	0.375	66.2 %
This model	sampling + best-of-6	0.155	0.218	84.5 %

Greedy decoding under-reports quality (occasional early end-of-speech); with sampling + best-of-N the model reaches ~84.5 % character accuracy. Most remaining errors are homophone / kanji-vs-kana substitutions that are phonetically correct.

Recommended decoding

The model is a transcript-conditioned TTS: always pass the reference clip's transcript. For production use sampling (temperature ≈ 0.4–0.5, top_p = 0.9, repetition_penalty = 1.1) and, when quality matters, best-of-N (generate 4–6 candidates and keep the one whose ASR transcript best matches the target).

Usage

import soundfile as sf
import torch
import torchaudio
from transformers import (AutoModelForCausalLM, AutoTokenizer,
                          AutoFeatureExtractor, MimiModel)

REPO = "Vyvo/Vyvo-Multilingual-JA-FT-v0.1"
DEVICE = "cuda"

BASE = 151669
NUM_CODEBOOKS, CODEBOOK_SIZE, AUDIO_OFFSET = 32, 2048, 10
SOS, EOS, SOH, EOH, SOA = 1, 2, 3, 4, 5

tokenizer = AutoTokenizer.from_pretrained(REPO)
model = AutoModelForCausalLM.from_pretrained(REPO, dtype=torch.bfloat16).to(DEVICE).eval()
mimi = MimiModel.from_pretrained("kyutai/mimi").to(DEVICE).eval()
feature_extractor = AutoFeatureExtractor.from_pretrained("kyutai/mimi")


def encode_reference(path):
    wav, sr = sf.read(path, dtype="float32", always_2d=False)
    if wav.ndim > 1:
        wav = wav.mean(axis=1)
    target_sr = feature_extractor.sampling_rate
    if sr != target_sr:
        wav = torchaudio.functional.resample(
            torch.from_numpy(wav).unsqueeze(0), sr, target_sr).squeeze(0).numpy()
    inputs = feature_extractor(raw_audio=wav, sampling_rate=target_sr, return_tensors="pt")
    codes = mimi.encode(inputs["input_values"].to(DEVICE),
                        num_quantizers=NUM_CODEBOOKS).audio_codes[0].cpu()
    flat = codes.transpose(0, 1).reshape(-1).tolist()
    return [c + AUDIO_OFFSET + (i % NUM_CODEBOOKS) * CODEBOOK_SIZE + BASE
            for i, c in enumerate(flat)]


def build_prompt(ref_tokens, ref_text, target_text):
    text_ids = tokenizer(ref_text + " " + target_text, add_special_tokens=False).input_ids
    head = [BASE + SOH] + text_ids + [tokenizer.eos_token_id, BASE + EOH, BASE + SOA, BASE + SOS]
    return head + ref_tokens


def decode_audio(generated_ids):
    codes = []
    for i, tok in enumerate(generated_ids):
        v = tok - BASE - AUDIO_OFFSET - (i % NUM_CODEBOOKS) * CODEBOOK_SIZE
        if 0 <= v < CODEBOOK_SIZE:
            codes.append(v)
        else:
            break
    frames = len(codes) // NUM_CODEBOOKS
    codes = torch.tensor(codes[:frames * NUM_CODEBOOKS]).view(frames, NUM_CODEBOOKS)
    codes = codes.t().unsqueeze(0).to(DEVICE)
    return mimi.decode(codes).audio_values.squeeze().cpu().float().numpy()


reference_wav = "reference.wav"
reference_text = "リファレンス音声の書き起こし"          # transcript of the reference clip
target_text = "こんにちは、これは音声合成モデルのテストです。"

prompt = build_prompt(encode_reference(reference_wav), reference_text, target_text)
input_ids = torch.tensor([prompt], device=DEVICE)
output = model.generate(
    input_ids, attention_mask=torch.ones_like(input_ids),
    max_new_tokens=9600, min_new_tokens=960,
    do_sample=True, temperature=0.45, top_p=0.9, repetition_penalty=1.1,
    eos_token_id=BASE + EOS, pad_token_id=tokenizer.eos_token_id)
audio = decode_audio(output[0, input_ids.shape[1]:].tolist())
sf.write("output.wav", audio, mimi.config.sampling_rate)

Training

Method	full fine-tuning (liger fused cross-entropy)
Learning rate	1e-4, cosine, warmup 0.03
Epochs / batch	2 / 8
Hardware	1× H100

Recipe carried over from the best English fine-tune (full FT, lr 1e-4).

Limitations

• Requires the reference transcript. Transcript-free / zero-shot cloning collapses.
• Single speaker / Japanese only. CER over-penalises homophone kanji/kana variants, so true intelligibility is somewhat higher than the number suggests.
• Greedy decoding may truncate long sentences — see Recommended decoding.

License

Fine-tuned from Vyvo/Vyvo-Multilingual-v0.1. Released under the MIT License.