MulliVC: Multi-lingual Voice Conversion With Cycle Consistency
Authors: name: Jiawei Huang; name: Chen Zhang; name: Yi Ren; name: Ziyue Jiang; name: Zhenhui Ye; name: Jinglin Liu; name: Jinzheng He; name: Xiang Yin; name: Zhou Zhao
Published: 2024-08-08 18:12:51

Abstract: Voice conversion aims to modify the source speaker's voice to resemble the
target speaker while preserving the original speech content. Despite notable
advancements in voice conversion these days, multi-lingual voice conversion
(including both monolingual and cross-lingual scenarios) has yet to be
extensively studied. It faces two main challenges: 1) the considerable
variability in prosody and articulation habits across languages; and 2) the
rarity of paired multi-lingual datasets from the same speaker. In this paper,
we propose MulliVC, a novel voice conversion system that only converts timbre
and keeps original content and source language prosody without multi-lingual
paired data. Specifically, each training step of MulliVC contains three
substeps: In step one the model is trained with monolingual speech data; then,
steps two and three take inspiration from back translation, construct a
cyclical process to disentangle the timbre and other information (content,
prosody, and other language-related information) in the absence of
multi-lingual data from the same speaker. Both objective and subjective results
indicate that MulliVC significantly surpasses other methods in both monolingual
and cross-lingual contexts, demonstrating the system's efficacy and the
viability of the three-step approach with cycle consistency. Audio samples can
be found on our demo page (mullivc.github.io).