MulliVC: Multi-lingual Voice Conversion With Cycle Consistency
Authors: name: Jiawei Huang, name: Chen Zhang, name: Yi Ren, name: Ziyue Jiang, name: Zhenhui Ye, name: Jinglin Liu, name: Jinzheng He, name: Xiang Yin, name: Zhou Zhao

Voice conversion aims to modify the source speaker's voice to resemble the
target speaker while preserving the original speech content. Despite notable
advancements in voice conversion these days, multi-lingual voice conversion
(including both monolingual and cross-lingual scenarios) has yet to be
extensively studied. It faces two main challenges: 1) the considerable
variability in prosody and articulation habits across languages; and 2) the
rarity of paired multi-lingual datasets from the same speaker. In this paper,
we propose MulliVC, a novel voice conversion system that only converts timbre
and keeps original content and source language prosody without multi-lingual
paired data. Specifically, each training step of MulliVC contains three
substeps: In step one the model is trained with monolingual speech data; then,
steps two and three take inspiration from back translation, construct a
cyclical process to disentangle the timbre and other information (content,
prosody, and other language-related information) in the absence of
multi-lingual data from the same speaker. Both objective and subjective results
indicate that MulliVC significantly surpasses other methods in both monolingual
and cross-lingual contexts, demonstrating the system's efficacy and the
viability of the three-step approach with cycle consistency. Audio samples can
be found on our demo page (mullivc.github.io).