# CycleDiffusion: Voice Conversion Using Cycle-Consistent Diffusion Models
###### Abstract  
Voice conversion (VC) refers to the technique of modifying one speaker's voice to mimic another's while retaining the original linguistic content. This technology finds its applications in fields such as speech synthesis, accent modification, medicine, security, privacy, and entertainment. Among the various deep generative models used for voice conversion, including variational autoencoders (VAEs) and generative adversarial networks (GANs), diffusion models (DMs) have recently gained attention as promising methods due to their training stability and strong performance in data generation. Nevertheless, traditional DMs focus mainly on learning reconstruction paths like VAEs, rather than conversion paths as GANs do, thereby restricting the quality of the converted speech. To overcome this limitation and enhance voice conversion performance, we propose a cycle-consistent diffusion (CycleDiffusion) model, which comprises two DMs: one for converting the source speaker's voice to the target speaker's voice and the other for converting it back to the source speaker's voice. By employing two DMs and enforcing a cycle consistency loss, the CycleDiffusion model effectively learns both reconstruction and conversion paths, producing high-quality converted speech. The effectiveness of the proposed model in voice conversion is validated through experiments using the VCTK (Voice Cloning Toolkit) dataset. c 2024 14 9595 10.3390/2024 14 9595 10. Since the cycle-consistent adversarial network (CycleGAN) [8] was introduced for image-to-image translation, GAN-based voice conversion has been extensively studied [9; 10; 11; 12; 13; 14; 15; 16]. A CycleGAN comprises two GANs. One GAN is trained to transform the voice of the source speaker into that of the target speaker, while the other GAN is trained to revert the target speaker's voice back to the original source speaker's voice. This approach utilizes non-parallel training data and, unlike VAEs, it shows superior quality in converted speech by learning voice conversion paths explicitly, instead of just input reconstruction paths. Nonetheless, GANs suffer from training instability and mode collapse, and building many-to-many voice conversion models with CycleGANs involves significant computational complexity. Recently, diffusion models (DMs) [17; 18; 19; 20; 21; 22; 23; 24] have emerged as promising generative models. A DM is trained to reconstruct the original input data by progressively adding noise to the input data through a forward diffusion process, resulting in a noise distribution, and then progressively removing the noise through a reverse diffusion process. DMs are characterized by stability during training and demonstrate excellent performance in generating high-quality data via reverse diffusion. DMs have been applied not only to image generation but also to voice conversion [25; 26; 27]. However, DMs fundamentally learn only the reconstruction paths, similar to VAEs, rather than the conversion paths as GANs do, leaving much to be improved in the quality of the converted speech. To address this limitation of conventional DMs and enhance the quality of converted speech, we propose a cycle-consistent diffusion (CycleDiffusion) model, which consists of two DMs: one to convert the source speaker's voice to the target speaker's voice, and the other to revert the target speaker's voice back to the source speaker's voice. By employing two DMs and enforcing a cycle consistency loss, the proposed CycleDiffusion model can effectively learn not only the reconstruction paths but also the conversion paths, leading to high-quality converted speech. The effectiveness of the proposed model for voice conversion is experimentally validated using VCTK (Voice Cloning Toolkit) data. The contributions of this paper are as follows:  
* We propose a novel CycleDiffusion model that can learn both the reconstruction and conversion paths. * We provide an efficient training algorithm so that the proposed CycleDiffusion model can be reliably trained.