# CycleDiffusion: Voice Conversion Using Cycle-Consistent Diffusion Models
## 3 Proposed Method  
### Cycle-Consistent Diffusion (CycleDiffusion)  
As detailed in Section 2.1, conventional DM-based voice conversion learns only the reconstruction path as a VAE does and does not learn the conversion path, resulting in significant room for improvement in the quality of the converted speech. Therefore, we propose to utilize cycle consistency loss for training diffusion models, referred to as cycle-consistent diffusion (or CycleDiffusion for short). The proposed method aims to learn both the reconstruction path and the conversion path while enforcing that linguistic information remains unchanged. Figure 1 depicts the proposed CycleDiffusion training for a two-speaker scenario. Initially, the DM is trained to reconstruct the input. The input \(\mathbf{x}^{\zeta}\) from speaker \(\zeta\) is reconstructed to \(\mathbf{x}^{\zeta}\), and the input \(\mathbf{x}^{\zeta}\) from speaker \(\xi\) is reconstructed to \(\mathbf{x}^{\zeta}\), which is typical DM training using \(\mathcal{L}_{\text{diffusion}}\big{(}\mathbf{x}^{\zeta},\zeta\big{)}\) and \(\mathcal{L}_{\text{diffusion}}\big{(}\mathbf{x}^{\zeta},\xi\big{)}\), respectively. As the usual reconstruction path training stabilizes, the conversion path training commences as follows. The input \(\mathbf{x}^{\zeta}\) is converted to \(\mathbf{x}^{\zeta-\xi}\) through Equations (5) and (6):  
\[\mathbf{x}^{\zeta-\xi}=\mathcal{R}\Big{(}\mathcal{F}\Big{(}\mathbf{x}^{\zeta}\Big{)}, \xi\Big{)}. \tag{10}\]  
Subsequently, the converted speech vector \(\mathbf{x}^{\zeta-\xi}\) is reverted to \(\mathbf{x}^{\zeta-\xi-\gamma\xi}\) again via Equations (5) and (6):  
\[\mathbf{x}^{\zeta-\xi-\gamma\xi-\zeta}=\mathcal{R}\Big{(}\mathcal{F}\Big{(}\mathbf{x}^ {\zeta-\gamma\xi}\Big{)},\xi\Big{)}. \tag{11}\]  
Finally, the cycle consistency loss between \(\mathbf{x}^{\zeta}\) and \(\mathbf{x}^{\zeta-\gamma\xi-\gamma\xi}\) is minimized using \(\mathcal{L}_{\text{cycle}}\big{(}\mathbf{x}^{\zeta},\mathbf{x}^{\zeta-\gamma\xi- \gamma\xi}\big{)}\). Similarly, the cycle consistency loss between \(\mathbf{x}^{\zeta}\) and \(\mathbf{x}^{\zeta-\gamma\xi-\gamma\xi}\) is minimized as well. This cycle consistency loss minimization helps improve voice conversion performance in two aspects. First, the conversion path from speaker \(\xi\) to speaker \(\zeta\) is explicitly trained by reducing the difference between the original target speaker's speech \(\mathbf{x}^{\zeta}\) and the converted speech \(\mathbf{x}^{\zeta-\gamma\xi-\zeta}\), which originates from the source speaker's speech \(\mathbf{x}^{\zeta-\gamma\xi}\). That is, the input \(\mathbf{x}^{\zeta-\gamma\xi}\) comes from speaker \(\xi\) and the output \(\mathbf{x}^{\zeta-\gamma\xi-\gamma\xi}\) belongs to speaker \(\zeta\). Similarly, the conversion path from speaker \(\zeta\) to speaker \(\zeta\) is explicitly trained as well. Second, since both \(\mathbf{x}^{\zeta}\) and \(\mathbf{x}^{\zeta-\gamma\xi-\gamma\xi}\) (as well as \(\mathbf{x}^{\zeta}\) and \(\mathbf{x}^{\zeta-\gamma\xi-\gamma\xi}\)) should represent the same utterance, minimizing their differences ensures that the linguistic information remains unchanged. The overall loss function for CycleDiffusion training is as follows:  
\[\mathcal{L}=\mathcal{L}_{\text{diffusion}}\big{(}\mathbf{x}^{\zeta},\xi\big{)}+ \mathcal{L}_{\text{diffusion}}\big{(}\mathbf{x}^{\zeta},\xi\big{)}+\lambda_{i} \Big{(}\mathcal{L}_{\text{cycle}}\big{(}\mathbf{x}^{\zeta},\mathbf{x}^{\zeta-\gamma \xi-\gamma\xi}\big{)}+\mathcal{L}_{\text{cycle}}\big{(}\mathbf{x}^{\zeta},\mathbf{x}^ {\zeta-\gamma\xi-\gamma\xi}\big{)}\Big{)}, \tag{12}\]  
where \(\lambda_{i}\) is a weight schedule for the cycle consistency loss, starting at 0 and gradually increasing to \(\lambda_{\text{max}}\) as the training progresses. This approach can be easily extended to accommodate multiple speakers. Figure 1: The proposed CycleDiffusion model for a two-speaker case.