## 3 Proposed Method
Since the output of the generator model is waveforms, they need to be converted to spectral features to calculate the losses. The losses for the other conversion direction, i.e., \(\mathcal{L}_{\text{adv1}}(G_{B\to A},D_{A})\), \(\mathcal{L}_{\text{adv2}}(G_{B\to A},D_{A})\), \(\mathcal{L}_{\text{cyc}}(G_{B\to A})\), \(\mathcal{L}_{\text{id}}(G_{B\to A})\), and \(\mathcal{L}_{\text{fm}}(G_{B\to A})\), are similarly defined. In the conventional analysis-mapping-reconstruction approaches, such as in Figure 1, the discriminator of the spectral feature conversion model and the vocoder model serve different purposes. The discriminator in the spectral feature conversion model aims to distinguish the speaker's identity from the target speaker's spectral feature and the converted  
Figure 4: Architecture of the proposed feature extractor model. spectral feature. On the other hand, the discriminator in the vocoder model aims to distinguish the original and synthesized waveforms. Since the generator in the proposed method includes the function of a vocoder, the loss in the proposed method is calculated by using the discriminator of the spectral feature conversion model and the vocoder model together. Through this, the proposed voice conversion model can generate waveforms similar to those from the target speaker while maintaining the quality of the original waveforms. We used a modified version of HiFi-GAN as the generator of the proposed method. Therefore, the discriminators of the proposed method utilize the HiFi-GAN discriminator, as well as the conventional GAN discriminator. One important limitation of GAN-based models is that they are difficult to train, especially for complex models like the proposed one, where the HiFi-GAN-style vocoder is used as the generator in the CycleGAN-style training. Since it is not easy to stabilize the training process and to avoid the collapsing problem, we applied two techniques to alleviate this problem. The first technique was to pre-train the generators using the speech data of the source and target speakers in the training data. Since the supervised training of the generator was relatively easy to converge, we first trained them to bootstrap the training in the CycleGAN-style training process. The second technique was to use a phase-wise model update method. The generator and discriminator models were trained in two alternating phases. The first phase used the following loss function:  
\[\mathcal{L}_{\text{phasel}}=\mathcal{L}_{\text{adv1}}(G_{A\to B },D_{B})+\mathcal{L}_{\text{adv1}}(G_{B\to A},D_{A})+\lambda_{\text{id}} \left(\mathcal{L}_{\text{id}}(G_{A\to B})+\mathcal{L}_{\text{id}}(G_{B\to A })\right)\,, \tag{9}\]  
where \(\lambda_{\text{id}}\) is the weight for the identity-mapping loss. The second phase utilized the following loss function:  
\[ \begin{array}{rl}\mathcal{L}_{\text{phas2}}=\mathcal{L}_{\text{adv2}}(G_{A \to B},D_{B})+\mathcal{L}_{\text{adv2}}(G_{A\to B},MSD_{B})+\mathcal{L}_{ \text{adv2}}(G_{A\to B},MPD_{B})\\ \qquad\qquad+\mathcal{L}_{\text{adv2}}(G_{B\to A},D_{A})+\mathcal{L}_{\text{ adv2}}(G_{B\to A},MSD_{A})+\mathcal{L}_{\text{adv2}}(G_{B\to A},MPD_{A})\\ \qquad\qquad+\lambda_{\text{cyc}}\big{(}\mathcal{L}_{\text{cyc}}(G_{A\to B})+ \mathcal{L}_{\text{cyc}}(G_{B\to A})\big{)}\\ \qquad\qquad+\lambda_{\text{fm}}(\mathcal{L}_{\text{fm}}(G_{A\to B})+ \mathcal{L}_{\text{fm}}(G_{B\to A}))\end{array}  \tag{10}\]  
where \(\lambda_{\text{cyc}}\) and \(\lambda_{\text{fm}}\) are the weights for the cycle consistency loss and feature-matching loss, respectively; \(MSD_{A}\), \(MPD_{A}\), \(MSD_{B}\), and \(MPD_{B}\) are the MSD and MPD of speaker \(A\) and speaker \(B\), respectively. Algorithm 1 summarizes the alternating phase-wise training process of the proposed method. The parameters of the two generators are initialized with the pre-trained vocoder using the speech data from the source and target speakers. The parameters of the feature-extracting CNN are also initialized with the coefficients of the DFT matrix and mel-filterbanks.