# Overview of Voice Conversion Methods Based on Deep Learning
## 2 Voice Conversion Process
### Vocoders  
Vocoders are tools used to convert a speech spectrogram into sound waves. They are crucial to the voice conversion process, as they enable the generation of the appropriate sound based on the spectrogram. Table 1 compares the described methods in terms of the solutions used. The described methods use Parallel WaveGAN, WaveNet [36], and HiFi-GAN [35]. WaveNet is a neural network architecture that DeepMind proposed in 2016. WaveNet is a generative model that can generate audio data based on previous predictions. The authors proposed a model architecture based on PixelCNN [41], which is applied to images. WaveNet is very effective at generating natural voice sounds because it relies on dilated convolutions, which allow for modeling a wide range of information in the input data. Dilated convolutions previously used in signal processing were used in various contexts, e.g., signal processing [42; 43] and image segmentation [44; 45]. In addition, the authors noted that using gated activation units in PixelCNN works better in modeling audio signals than rectified linear activation functions [46]. Authors also used residual connections [47] to accelerate training for deeper models. Parallel WaveGAN is a GAN-based generative model that was proposed in 2020. WaveGAN is specifically designed to generate audio data, such as nature sounds or music. The model uses GAN to develop an artificial audio signal. WaveGAN models are very effective at generating audio data that is highly realistic and natural. The developers used two models: a generator and a discriminator typically used for GANs [48]. As part of the generator, they used a modified WaveNet, but the authors used non-causal convolutions instead of causal convolutions; the input is random noise drawn from a Gaussian distribution, and the model is non-autoregressive at both the training and inference steps. In addition, to increase the stability, they introduced multi-resolution STFT auxiliary loss [49; 50]. Combining multiple STFT losses with different analysis parameters helps the generator to obtain the time-frequency characteristics of speech [51]. As research showed [52], the best performance in vocoder tasks among the presented three was achieved by the third model: the HiFi-GAN. The authors of the generator developed a proprietary model using multi-receptive field fusion to work in parallel on \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Paper** &
\begin{tabular}{c} **Speaker Identity** \\ **Extraction** \\ \end{tabular} & **Linguistic Content Extraction** & **Generation** & **Vocoder** \\ \hline
[8] & GE2E [18] & VQWav2 vec features [9] &
\begin{tabular}{c} Dynamic convolutions [10] \\ WadaIN [34] \\ LN + RC as in [21] \\ \end{tabular} & Parallel WaveGAN [33] \\ \hline
[7] & 5-layer residual FC similar to [37] &
\begin{tabular}{c} Linguistic encoder: \\ TTS Skins [26] \\ Conformer [27] \\ F0 encoder: \\ BNE-Seq2 seqMol [25] \\ \end{tabular} & Modified HiFi-GAN [35] & Modified HiFi-GAN [35] \\ \hline
[11] & ECAPA-TDNN [22] & Wav2 vec [28] & \
\begin{tabular}{c} Vec2 wav model based on \\ HiFi-GAN [35] \\ \end{tabular} &
\begin{tabular}{c} Vec2 wav model based on \\ HiFi-GAN [35] \\ \end{tabular} \\ \hline
[13] & \(\beta\)-VAE [38]—average distribution for each audio & \(\beta\)-VAE [38]—individual distribution for each audio &
\begin{tabular}{c} \(\beta\)-VAE [38] \\ RGSM [23] \\ Attention [21] \\ Post-Net as in [39] \\ \end{tabular} & WaveNet [36] \\ \hline
[14] &
\begin{tabular}{c} Modified DSVAE [40]—time-invariant disentanglement \\ \end{tabular} & Modified DSVAE [40]—time-variant disentanglement & Modified AutoVC Decoder [20] & WaveNet [36] \\ \hline
[15] & Mapping network/style encoder & \
\begin{tabular}{c} Encoder + \\ F0 Encoder: \\ JDC network [29] \\ \end{tabular} &
\begin{tabular}{c} Encoder output + F0 \\ output + style injected by \\ AdaIN [32] \\ \end{tabular} & Parallel WaveGAN [33] \\ \hline
[16] &
\begin{tabular}{c} Prior encoder: \\ WavLM [30] \\ \end{tabular} & HiFi-GAN [35] & HiFi-GAN [35] \\ \end{tabular}
\end{table}