# Overview of Voice Conversion Methods Based on Deep Learning
## 3 Challenges  
Although neural networks themselves adapt very well to new datasets, there are other problems that the presented algorithms solve or, in contrast, generate.  
One problem may be the lack of sufficient training data. Algorithms of deep neural networks need a large amount of training data to train neural networks correctly, and in the case of voice conversion, even more data is usually needed. Algorithm developers are looking for solutions to disentangle time-variant features from these time-invariant ones to swap some of these features. This approach opens access to databases that do not require advanced labeling. Thus, developers mostly use a single training database (VCTK [55]), which allows for a good performance comparison in a constrained environment. Concerning the presented applications, algorithms using VAE require fewer training data than those based on GANs or attentions. The authors also see the importance of the training data in using an external pre-trained vocoder. Developers do not train it from scratch for the same dataset as the converter but instead use an already appropriately generalized model.  
Another problem is the complexity of human speech and its appropriate resolution. VAE-based algorithms often require careful adjustment of the bottleneck, which will prevent redundant information from being transferred to the generated signal, and thus, degrading it. GANs themselves, on the other hand, are not always able to build a sufficiently generalized distribution to allow for the generation of samples outside the training data area. This problem is evident, for example, in StarGANv2-VC [15], which does not even include any-to-any covariance in its coverage, or even in FreeVC [16], where one can see the poor performance of the similarity MOS for out-of-sample data. The developers of FreeVC [16] and HiFi-VC [7] implemented a combination of these models, obtaining excellent results in many-to-many conversion but worse results in SMOS for any-to-any conversion.  
One standard procedure for developing a conversion pipeline is for developers to use pre-trained speaker encoders. This action can generate problems related to the inadequate quality of a specific component contributing to a decrease in the entire system's performance. However, algorithms are emerging, such as the input-degrading FreeVC spectrogram [16] or the fully VAE-based [11], which integrate speaker feature extractor training into a single training loop jointly with linguistic feature extraction, which solves this problem to some extent. In addition, using advanced models, which are most often trained for other purposes  
\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline \multirow{2}{*}{**Paper**} & \multirow{2}{*}{**Evaluations Methods**} & **MOS** & **MOS** & **MOS** & **MOS** & \multirow{2}{*}{**Dataset**} & **Public** \\  & & **M-M** & **M-M** & **A-A** & **A-A** & & **Code/Demo** \\  & & **Quality** & **Similarity** & **Quality** & **Similarity** & & \\ \hline \multirow{4}{*}{[13]} & MOSNet [64], average & \multirow{4}{*}{3.74} & \multirow{4}{*}{-} & \multirow{4}{*}{3.58} & \multirow{4}{*}{-} & \multirow{4}{*}{VCTK [55]} & \multirow{4}{*}{\(\boldsymbol{\chi/\chi}\)} \\  & speaker classification & accuracy & & & & & \\ \cline{1-1} \cline{5-7} WaveNet [14] & & 3.40 & 3.56 & 3.22 & 3.54 & \\ \cline{1-1} \cline{5-7} HiFi-GAN [65] & & 3.76 & 3.83 & 3.65 & 3.89 & \\ \cline{1-1} \cline{2-7} \cline{5  
in the model, can slow down the algorithm's performance during use and training. Among others, one such algorithm, Wav2 vec, was used in [11].  
Except for [11], the algorithms presented do not have built-in mechanisms that improve performance with access to more data during use. Working on algorithms of this type can increase the quality of the generated samples.