# Effects of Sinusoidal Model on Non-Parallel Voice Conversion with Adversarial Learning
## 2 Related Work and Motivation  
In recent times, successful efforts have been considered to develop non-parallel methods. Among them, a conditional variational autoencoder (CVAE) framework was proposed in [23], consisting of two main networks: encoder and decoder. The idea is that the input speech samples are firstly converted by an encoder to latent vectors having the input linguistic variables. Then, a one-hot vector having the target speaker identity is created and later fed together with latent vectors into a decoder to produce the target utterance features. This approach is called a conditional VAE due to the decoder which is trained on a target speaker identity vector. One of the problems encountered here is that the decoder is over-smoothed. This over-smoothing effect is caused by the Gaussian assumptions on the encoder and decoder distributions. In addition, over-smoothness mostly occurs in spectral features as they have low-dimensional features. Parametric techniques usually suffer from over-smoothing because they use the minimum mean square error or the maximum likelihood function as the optimization criterion. Consequently, VAE fails to capture the desired details of temporal and spectral dynamics. This commonly results in poor-quality converted speech.  
One powerful technique that can possibly overcome the weakness of CVAEs is the cycle-consistent generative adversarial network (CycleGAN), which was proposed in [24] using generators, discriminators, and an identity-mapping loss. The idea is that the source features are transformed to match the features of a target voice via a GAN model. Then, the result is again converted back to match the source characteristics via an additional GAN. A conjunction of both cycle consistency and adversarial losses is finally used to force the linguistic contexts to be retained in the converted speech. The adversarial loss only tells us whether the generator follows the distribution of target data and does not ensure that the contextual information is preserved. The cycle-consistency loss is introduced to encourage the generator to find the mapping that preserves underlying linguistic content between the input and output. For these reasons, CycleGAN represents a successful deep learning implementation to find an optimal pseudo pair from non-parallel data of paired speakers. It does not require any frame alignment mechanism such as dynamic time warping or attention. While this approach was found to work reasonably well in one-to-one mappings, CycleGAN fails to work in many-to-many VC tasks because it may require several generator-discriminator pairs trained separately. Hence, this increases the computational complexity of learning parameters, training time, and the memory space prohibitively. For VC applications, this can degrade conversion performance as multiple speakers are trained simultaneously. To resolve this issue, StarGAN-VC [25] was recently introduced as a unified model architecture. It accepts concurrent training of various domains, i.e., many-to-many mapping within a single network. It can also be noted that StarGAN gives good quality among the GAN-based VC frameworks. However, it lacks stable training [26], which can be overcome by the proposed sinusoidal model using continuous parameters.  
Despite the vast amount of research that has been provided in the literature for non-parallel voice conversion, the synthesized speech obtained from converted features is still far from achieving the quality of the target speaker. Therefore, the new generation of high-quality converted speech is still very challenging and has room for improvement. Motivated by this, we propose a new less expensive sinusoidal architecture as an independent study, which is the first GAN-based sinusoidal model to perform non-parallel VC with only a few training examples. We observe that the proposed model is able to give objective results more efficiently than the StarGAN scheme, and can achieve a good speech quality to the target speaker.