ClsVC: Learning Speech Representations with two different classification tasks.

1 INTRODUCTION . Voice conversion ( VC ) is an exciting topic committed to converting one utterance of a source speaker into another utterance of a target person by keeping the content in the original utterance while replacing it with the vocal features from the target speaker . Up to now , many methods have been applied in VC successfully . Commonly , these methods can be roughly categorized into two classes , i.e. , parallel VC and non-parallel VC Mohammadi & Kain ( 2017 ) . Specifically , parallel VC means that model training requires parallel corpus , which is unnecessary for non-parallel VC . Recently , more researchers have focused on the solutions of non-parallel VC since it is not easy for us to collect so many paired source-target speech datasets . Early VC systems , like Gaussian Mixture Model ( GMM ) , Stylianou et al . ( 1998 ) ; Toda et al . ( 2007 ) needed a lot of parallel data for model training , and the generated speech quality was not good enough . With the advance of deep learning , a variety of novel VC methods have been proposed in recent years . Among them , GAN-based model is one of the most popular methods , Goodfellow et al . ( 2014 ) ; Hsu et al . ( 2017a ) ; Kaneko & Kameoka ( 2018 ) ; Kaneko et al . ( 2019a ) ; Kameoka et al . ( 2018 ) ; Kaneko et al . ( 2019b ) which could learn a global generative distribution of the target speech without explicit approximation . These GAN-based models jointly train a generator and a discriminator . An adversarial loss derived from the discriminator is used to encourage the generator outputs to build indistinguishable from real speech . Thanks to the cycle consistency training , GANbased VC models can be trained with non-parallel speech datasets . Besides , learning discrete speech representations has also attracted much attention . Vector Quantization ( VQ ) , an extremely important signal compression method , which can quantify continuous data into discrete data . Previous studies have confirmed that the quantized discrete data produced by the continuous speech data is closely related with the phoneme information Chorowski et al . ( 2019 ) . Recently , VQVC Wu & Lee ( 2020 ) has been proposed to learn to disentangle the content and speaker information with reconstruction loss only . Then , VQVC+ Wu et al . ( 2020 ) was soon presented to improve the conversion performance of VQVC by adding the U-Net architecture within an auto-encoder-based VC system . To far improve the performance of disentangling content and speaker information , many other existing studies were introduced to combine with VQ , such as VQ-Wav2Vec , VQ-VAE and VQ-CPC . Baevski et al . ( 2019 ) ; Ding & Gutierrez-Osuna ( 2019 ) ; van Niekerk et al . ( 2020 ) There is also another line of research focus on learning latent representations with Autoencoder . In particular , Variational Auto Encoder ( VAE ) is the most famous . The network structure of VAE contains an encoder and a decoder and the core idea is very clear : the encoder learns a specific latent space from input speech and the decoder outputs a reconstructed speech from this latent space . In this process , VAE focuses on how to force the encoder to learn a specific latent space . So far , many VAE-based models have been successfully applied to VC Hsu et al . ( 2016 ) ; van den Oord et al . ( 2017 ) ; Hsu et al . ( 2017b ) ; Ding & Gutierrez-Osuna ( 2019 ) ; Hsu et al . ( 2017a ) . In addition , AutoVC is another successful application of Autoencoder . Qian et al . ( 2019 ) Through ingenious experimental design , AutoVC uses two different encoders to learn content and speaker information , respectively , so that this model can achieve distribution-matching style transfer by only on a self-reconstruction loss . Unfortunately , in the field of VC , all the models mentioned above have their inherent disadvantages . For example , GAN-based models can usually achieve a good conversion effect and ensure the matching between the generated data and the input data , but it is recognized that the training of GAN is very unstable . On the contrary , the training of VQVC is simple and fast enough , but the quantity of audio produced by this method is very poor . This may be because the discrete speech representations will inevitably lose some content information . In addition , although the VAE-based model also has a great conversion effect , it can not guarantee distribution matching . AutoVC is a great study , the training is very simple and achieves state-of-the-art results . However , in order to realize style conversion , it has to introduce a pre-trained speaker encoder . Based on these existing methods , we naturally wonder if there is a new solution that can achieve the distribution matching as AutoVC and GAN , that trains as easily as VQVC and VAE , that can disentangle content and speaker information by only one encoder as VQ does , and also has better performance in voice conversion or in decoupling linguistic and timbre information from speech ? In this paper , we proposed a novel voice conversion framework to meet all the above requirements . Specifically , our model is similar to VAE , Autoencoder is the main framework of our model , and two different types of classification tasks are applied to force our model to separate the content and speaker information correctly . Here , the two classification tasks refer to general classification tasks and adversarial classification tasks respectively . The goal of the general classification task is to identify the features related to the speaker as accurately as possible , that is , the speaker information . While the latter is designed to eliminate speaker information in latent space to get speakerindependent features , that is , content information . Experiment results are carried out on the VCTK dataset . Objective and Subjective evaluations demonstrate that the proposed method outperforms VQVC , AutoVC , VQ-VAE and StarGAN-VC in terms of naturalness and speaker similarity . 2 BACKGROUND . In mathematical statistics , if we already know the joint probability density functions of X and Y , we can easily find the marginal probability density functions of X and Y respectively . Formally , if ( x , y ) ∼ p ( x , y ) is known , we can get the marginal distributions of X and Y by the following formula : fX ( x ) = w p ( x , y ) dy fY ( y ) = w p ( x , y ) dx ( 1 ) Further , under the constraints of some setting conditions , although the closed-form of joint probability density functions p ( x , y ) in Eq . ( 1 ) is generally unknown , it is still feasible for a neural network to learn the marginal distributions of x and y from input samples z when each z corresponds to a unique pair of ( x , y ) . Mutual information ( MI ) , a crucial indicator to measure the dependence between two different variables . Recently , many MI estimators have been successfully applied to constrain neural networks to disentangle different components of the input data . which can be formulated as I ( X , Y ) = ∫ X ∫ Y P ( X , Y ) log P ( X , Y ) P ( X ) P ( Y ) ( 2 ) Where P ( X ) and P ( Y ) are the marginal distributions of X and Y respectively , and P ( X , Y ) denotes the joint distribution of X and Y . Since it is hard to obtain the required distribution formula P ( X , Y ) , many studies focus on proposing a sample-based MI lower bound ( or upper bound ) to get an approximation that can be calculated . Moon & Hero ( 2014 ) ; Hjelm et al . ( 2018 ) ; Cheng et al . ( 2020 ) Recently , Yuan et al . Yuan et al . ( 2020 ) , introduced a new MI estimator to learn content and style information from speech for voice conversion . Specifically , they proposed a novel MI-based learning objective to encourage a content encoder to output the content embedding and guide a speaker encoder to output the speaker embedding . Inspired by this , we proposed a new , simple and more effective framework for learning latent speech representation . Gradient Reversal Layer ( GRL ) Gradient Reversal Layer ( GRL ) was first proposed to address the issue of domain adaption Ganin & Lempitsky ( 2015 ) , which aims to force model to output domainshared features which are independent with domains . Specifically , GRL is often located between an encoder and a domain classifier . During the forward propagation , GRL acts as an identity transform . During the backpropagation , GRL takes the gradient from the subsequent level , multiplies it by −1 and passes it to the preceding layer so that the encoder and domain classifier have completely opposite optimization objectives . 3 METHOD . Firstly , for every speech x , we use content embedding Cx to represent linguistic information and speaker embedding Sx is proposed to represent timbre and style information . And , U means the set of speakers . The following two theorems are the premise of our framework : Theorem 3.1 The content embedding Cx and speaker identity U are independent for each other , In addition , the probability of each speaker ’ s speech being selected is the same , Formally , P ( U = u|Cx ) = P ( U = u ) = constant regardless of the speaker identity u . Theorem 3.2 The speaker embeddingSx and speaker identityU are in one-to-one correspondence . That is , for a speaker u who produced speech x , P ( U = u|Sx ) = 1 and for other speakers v ( v 6= u ) , P ( U = v|Sx ) = 0 . 3.1 PROBLEM FORMULATION . Given a dataset of multi speakers and their audio recordings X , where speaker u has Nu audio recordings . Formally , for each speaker u , Xu = { xui } Nu i=1 . For every input speech x ∈ X , we use T to represent the number of frames of speech x , noted that T is constant , which means that for any speech segments longer than T , we randomly select T frames , at the same time , for those speech segments with a length shorter than T , we pad them with constant . In this case , x can also expressed as x ( 1 : T ) which is a random process randomly sampled from the speech distribution pX ( ·|CX = Cx , SX = Sx ) . Here , we assume that every speech x can be expressed as a function f ( · ) of content embeddingCx and speaker embedding Sx . That is , x = f ( Cx , Sx ) . And , we also assume that x is uniquely determined by Cx and Sx . Formally , x1 = x2 only if Cx1 = Cx2 and Sx1 = Sx2 . Based on this assumption , for ∀ xi , xj ∈ X , the following formula must be true : I ( Sxi , f ( Cx , Sxi ) ) = I ( Sxj , f ( Cx , Sxj ) ) , I ( Cxi , f ( Cxi , Sx ) ) = I ( Cxj , f ( Cxj , Sx ) ) . ( 3 ) Now , assume two speech x1 and x2 from speaker u and speaker v respectively , x1 = f ( Cx1 , Sx1 ) and x2 = f ( Cx2 , Sx2 ) . Our goal is to design a speech conversion framework to generate a new speech x̂1→2 which preserves the content information of x1 but the speaker information is matched with x2 . Formally , an ideal converted speech should satisfy the following forms : I ( Sx1 , x1 ) = I ( Sx2 , x̂1→2 ) , I ( Cx2 , x2 ) = I ( Cx1 , x̂1→2 ) . ( 4 ) Based on the above assumption , the formula in Eq . ( 4 ) can be equivalent to x̂1→2 = f ( Cx1 , Sx2 ) . This conclusion is quite strong , and the formal proof will be presented in the appendix .

Summary: This paper proposes methods for voice conversion. The proposed approach follows an encoder-decoder framework in which it adds classification tasks over the learned embeddings to enhance voice conversion. The representation space is divided into two parts - content and speaker, to enable disentangling of the content and the speaker. Experiments are shown on “traditional” and one-shot VC tasks. Objective as well as subjective scores are used to evaluate all systems.