ClsVC: Learning Speech Representations with two different classification tasks.

1 INTRODUCTION . Voice conversion ( VC ) is an exciting topic committed to converting one utterance of a source speaker into another utterance of a target person by keeping the content in the original utterance while replacing it with the vocal features from the target speaker . Up to now , many methods have been applied in VC successfully . Commonly , these methods can be roughly categorized into two classes , i.e. , parallel VC and non-parallel VC Mohammadi & Kain ( 2017 ) . Specifically , parallel VC means that model training requires parallel corpus , which is unnecessary for non-parallel VC . Recently , more researchers have focused on the solutions of non-parallel VC since it is not easy for us to collect so many paired source-target speech datasets . Early VC systems , like Gaussian Mixture Model ( GMM ) , Stylianou et al . ( 1998 ) ; Toda et al . ( 2007 ) needed a lot of parallel data for model training , and the generated speech quality was not good enough . With the advance of deep learning , a variety of novel VC methods have been proposed in recent years . Among them , GAN-based model is one of the most popular methods , Goodfellow et al . ( 2014 ) ; Hsu et al . ( 2017a ) ; Kaneko & Kameoka ( 2018 ) ; Kaneko et al . ( 2019a ) ; Kameoka et al . ( 2018 ) ; Kaneko et al . ( 2019b ) which could learn a global generative distribution of the target speech without explicit approximation . These GAN-based models jointly train a generator and a discriminator . An adversarial loss derived from the discriminator is used to encourage the generator outputs to build indistinguishable from real speech . Thanks to the cycle consistency training , GANbased VC models can be trained with non-parallel speech datasets . Besides , learning discrete speech representations has also attracted much attention . Vector Quantization ( VQ ) , an extremely important signal compression method , which can quantify continuous data into discrete data . Previous studies have confirmed that the quantized discrete data produced by the continuous speech data is closely related with the phoneme information Chorowski et al . ( 2019 ) . Recently , VQVC Wu & Lee ( 2020 ) has been proposed to learn to disentangle the content and speaker information with reconstruction loss only . Then , VQVC+ Wu et al . ( 2020 ) was soon presented to improve the conversion performance of VQVC by adding the U-Net architecture within an auto-encoder-based VC system . To far improve the performance of disentangling content and speaker information , many other existing studies were introduced to combine with VQ , such as VQ-Wav2Vec , VQ-VAE and VQ-CPC . Baevski et al . ( 2019 ) ; Ding & Gutierrez-Osuna ( 2019 ) ; van Niekerk et al . ( 2020 ) There is also another line of research focus on learning latent representations with Autoencoder . In particular , Variational Auto Encoder ( VAE ) is the most famous . The network structure of VAE contains an encoder and a decoder and the core idea is very clear : the encoder learns a specific latent space from input speech and the decoder outputs a reconstructed speech from this latent space . In this process , VAE focuses on how to force the encoder to learn a specific latent space . So far , many VAE-based models have been successfully applied to VC Hsu et al . ( 2016 ) ; van den Oord et al . ( 2017 ) ; Hsu et al . ( 2017b ) ; Ding & Gutierrez-Osuna ( 2019 ) ; Hsu et al . ( 2017a ) . In addition , AutoVC is another successful application of Autoencoder . Qian et al . ( 2019 ) Through ingenious experimental design , AutoVC uses two different encoders to learn content and speaker information , respectively , so that this model can achieve distribution-matching style transfer by only on a self-reconstruction loss . Unfortunately , in the field of VC , all the models mentioned above have their inherent disadvantages . For example , GAN-based models can usually achieve a good conversion effect and ensure the matching between the generated data and the input data , but it is recognized that the training of GAN is very unstable . On the contrary , the training of VQVC is simple and fast enough , but the quantity of audio produced by this method is very poor . This may be because the discrete speech representations will inevitably lose some content information . In addition , although the VAE-based model also has a great conversion effect , it can not guarantee distribution matching . AutoVC is a great study , the training is very simple and achieves state-of-the-art results . However , in order to realize style conversion , it has to introduce a pre-trained speaker encoder . Based on these existing methods , we naturally wonder if there is a new solution that can achieve the distribution matching as AutoVC and GAN , that trains as easily as VQVC and VAE , that can disentangle content and speaker information by only one encoder as VQ does , and also has better performance in voice conversion or in decoupling linguistic and timbre information from speech ? In this paper , we proposed a novel voice conversion framework to meet all the above requirements . Specifically , our model is similar to VAE , Autoencoder is the main framework of our model , and two different types of classification tasks are applied to force our model to separate the content and speaker information correctly . Here , the two classification tasks refer to general classification tasks and adversarial classification tasks respectively . The goal of the general classification task is to identify the features related to the speaker as accurately as possible , that is , the speaker information . While the latter is designed to eliminate speaker information in latent space to get speakerindependent features , that is , content information . Experiment results are carried out on the VCTK dataset . Objective and Subjective evaluations demonstrate that the proposed method outperforms VQVC , AutoVC , VQ-VAE and StarGAN-VC in terms of naturalness and speaker similarity . 2 BACKGROUND . In mathematical statistics , if we already know the joint probability density functions of X and Y , we can easily find the marginal probability density functions of X and Y respectively . Formally , if ( x , y ) ∼ p ( x , y ) is known , we can get the marginal distributions of X and Y by the following formula : fX ( x ) = w p ( x , y ) dy fY ( y ) = w p ( x , y ) dx ( 1 ) Further , under the constraints of some setting conditions , although the closed-form of joint probability density functions p ( x , y ) in Eq . ( 1 ) is generally unknown , it is still feasible for a neural network to learn the marginal distributions of x and y from input samples z when each z corresponds to a unique pair of ( x , y ) . Mutual information ( MI ) , a crucial indicator to measure the dependence between two different variables . Recently , many MI estimators have been successfully applied to constrain neural networks to disentangle different components of the input data . which can be formulated as I ( X , Y ) = ∫ X ∫ Y P ( X , Y ) log P ( X , Y ) P ( X ) P ( Y ) ( 2 ) Where P ( X ) and P ( Y ) are the marginal distributions of X and Y respectively , and P ( X , Y ) denotes the joint distribution of X and Y . Since it is hard to obtain the required distribution formula P ( X , Y ) , many studies focus on proposing a sample-based MI lower bound ( or upper bound ) to get an approximation that can be calculated . Moon & Hero ( 2014 ) ; Hjelm et al . ( 2018 ) ; Cheng et al . ( 2020 ) Recently , Yuan et al . Yuan et al . ( 2020 ) , introduced a new MI estimator to learn content and style information from speech for voice conversion . Specifically , they proposed a novel MI-based learning objective to encourage a content encoder to output the content embedding and guide a speaker encoder to output the speaker embedding . Inspired by this , we proposed a new , simple and more effective framework for learning latent speech representation . Gradient Reversal Layer ( GRL ) Gradient Reversal Layer ( GRL ) was first proposed to address the issue of domain adaption Ganin & Lempitsky ( 2015 ) , which aims to force model to output domainshared features which are independent with domains . Specifically , GRL is often located between an encoder and a domain classifier . During the forward propagation , GRL acts as an identity transform . During the backpropagation , GRL takes the gradient from the subsequent level , multiplies it by −1 and passes it to the preceding layer so that the encoder and domain classifier have completely opposite optimization objectives . 3 METHOD . Firstly , for every speech x , we use content embedding Cx to represent linguistic information and speaker embedding Sx is proposed to represent timbre and style information . And , U means the set of speakers . The following two theorems are the premise of our framework : Theorem 3.1 The content embedding Cx and speaker identity U are independent for each other , In addition , the probability of each speaker ’ s speech being selected is the same , Formally , P ( U = u|Cx ) = P ( U = u ) = constant regardless of the speaker identity u . Theorem 3.2 The speaker embeddingSx and speaker identityU are in one-to-one correspondence . That is , for a speaker u who produced speech x , P ( U = u|Sx ) = 1 and for other speakers v ( v 6= u ) , P ( U = v|Sx ) = 0 . 3.1 PROBLEM FORMULATION . Given a dataset of multi speakers and their audio recordings X , where speaker u has Nu audio recordings . Formally , for each speaker u , Xu = { xui } Nu i=1 . For every input speech x ∈ X , we use T to represent the number of frames of speech x , noted that T is constant , which means that for any speech segments longer than T , we randomly select T frames , at the same time , for those speech segments with a length shorter than T , we pad them with constant . In this case , x can also expressed as x ( 1 : T ) which is a random process randomly sampled from the speech distribution pX ( ·|CX = Cx , SX = Sx ) . Here , we assume that every speech x can be expressed as a function f ( · ) of content embeddingCx and speaker embedding Sx . That is , x = f ( Cx , Sx ) . And , we also assume that x is uniquely determined by Cx and Sx . Formally , x1 = x2 only if Cx1 = Cx2 and Sx1 = Sx2 . Based on this assumption , for ∀ xi , xj ∈ X , the following formula must be true : I ( Sxi , f ( Cx , Sxi ) ) = I ( Sxj , f ( Cx , Sxj ) ) , I ( Cxi , f ( Cxi , Sx ) ) = I ( Cxj , f ( Cxj , Sx ) ) . ( 3 ) Now , assume two speech x1 and x2 from speaker u and speaker v respectively , x1 = f ( Cx1 , Sx1 ) and x2 = f ( Cx2 , Sx2 ) . Our goal is to design a speech conversion framework to generate a new speech x̂1→2 which preserves the content information of x1 but the speaker information is matched with x2 . Formally , an ideal converted speech should satisfy the following forms : I ( Sx1 , x1 ) = I ( Sx2 , x̂1→2 ) , I ( Cx2 , x2 ) = I ( Cx1 , x̂1→2 ) . ( 4 ) Based on the above assumption , the formula in Eq . ( 4 ) can be equivalent to x̂1→2 = f ( Cx1 , Sx2 ) . This conclusion is quite strong , and the formal proof will be presented in the appendix .

Summary: The paper has proposed a VC system called ClsVC by learning the latent speech representation from reference speech.  To ensure the disentanglement of speaker and content inforamtion, the proposed method separates the latent speech representation into two parts, for speaker and content embedding respectively.  To ensure the performance of disentanglement, a common speaker classifier and an adversarial speaker classifier are proposed.  Furthermore, a new loss called code-reconstruction loss is proposed. Experimental results validate the effectiveness of the proposed method.