AdaGAN: Adaptive GAN for Many-to-Many Non-Parallel Voice Conversion

1 INTRODUCTION . Language is the core of civilization , and speech is the most powerful and natural form of communication . Human voice mimicry has always been considered as one of the most difficult tasks since it involves understanding of the sophisticated human speech production mechanism ( Eriksson & Wretling ( 1997 ) ) and challenging concepts of prosodic transfer ( Gomathi et al . ( 2012 ) ) . In the literature , this is achieved using Voice Conversion ( VC ) technique ( Stylianou ( 2009 ) ) . Recently , VC has gained more attention due to its fascinating real-world applications in privacy and identity protection , military operations , generating new voices for animated and fictional movies , voice repair in medical-domain , voice assistants , etc . Voice Conversion ( VC ) technique converts source speaker ’ s voice in such a way as if it were spoken by the target speaker . This is primarily achieved by modifying spectral and prosodic features while retaining the linguistic information in the given speech signal ( Stylianou et al . ( 1998 ) ) . In addition , Voice cloning is one of the closely related task to VC ( Arik et al . ( 2018 ) ) . However , in this research work we only focus to advance the Voice Conversion . With the emergence of deep learning techniques , VC has become more efficient . Deep learningbased techniques have made remarkable progress in parallel VC . However , it is difficult to get parallel data , and such data needs alignment ( which is a arduous process ) to get better results . Building a VC system from non-parallel data is highly challenging , at the same time valuable for practical application scenarios . Recently , many deep learning-based style transfer algorithms have been applied for non-parallel VC task . Hence , this problem can be formulated as a style transfer problem , where one speaker ’ s style is converted into another while preserving the linguistic content as it is . In particular , Conditional Variational AutoEncoders ( CVAEs ) , Generative Adversarial Networks ( GANs ) ( proposed by Goodfellow et al . ( 2014 ) ) , and its variants have gained significant attention in non-parallel VC . However , it is known that the training task for GAN is hard , and the convergence property of GAN is fragile ( Salimans et al . ( 2016 ) ) . There is no substantial evidence that the gen- erated speech is perceptually good . Moreover , CVAEs alone do not guarantee distribution matching and suffers from the issue of over smoothing of the converted features . Although , there are few GAN-based systems that produced state-of-the-art results for non-parallel VC . Among these algorithms , even fewer can be applied for many-to-many VC tasks . At last , there is the only system available for zero-shot VC proposed by Qian et al . ( 2019 ) . Zero-shot conversion is a technique to convert source speaker ’ s voice into an unseen target speaker ’ s speaker via looking at a few utterances of that speaker . As known , solutions to a challenging problem comes with trade-offs . Despite the results , architectures have become more complex , which is not desirable in real-world scenarios because the quality of algorithms or architectures is also measured by the training time and computational complexity of learning trainable parameters ( Goodfellow et al . ( 2016 ) ) . Motivated by this , we propose computationally less expensive Adaptive GAN ( AdaGAN ) , a new style transfer framework , and a new architectural training procedure that we apply to the GAN-based framework . In AdaGAN , the generator encapsulates Adaptive Instance Normalization ( AdaIN ) for style transfer , and the discriminator is responsible for adversarial training . Recently , StarGAN-VC ( proposed by Kameoka et al . ( 2018 ) ) is a state-of-the-art method among all the GAN-based frameworks for non-parallel many-to-many VC . AdaGAN is also GAN-based framework . Therefore , we compare AdaGAN with StarGAN-VC for non-parallel many-to-many VC in terms of naturalness , speaker similarity , and computational complexity . We observe that AdaGAN yields state-of-the-art results for this with almost 88.6 % less computational complexity . Recently proposed AutoVC ( by Qian et al . ( 2019 ) ) is the only framework for zero-shot VC . Inspired by this , we propose AdaGAN for zero-shot VC as an independent study , which is the first GAN-based framework to perform zeroshot VC . We reported initial results for zero-shot VC using AdaGAN.The main contributions of this work are as follows : • We introduce the concept of latent representation based many-to-many VC using GAN for the first time in literature . • We show that in the latent space content of the speech can be represented as the distribution and the properties of this distribution will represent the speaking style of the speaker . • Although AdaGAN has much lesser computation complexity , AdaGAN shows much better results in terms of naturalness and speaker similarity compared to the baseline . 2 RELATED WORK . Developing a non-parallel VC framework is challenging task because of the problems associated with the training conditions using non-parallel data in deep learning architectures . However , attempts have been made to develop many non-parallel VC frameworks in the past decade . For example , Maximum Likelihood ( ML ) -based approach proposed by Ye & Young ( 2006 ) , speaker adaptation technique by Mouchtaris et al . ( 2006 ) , GMM-based VC method using Maximum a posteriori ( MAP ) adaptation technique by Lee & Wu ( 2006 ) , iterative alignment method by Erro et al . ( 2010 ) , Automatic Speech Recognition ( ASR ) -based method by Xie et al . ( 2016 ) , speaker verification-based method using i-vectors by Kinnunen et al . ( 2017 ) , and many other frameworks ( Chen et al . ( 2014 ) ; Nakashika et al . ( 2014 ) ; Blaauw & Bonada ( 2016 ) ; Hsu et al . ( 2016 ) ; Kaneko & Kameoka ( 2017 ) ; Saito et al . ( 2018a ) ; Sun et al . ( 2015 ) ; Shah et al . ( 2018b ; c ) ; Shah & Patil ( 2018 ) ; Biadsy et al . ( 2019 ) ) . Recently , a method using Conditional Variational Autoencoders ( CVAEs ) ( Kingma & Welling ( 2013 ) ) was proposed for non-parallel VC by ( Hsu et al . ( 2016 ) ; Saito et al . ( 2018a ) ) . Recently , VAE based method for VC was proposed , which also uses AdaIN to transfer the speaking style ( Chou et al . ( 2019 ) ) . One powerful framework that can potentially overcome the weakness of VAEs involves GANs . While GAN-based methods were originally applied for image translation problems , these methods have also been employed with noteworthy success for various speech technology-related applications , we can see via architectures proposed by ( Michelsanti & Tan ( 2017 ) ; Saito et al . ( 2018b ) ; Shah et al . ( 2018a ) ) , and many others . In GANs-based methods , Cycle-consistent Adversarial Network ( CycleGAN ) -VC is one of the state-of-the-art methods in the non-parallel VC task proposed by ( Kaneko & Kameoka ( 2017 ) ) . Among these non-parallel algorithms , a few can produce good results for non-parallel many-tomany VC . Recently , StarGAN-VC ( Kameoka et al . ( 2018 ) ) is a state-of-the-art method for the nonparallel many-to-many VC among all the GAN-based frameworks . Past attempts have been made to achieve conversion using style transfer algorithms ( Atalla et al . ( 2018 ) ; Chou et al . ( 2018 ) ; Qian et al . ( 2019 ) ) . The most recent framework is the AutoVC ( proposed by Qian et al . ( 2019 ) ) using style transfer scheme , the first and the only framework in VC literature which achieved state-of-the-art results in zero-shot VC . 3 APPROACH . 3.1 PROBLEM FORMULATION . The traditional VC problem is being reformulated as a style transfer problem . Here , we assume Z is a set of n speakers denoted by Z = { Z1 , Z2 , ... , Zn } , where Zi is the ith speaker , and U is the set of m speech utterances denoted by U = { U1 , U2 , ... , Um } , where Ui is the ith speech utterance . Now , probability density function ( pdf ) is generated for given Zi , and Ui denoted by pX ( .|Zi , Ui ) via the stochastic process of random sampling from the distributions Zi and Ui . Here , Xi ∼ pX ( .|Zi , Ui ) can be referred as features of given Ui with speaking style of Zi . The key idea is to transfer the speaking style of one speaker into another in order to achieve VC . For this , let us consider a set of random variables ( Z1 , U1 ) corresponding to a source speaker , and ( Z2 , U2 ) corresponding to a target speaker . Here , U1 and U2 are spoken by Z1 and Z2 , respectively . Our goal is to achieve pX̂ ( .|Z2 , U1 ) . Now , we want to learn a mapping function to achieve our goal for VC . Our mapping function is able to generate the distribution denoted by X̂Z1→Z2 with speaking style of Z2 while retaining the linguistic content of U1 . Formally , we want to generate the pdf ( i.e. , pX̂Z1→Z2 ( .|Z1 , U1 , Z2 , U2 ) ) to be close or equal to the pX̂ ( .|Z2 , U1 ) . Accurately , our mapping function will achieve this property , as shown in eq . 1. pX̂Z1→Z2 ( .|Z1 , U1 , Z2 , U2 ) = pX̂ ( .|Z2 , U1 ) . ( 1 ) Intuitively , we want to transfer the speaking style of Z2 to the Z1 while preserving the linguistic content of U1 . Therefore , converted voice is perceptually sound as if utterance U1 were spoken by Z2 . With this , AdaGAN is also designed to achieve zero-shot VC . During zero-shot conversion , U1 and U2 can be seen or unseen utterances , and Z1 and Z2 can be seen or unseen speakers . 3.2 ADAPTIVE INSTANCE NORMALIZATION ( AdaIN ) Our key idea for style transfer in VC revolves around the AdaIN . First , AdaIN was introduced for arbitrary style transfer in image-to-image translation tasks by Huang & Belongie ( 2017 ) . In this paper , AdaIN helps us to capture the speaking style and linguistic content into a single feature representation . AdaIN takes features of a source speaker ’ s speech ( i.e. , X ) and sample features of the target speaker ’ s speech ( i.e. , Y ) . Here , x is a feature from the set X related to the linguistic content of source speech , and Y is features related to the speaking style of the target speaker . AdaIN will map the mean and standard deviation of X ( i.e. , µX and σx ) in such a way that it will match with mean , and standard deviation of Y ( i.e. , µY and σY ) . Mathematical equation of AdaIN is defined as ( Huang & Belongie ( 2017 ) ) : AdaIN ( x , Y ) = σY ( x− µX σX ) + µY . ( 2 ) From eq . ( 2 ) , we can infer that AdaIN first normalizes x , and scales back based on mean and standard deviations of y . Intuitively , let ’ s assume that we have one latent space which represents the linguistic content in the distribution and also contains speaking style in terms of the mean and standard deviation of the same distribution . To transfer the speaking style , we have adopted the distribution properties ( i.e. , its mean and standard deviation ) of the target speaker . As a result , the output produced by AdaIN has the high average activation for the features which are responsible for style ( y ) while preserving linguistic content . AdaIN does not have any learning parameters . Hence , it will not affect the computational complexity of the framework .

Summary: This work describes an efficient voice conversion system that can operate on non-parallel samples and convert from and to multiple voices.  The central element of the methodology is the AdaIn modification.  This is an efficient speaker adaptive technique where features are re-normalized to a particular speaker's domain.  The rest of the machinery is well motivated and well executed, but less novel.  This addition enables the voice conversion between speakers.