Improving Zero-Shot Voice Style Transfer via Disentangled Representation Learning

1 INTRODUCTION . Style transfer , which automatically converts a data instance into a target style , while preserving its content information , has attracted considerable attention in various machine learning domains , including computer vision ( Gatys et al. , 2016 ; Luan et al. , 2017 ; Huang & Belongie , 2017 ) , video processing ( Huang et al. , 2017 ; Chen et al. , 2017 ) , and natural language processing ( Shen et al. , 2017 ; Yang et al. , 2018 ; Lample et al. , 2019 ; Cheng et al. , 2020b ) . In speech processing , style transfer was earlier recognized as voice conversion ( VC ) ( Muda et al. , 2010 ) , which converts one speaker ’ s utterance , as if it was from another speaker but with the same semantic meaning . Voice style transfer ( VST ) has received long-term research interest , due to its potential for applications in security ( Sisman et al. , 2018 ) , medicine ( Nakamura et al. , 2006 ) , entertainment ( Villavicencio & Bonada , 2010 ) and education ( Mohammadi & Kain , 2017 ) , among others . Although widely investigated , VST remains challenging when applied to more general application scenarios . Most of the traditional VST methods require parallel training data , i.e. , paired voices from two speakers uttering the same sentence . This constraint limits the application of such models in the real world , where data are often not pair-wise available . Among the few existing models that address non-parallel data ( Hsu et al. , 2016 ; Lee & Wu , 2006 ; Godoy et al. , 2011 ) , most methods can not handle many-to-many transfer ( Saito et al. , 2018 ; Kaneko & Kameoka , 2018 ; Kameoka et al. , 2018 ) , which prevents them from converting multiple source voices to multiple target speaker styles . Even among the few non-parallel many-to-many transfer models , to the best of our knowledge , only two models ( Qian et al. , 2019 ; Chou & Lee , 2019 ) allow zero-shot transfer , i.e. , conversion from/to newly-coming speakers ( unseen during training ) without re-training the model . The only two zero-shot VST models ( AUTOVC ( Qian et al. , 2019 ) and AdaIN-VC ( Chou & Lee , 2019 ) ) share a common weakness . Both methods construct encoder-decoder frameworks , which extract the style and the content information into style and content embeddings , and generate a voice sample by combining a style embedding and a content embedding through the decoder . With the combination of the source content embedding and the target style embedding , the models generate ∗Equal contribution . the transferred voice , based only on source and target voice samples . AUTOVC ( Qian et al. , 2019 ) uses a GE2E ( Wan et al. , 2018 ) pre-trained style encoder to ensure rich speaker-related information in style embeddings . However , AUTOVC has no regularizer to guarantee that the content encoder does not encode any style information . AdaIN-VC ( Chou & Lee , 2019 ) applies instance normalization ( Ulyanov et al. , 2016 ) to the feature map of content representations , which helps to eliminate the style information from content embeddings . However , AdaIN-VC fails to prevent content information from being revealed in the style embeddings . Both methods can not assure that the style and content embeddings are disentangled without information revealed from each other . With information-theoretic guidance , we propose a disentangled-representation-learning method to enhance the encoder-decoder zero-shot VST framework , for both style and content information preservation . We call the proposed method Information-theoretic Disentangled Embedding for Voice Conversion ( IDE-VC ) . Our model successfully induces the style and content of voices into independent representation spaces by minimizing the mutual information between style and content embeddings . We also derive two new multi-group mutual information lower bounds , to further improve the representativeness of the latent embeddings . Experiments demonstrate that our method outperforms previous works under both many-to-many and zero-shot transfer setups on two objective metrics and two subjective metrics . 2 BACKGROUND . In information theory , mutual information ( MI ) is a crucial concept that measures the dependence between two random variables . Mathematically , the MI between two variables x and y is I ( x ; y ) : = Ep ( x , y ) [ log p ( x , y ) p ( x ) p ( y ) ] , ( 1 ) where p ( x ) and p ( y ) are marginal distributions of x and y , and p ( x , y ) is the joint distribution . Recently , MI has attracted considerable interest in machine learning as a criterion to minimize or maximize the dependence between different parts of a model ( Chen et al. , 2016 ; Alemi et al. , 2016 ; Hjelm et al. , 2018 ; Veličković et al. , 2018 ; Song et al. , 2019 ) . However , the calculation of exact MI values is challenging in practice , since the closed form of joint distribution p ( x , y ) in equation ( 1 ) is generally unknown . To solve this problem , several MI estimators have been proposed . For MI maximization tasks , Nguyen , Wainwright and Jordan ( NWJ ) ( Nguyen et al. , 2010 ) propose a lower bound by representing ( 1 ) as an f -divergence ( Moon & Hero , 2014 ) : INWJ : = Ep ( x , y ) [ f ( x , y ) ] − e−1Ep ( x ) p ( y ) [ ef ( x , y ) ] , ( 2 ) with a score function f ( x , y ) . Another widely-used sample-based MI lower bound is InfoNCE ( Oord et al. , 2018 ) , which is derived with Noise Contrastive Estimation ( NCE ) ( Gutmann & Hyvärinen , 2010 ) . With sample pairs { ( xi , yi ) } Ni=1 drawn from the joint distribution p ( x , y ) , the InfoNCE lower bound is defined as INCE : = E [ 1 N N∑ i=1 log ef ( xi , yi ) 1 N ∑N j=1 e f ( xi , yj ) ] . ( 3 ) For MI minimization tasks , Cheng et al . ( 2020a ) proposed a contrastively learned upper bound that requires the conditional distribution p ( x|y ) : I ( x ; y ) ≤ E [ 1 N N∑ i=1 [ log p ( xi|yi ) − 1 N N∑ j=1 log p ( xj |yi ) ] ] . ( 4 ) where the MI is bounded by the log-ratio of conditional distribution p ( x|y ) between positive and negative sample pairs . In the following , we derive our information-theoretic disentangled representation learning framework for voice style transfer based on the MI estimators described above . 3 PROPOSED MODEL . We assume access to N audio ( voice ) recordings from M speakers , where speaker u has Nu voice samples Xu = { xui } Nui=1 . The proposed approach encodes each voice input x ∈ X = ∪Mu=1Xu into a speaker-related ( style ) embedding s = Es ( x ) and a content-related embedding c = Ec ( x ) , using respectively a style encoder Es ( · ) and a content encoder Ec ( · ) . To transfer a source xui from speaker u to the target style of the voice of speaker v , xvj , we combine the content embedding cui = Ec ( xui ) and the style embedding svj = Es ( xvj ) to generate the transferred voice x̂u→v , i = D ( svj , cui ) with a decoder D ( s , c ) . To implement this two-step transfer process , we introduce a novel mutual information ( MI ) -based learning objective , that induces the style embedding s and content embedding c into independent representation spaces ( i.e. , ideally , s contains rich style information of xwith no content information , and vice versa ) . In the following , we first describe our MI-based training objective in Section 3.1 , and then discuss the practical estimation of the objective in Sections 3.2 and 3.3 . 3.1 MI-BASED DISENTANGLING OBJECTIVE . From an information-theoretic perspective , to learn representative latent embedding ( s , c ) , it is desirable to maximize the mutual information between the embedding pair ( s , c ) and the input x . Meanwhile , the style embedding s and the content c are desired to be independent , so that we can control the style transfer process with different style and content attributes . Therefore , we minimize the mutual information I ( s ; c ) to disentangle the style embedding and content embedding spaces . Consequently , our overall disentangled-representation-learning objective seeks to minimize L = I ( s ; c ) − I ( x ; s , c ) = I ( s ; c ) − I ( x ; c|s ) − I ( x ; s ) . ( 5 ) As discussed in Locatello et al . ( Locatello et al. , 2019 ) , without inductive bias for supervision , the learned representation can be meaningless . To address this problem , we use the speaker identity u as a variable with values { 1 , . . . , M } to learn representative style embedding s for speaker-related attributes . Noting that the process from speaker u to his/her voice xui to the style embedding sui ( as u → x → s ) is a Markov Chain , we conclude I ( s ; x ) ≥ I ( s ; u ) based on the MI data-processing inequality ( Cover & Thomas , 2012 ) ( as stated in the Supplementary Material ) . Therefore , we replace I ( s ; x ) in L with I ( s ; u ) and minimize an upper bound instead : L̄ = I ( s ; c ) − I ( x ; c|s ) − I ( u ; s ) ≥ I ( s ; c ) − I ( x ; c|s ) − I ( x ; s ) , ( 6 ) In practice , calculating the MI is challenging , as we typically only have access to samples , and lack the required distributions ( Chen et al. , 2016 ) . To solve this problem , below we provide several MI estimates to the objective terms I ( s ; c ) , I ( x ; c|s ) and I ( u ; s ) . 3.2 MI LOWER BOUND ESTIMATION . To maximize I ( u ; s ) , we derive the following multi-group MI lower bound ( Theorem 3.1 ) based on the NWJ bound developed in Nguyen et al . ( Nguyen et al. , 2010 ) . The detailed proof is provided in the Supplementary Material . Let µ ( −ui ) v = µv represent the mean of all style embeddings in group Xv , constituting the style centroid of speaker v ; µ ( −ui ) u is the mean of all style embeddings in group Xu except data point xui , representing a leave-xui-out style centroid of speaker u . Intuitively , we minimize ‖sui − µ ( −ui ) u ‖ to encourage the style embedding of voice xui to be more similar to the style centroid of speaker u , while maximizing ‖sui − µ ( −ui ) v ‖ to enlarge the margin between sui and the other speakers ’ style centroids µv . We denote the right-hand side of ( 7 ) as Î1 . Theorem 3.1 . Let µ ( −ui ) v = 1Nv ∑Nv k=1 svk if u 6= v ; and µ ( −ui ) u = 1 Nu−1 ∑ j 6=i suj . Then , I ( u ; s ) ≥ E [ 1 N M∑ u=1 Nu∑ i=1 [ − ‖sui − µ ( −ui ) u ‖2 − e−1 N M∑ v=1 Nv exp { −‖sui − µ ( −ui ) v ‖2 } ] ] . ( 7 ) To maximize I ( x ; c|s ) , we derive a conditional mutual information lower bound below : Theorem 3.2 . Assume that given s = su , samples { ( xui , cui ) } Nui=1 are observed . With a variational distribution qφ ( x|s , c ) , we have I ( x ; c|s ) ≥ E [ Î ] , where Î = 1 N M∑ u=1 Nu∑ i=1 [ log qφ ( xui|cui , su ) − log ( 1 Nu Nu∑ j=1 qφ ( xuj |cui , su ) ) ] . ( 8 ) Based on the criterion for s in equation ( 7 ) , a well-learned style encoder Es pulls all style embeddings sui from speaker u together . Suppose su is representative of the style embeddings of set Xu . If we parameterize the distribution qφ ( x|s , c ) ∝ exp ( −‖x−D ( s , c ) ‖2 ) with decoder D ( s , c ) , then based on Theorem 3.2 , we can estimate the lower bound of I ( x ; c|s ) with the following objective : Î2 : = 1 N M∑ u=1 Nu∑ i=1 [ − ‖xui −D ( cui , su ) ‖2 − log ( 1 Nu Nu∑ j=1 exp { −‖xuj −D ( cui , su ) ‖2 } ) ] . When maximizing Î2 , for speaker u with his/her given voice style su , we encourage the content embedding cui to well reconstruct the original voice xui , with small ‖xui − D ( cui , su ) ‖ . Additionally , the distance ‖xuj −D ( cui , su ) ‖ is minimized , ensuring cui does not contain information to reconstruct other voices xuj from speaker u . With Î2 , the correlation between xui and cui is amplified , which improves cui in preserving the content information .

Summary: This submission proposes a training approach for voice style transfer using encoder-decoder framework and content and style representations. The approach combines multiple mutual-information (MI) based terms into a single objective function. One of the MI based terms is the MI between content and style representations. By minimising mutual information between these representations, the training approach yields models where these representations are disentangled. Experimental results show that this approach leads to improved performance in speaker verification and speech similarity tasks. Experimental results in challenging zero-shot conditions also demonstrate improved performance in speaker verification, speech naturalness and speech similarity tasks.