DISENTANGLED REPRESENTATION LEARNING
Year: 2021

Abstract:
Voice style transfer, also called voice conversion, seeks to modify one speaker’s voice to generate speech as if it came from another (target) speaker. Previous works have made progress on voice conversion with parallel training data and pre-known speakers. However, zero-shot voice style transfer, which learns from non-parallel data and generates voices for previously unseen speakers, remains a challenging problem. We propose a novel zero-shot voice transfer method via disentangled representation learning. The proposed method first encodes speakerrelated style and voice content of each input voice into separated low-dimensional embedding spaces, and then transfers to a new voice by combining the source content embedding and target style embedding through a decoder. With informationtheoretic guidance, the style and content embedding spaces are representative and (ideally) independent of each other. On real-world VCTK datasets, our method outperforms other baselines and obtains state-of-the-art results in terms of transfer accuracy and voice naturalness for voice style transfer experiments under both many-to-many and zero-shot setups.

heading: 1 INTRODUCTION
text: Style transfer, which automatically converts a data instance into a target style, while preserving its content information, has attracted considerable attention in various machine learning domains, including computer vision (Gatys et al., 2016; Luan et al., 2017; Huang & Belongie, 2017), video processing (Huang et al., 2017; Chen et al., 2017), and natural language processing (Shen et al., 2017; Yang et al., 2018; Lample et al., 2019; Cheng et al., 2020b). In speech processing, style transfer was earlier recognized as voice conversion (VC) (Muda et al., 2010), which converts one speaker’s utterance, as if it was from another speaker but with the same semantic meaning. Voice style transfer (VST) has received long-term research interest, due to its potential for applications in security (Sisman et al., 2018), medicine (Nakamura et al., 2006), entertainment (Villavicencio & Bonada, 2010) and education (Mohammadi & Kain, 2017), among others.
Although widely investigated, VST remains challenging when applied to more general application scenarios. Most of the traditional VST methods require parallel training data, i.e., paired voices from two speakers uttering the same sentence. This constraint limits the application of such models in the real world, where data are often not pair-wise available. Among the few existing models that address non-parallel data (Hsu et al., 2016; Lee & Wu, 2006; Godoy et al., 2011), most methods cannot handle many-to-many transfer (Saito et al., 2018; Kaneko & Kameoka, 2018; Kameoka et al., 2018), which prevents them from converting multiple source voices to multiple target speaker styles. Even among the few non-parallel many-to-many transfer models, to the best of our knowledge, only two models (Qian et al., 2019; Chou & Lee, 2019) allow zero-shot transfer, i.e., conversion from/to newly-coming speakers (unseen during training) without re-training the model.
The only two zero-shot VST models (AUTOVC (Qian et al., 2019) and AdaIN-VC (Chou & Lee, 2019)) share a common weakness. Both methods construct encoder-decoder frameworks, which extract the style and the content information into style and content embeddings, and generate a voice sample by combining a style embedding and a content embedding through the decoder. With the combination of the source content embedding and the target style embedding, the models generate
∗Equal contribution.
the transferred voice, based only on source and target voice samples. AUTOVC (Qian et al., 2019) uses a GE2E (Wan et al., 2018) pre-trained style encoder to ensure rich speaker-related information in style embeddings. However, AUTOVC has no regularizer to guarantee that the content encoder does not encode any style information. AdaIN-VC (Chou & Lee, 2019) applies instance normalization (Ulyanov et al., 2016) to the feature map of content representations, which helps to eliminate the style information from content embeddings. However, AdaIN-VC fails to prevent content information from being revealed in the style embeddings. Both methods cannot assure that the style and content embeddings are disentangled without information revealed from each other.
With information-theoretic guidance, we propose a disentangled-representation-learning method to enhance the encoder-decoder zero-shot VST framework, for both style and content information preservation. We call the proposed method Information-theoretic Disentangled Embedding for Voice Conversion (IDE-VC). Our model successfully induces the style and content of voices into independent representation spaces by minimizing the mutual information between style and content embeddings. We also derive two new multi-group mutual information lower bounds, to further improve the representativeness of the latent embeddings. Experiments demonstrate that our method outperforms previous works under both many-to-many and zero-shot transfer setups on two objective metrics and two subjective metrics.

heading: 2 BACKGROUND
text: In information theory, mutual information (MI) is a crucial concept that measures the dependence between two random variables. Mathematically, the MI between two variables x and y is
I(x;y) := Ep(x,y) [ log p(x,y)
p(x)p(y)
] , (1)
where p(x) and p(y) are marginal distributions of x and y, and p(x,y) is the joint distribution. Recently, MI has attracted considerable interest in machine learning as a criterion to minimize or maximize the dependence between different parts of a model (Chen et al., 2016; Alemi et al., 2016; Hjelm et al., 2018; Veličković et al., 2018; Song et al., 2019). However, the calculation of exact MI values is challenging in practice, since the closed form of joint distribution p(x,y) in equation (1) is generally unknown. To solve this problem, several MI estimators have been proposed. For MI maximization tasks, Nguyen, Wainwright and Jordan (NWJ) (Nguyen et al., 2010) propose a lower bound by representing (1) as an f -divergence (Moon & Hero, 2014):
INWJ := Ep(x,y)[f(x,y)]− e−1Ep(x)p(y)[ef(x,y)], (2) with a score function f(x,y). Another widely-used sample-based MI lower bound is InfoNCE (Oord et al., 2018), which is derived with Noise Contrastive Estimation (NCE) (Gutmann & Hyvärinen, 2010). With sample pairs {(xi,yi)}Ni=1 drawn from the joint distribution p(x,y), the InfoNCE lower bound is defined as
INCE := E [ 1 N N∑ i=1 log ef(xi,yi) 1 N ∑N j=1 e f(xi,yj) ] . (3)
For MI minimization tasks, Cheng et al. (2020a) proposed a contrastively learned upper bound that requires the conditional distribution p(x|y):
I(x;y) ≤ E [ 1 N N∑ i=1 [ log p(xi|yi)− 1 N N∑ j=1 log p(xj |yi) ]] . (4)
where the MI is bounded by the log-ratio of conditional distribution p(x|y) between positive and negative sample pairs. In the following, we derive our information-theoretic disentangled representation learning framework for voice style transfer based on the MI estimators described above.

heading: 3 PROPOSED MODEL
text: We assume access to N audio (voice) recordings from M speakers, where speaker u has Nu voice samples Xu = {xui}Nui=1. The proposed approach encodes each voice input x ∈ X = ∪Mu=1Xu into a speaker-related (style) embedding s = Es(x) and a content-related embedding c = Ec(x),
using respectively a style encoder Es(·) and a content encoder Ec(·). To transfer a source xui from speaker u to the target style of the voice of speaker v, xvj , we combine the content embedding cui = Ec(xui) and the style embedding svj = Es(xvj) to generate the transferred voice x̂u→v,i = D(svj , cui) with a decoder D(s, c). To implement this two-step transfer process, we introduce a novel mutual information (MI)-based learning objective, that induces the style embedding s and content embedding c into independent representation spaces (i.e., ideally, s contains rich style information of xwith no content information, and vice versa). In the following, we first describe our MI-based training objective in Section 3.1, and then discuss the practical estimation of the objective in Sections 3.2 and 3.3.

heading: 3.1 MI-BASED DISENTANGLING OBJECTIVE
text: From an information-theoretic perspective, to learn representative latent embedding (s, c), it is desirable to maximize the mutual information between the embedding pair (s, c) and the input x. Meanwhile, the style embedding s and the content c are desired to be independent, so that we can control the style transfer process with different style and content attributes. Therefore, we minimize the mutual information I(s; c) to disentangle the style embedding and content embedding spaces. Consequently, our overall disentangled-representation-learning objective seeks to minimize
L = I(s; c)− I(x; s, c) = I(s; c)− I(x; c|s)− I(x; s). (5) As discussed in Locatello et al. (Locatello et al., 2019), without inductive bias for supervision, the learned representation can be meaningless. To address this problem, we use the speaker identity u as a variable with values {1, . . . ,M} to learn representative style embedding s for speaker-related attributes. Noting that the process from speaker u to his/her voice xui to the style embedding sui (as u → x → s) is a Markov Chain, we conclude I(s;x) ≥ I(s;u) based on the MI data-processing inequality (Cover & Thomas, 2012) (as stated in the Supplementary Material). Therefore, we replace I(s;x) in L with I(s;u) and minimize an upper bound instead:
L̄ = I(s; c)− I(x; c|s)− I(u; s) ≥ I(s; c)− I(x; c|s)− I(x; s), (6) In practice, calculating the MI is challenging, as we typically only have access to samples, and lack the required distributions (Chen et al., 2016). To solve this problem, below we provide several MI estimates to the objective terms I(s; c), I(x; c|s) and I(u; s).

heading: 3.2 MI LOWER BOUND ESTIMATION
text: To maximize I(u; s), we derive the following multi-group MI lower bound (Theorem 3.1) based on the NWJ bound developed in Nguyen et al. (Nguyen et al., 2010). The detailed proof is provided in the Supplementary Material. Let µ(−ui)v = µv represent the mean of all style embeddings in group Xv , constituting the style centroid of speaker v; µ(−ui)u is the mean of all style embeddings in group Xu except data point xui, representing a leave-xui-out style centroid of speaker u. Intuitively, we minimize ‖sui − µ(−ui)u ‖ to encourage the style embedding of voice xui to be more similar to the style centroid of speaker u, while maximizing ‖sui − µ(−ui)v ‖ to enlarge the margin between sui and the other speakers’ style centroids µv . We denote the right-hand side of (7) as Î1. Theorem 3.1. Let µ(−ui)v = 1Nv ∑Nv k=1 svk if u 6= v; and µ (−ui) u = 1 Nu−1 ∑ j 6=i suj . Then,
I(u; s) ≥ E [ 1 N M∑ u=1 Nu∑ i=1 [ − ‖sui − µ(−ui)u ‖2 − e−1 N M∑ v=1 Nv exp{−‖sui − µ(−ui)v ‖2} ]] . (7)
To maximize I(x; c|s), we derive a conditional mutual information lower bound below: Theorem 3.2. Assume that given s = su, samples {(xui, cui)}Nui=1 are observed. With a variational distribution qφ(x|s, c), we have I(x; c|s) ≥ E[Î], where
Î = 1 N M∑ u=1 Nu∑ i=1 [ log qφ(xui|cui, su)− log ( 1 Nu Nu∑ j=1 qφ(xuj |cui, su) )] . (8)
Based on the criterion for s in equation (7), a well-learned style encoder Es pulls all style embeddings sui from speaker u together. Suppose su is representative of the style embeddings of set Xu. If we parameterize the distribution qφ(x|s, c) ∝ exp(−‖x−D(s, c)‖2) with decoder D(s, c), then based on Theorem 3.2, we can estimate the lower bound of I(x; c|s) with the following objective:
Î2 := 1
N M∑ u=1 Nu∑ i=1 [ − ‖xui −D(cui, su)‖2 − log ( 1 Nu Nu∑ j=1 exp{−‖xuj −D(cui, su)‖2} )] .
When maximizing Î2, for speaker u with his/her given voice style su, we encourage the content embedding cui to well reconstruct the original voice xui, with small ‖xui − D(cui, su)‖. Additionally, the distance ‖xuj −D(cui, su)‖ is minimized, ensuring cui does not contain information to reconstruct other voices xuj from speaker u. With Î2, the correlation between xui and cui is amplified, which improves cui in preserving the content information.

heading: 3.3 MI UPPER BOUND ESTIMATION
text: The crucial part of our framework is disentangling the style and the content embedding spaces, which imposes (ideally) that the style embedding s excludes any content information and vice versa. Therefore, the mutual information between s and c is expected to be minimized. To estimate I(s; c), we derive a sample-based MI upper bound in Theorem 3.3 base on (4).
Theorem 3.3. If p(s|c) provides the conditional distribution between variables s and c, then
I(s; c) ≤ E [ 1 N M∑ u=1 Nu∑ i=1 [ log p(sui|cui)− 1 N M∑ v=1 Nv∑ j=1 log p(sui|cvj) ]] . (9)
The upper bound in (9) requires the ground-truth conditional distribution p(s|c), whose closed form is unknown. Therefore, we use a probabilistic neural network qθ(s|c) to approximate p(s|c) by maximizing the log-likelihood F(θ) = ∑Mu=1∑Nui=1 log qθ(sui|cui). With the learned qθ(s|c), the
objective for minimizing I(s; c) becomes:
Î3 := 1
N M∑ u=1 Nu∑ i=1 [ log qθ(sui|cui)− 1 N M∑ v=1 Nv∑ j=1 log qθ(sui|cvj) ] . (10)
When weights of encoders Ec, Es are updated, the embedding spaces s, c change, which leads to the changing of conditional distribution p(s|c). Therefore, the neural approximation qθ(s|c) must be updated again. Consequently, during training, the encoders Ec, Es and the approximation qθ(s|c) are updated iteratively. In the Supplementary Material, we further discuss that with a good approximation qθ(s|c), Î3 remains an MI upper bound.

heading: 3.4 ENCODER-DECODER FRAMEWORK
text: With the aforementioned MI estimates Î1, Î2, and Î3, the final training loss of our method is L∗ = [Î3 − Î1 − Î2]− βF(θ), (11)
where β is a positive number re-weighting the two objective terms. Term Î3−Î1−Î2 is minimized w.r.t the parameters in encoders Ec, Es and decoder D; term F(θ) as the likelihood function of qθ(s|c) is maximized w.r.t the parameter θ. In practice, the two terms are updated iteratively with gradient descent (by fixing one and updating another). The training and transfer processes of our model are shown in Figure 1. We name this MI-guided learning framework as Information-theoretic Disentangled Embedding for Voice Conversion (IDE-VC).

heading: 4 RELATED WORK
text: Many-to-many Voice Conversion Traditional voice style transfer methods mainly focus on one-toone and many-to-one conversion tasks, which can only transfer voices into one target speaking style. This constraint limits the applicability of the methods. Recently, several many-to-many voice conversion methods have been proposed, to convert voices in broader application scenarios. StarGANVC (Kameoka et al., 2018) uses StarGAN (Choi et al., 2018) to enable many-to-many transfer, in which voices are fed into a unique generator conditioned on the target speaker identity. A discriminator is also used to evaluate generation quality and transfer accuracy. Blow (Serrà et al., 2019) is a flow-based generative model (Kingma & Dhariwal, 2018), that maps voices from different speakers into the same latent space via normalizing flow (Rezende & Mohamed, 2015). The conversion is accomplished by transforming the latent representation back to the observation space with the target speaker’s identifier. Two other many-to-many conversion models, AUTOVC (Qian et al., 2019) and AdaIN-VC (Chou & Lee, 2019), extend applications into zero-shot scenarios, i.e., conversion from/to a new speaker (unseen during training), based on only a few utterances. Both AUTOVC and AdaIN-VC construct an encoder-decoder framework, which extracts the style and content of one speech sample into separate latent embeddings. Then when a new voice from an unseen speaker comes, both its style and content embeddings can be extracted directly. However, as discussed in the Introduction, both methods do not have explicit regularizers to reduce the correlation between style and content embeddings, which limits their performance.
Disentangled Representation Learning Disentangled representation learning (DRL) aims to encode data points into separate independent embedding subspaces, where different subspaces represent different data attributes. DRL methods can be classified into unsupervised and supervised approaches. Under unsupervised setups, Burgess et al. (2018), Higgins et al. (2016) and Kim & Mnih (2018) use latent embeddings to reconstruct the original data while keeping each dimension of the embeddings independent with correlation regularizers. This has been challenged by Locatello et al. (2019), in that each part of the learned embeddings may not be mapped to a meaningful data attribute. In contrast, supervised DRL methods effectively learn meaningful disentangled embedding parts by adding different supervision to different embedding components. Between the two embedding parts, the correlation is still required to be reduced to prevent the revealing of information to each other. The correlation-reducing methods mainly focus on adversarial training between embedding parts (Hjelm et al., 2018; Kim & Mnih, 2018), and mutual information minimization (Chen et al., 2018; Cheng et al., 2020b). By applying operations such as switching and combining, one can use disentangled representations to improve empirical performance on downstream tasks, e.g. conditional generation (Burgess et al., 2018), domain adaptation (Gholami et al., 2020), and few-shot learning (Higgins et al., 2017).

heading: 5 EXPERIMENTS
text: We evaluate our IDE-VC on real-world voice a dataset under both many-to-many and zero-shot VST setups. The selected dataset is CSTR Voice Cloning Toolkit (VCTK) (Yamagishi et al., 2019), which includes 46 hours of audio from 109 speakers. Each speaker reads a different sets of utterances, and the training voices are provided in a non-parallel manner. The audios are downsampled at 16kHz. In the following, we first describe the evaluation metrics and the implementation details, and then analyze our model’s performance relative to other baselines under many-to-many and zero-shot VST settings.

heading: 5.1 EVALUATION METRICS
text: Objective Metrics We consider two objective metrics: Speaker verification accuracy (Verification) and the Mel-Cepstral Distance (Distance) (Kubichek, 1993). The speaker verification accuracy measures whether the transferred voice belongs to the target speaker. For fair comparison, we used a third-party pre-trained speaker encoder Resemblyzer1 to classify the speaker identity from the transferred voices. Specifically, style centroids for speakers are learned with ground-truth voice samples. For a transferred voice, we encode it via the pre-trained speaker encoder and find the speaker with the closest style centroid as the identity prediction. For the Distance, the vanilla Mel-Cepstral Distance (MCD) cannot handle the time alignment issue described in Section 2. To make reasonable comparisons between the generation and ground truth, we apply the Dynamic Time Warping (DTW) algorithm (Berndt & Clifford, 1994) to automatically align the time-evolving sequences before calculating MCD. This DTW-MCD distance measures the similarity of the transferred voice and the real voice from the target speaker. Since the calculation of DTW-MCD requires parallel data, we select voices with the same content from the VCTK dataset as testing pairs. Then we transfer one voice in the pair and calculate DTW-MCD with the other voice as reference.
Subjective Metrics Following Wester et al. (Wester et al., 2016), we use the naturalness of the speech (Naturalness), and the similarity of the transferred speech to target identity (Similarity) as subjective metrics. For Naturalness, annotators are asked to rate the score from 1-5 for each transferred speech.For the Similarity, the annotators are presented with two audios (the converted speech and the corresponding reference), and are asked to rate the score from 1 to 4. For both scores, the higher the better. Following the setting in Blow (Serrà et al., 2019), we report Similarity defined as a total percentage from the binary rating. The evaluation of both subjective metrics is conducted on Amazon Mechanical Turk (MTurk)2. More details about evaluation metrics are provided in the Supplementary Material.

heading: 5.2 IMPLEMENTATION DETAILS
text: Following AUTOVC (Qian et al., 2019), our model inputs are represented via mel-spectrogram. The number of mel-frequency bins is set as 80. When voices are generated, we adopt the WaveNet vocoder (Oord et al., 2016) pre-trained on the VCTK corpus to invert the spectrogram signal back to a waveform. The spectrogram is first upsampled with deconvolutional layers to match the sampling rate, and then a standard 40-layer WaveNet is applied to generate speech waveforms. Our model is implememted with Pytorch and takes 1 GPU day on an Nvidia Xp to train.
Encoder Architecture The speaker encoder consists of a 2-layer long short-term memory (LSTM) with cell size of 768, and a fully-connected layer with output dimension 256. The speaker encoder is initialized with weights from a pretrained GE2E (Wan et al., 2018) encoder. The input of the content encoder is the concatenation of the mel-spectrogram signal and the corresponding speaker embedding. The content encoder consists of three convolutional layers with 512 channels, and two layers of a bidirectional LSTM with cell dimension 32. Following the setup in AUTOVC (Qian et al., 2019), the forward and backward outputs of the bi-directional LSTM are downsampled by 16.
Decoder Architecture Following AUTOVC (Qian et al., 2019), the initial decoder consists of a three-layer convolutional neural network (CNN) with 512 channels, three LSTM layers with cell dimension 1024, and another convolutional layer to project the output of the LSTM to dimension of 80. To enhance the quality of the spectrogram, following AUTOVC (Qian et al., 2019), we use a post-network consisting of five convolutional layers with 512 channels for the first four layers, and
1https://github.com/resemble-ai/Resemblyzer 2https://www.mturk.com/
80 channels for the last layer. The output of the post-network can be viewed as a residual signal. The final conversion signal is computed by directly adding the output of the initial decoder and the post-network. The reconstruction loss is applied to both the output of the initial decoder and the final conversion signal.
Approximation Network Architecture As described in Section 3.3, minimizing the mutual information between style and content embeddings requires an auxiliary variational approximation qθ(s|c). For implementation, we parameterize the variational distribution in the Gaussian distribution family qθ(s|c) = N (µθ(c),σ2θ(c) · I), where mean µθ(·) and variance σ2θ(·) are two-layer fully-connected networks with tanh(·) as the activation function. With the Gaussian parameterization, the likelihoods in objective Î3 can be calculated in closed form.

heading: 5.3 STYLE TRANSFER PERFORMANCE
text: For the many-to-many VST task, we randomly select 10% of the sentences for validation and 10% of the sentences for testing from the VCTK dataset, following the setting in Blow (Serrà et al., 2019). The rest of the data are used for training in a non-parallel scheme. For evaluation, we select voice pairs from the testing set, in which each pair of voices have the same content but come from different speakers. In each testing pair, we conduct transfer from one voice to the other voice’s speaking style, and then we compare the transferred voice and the other voice as evaluating the model performance. We test our model with four competitive baselines: Blow (Serrà et al., 2019)3, AUTOVC (Qian et al., 2019), AdaIN-VC (Chou & Lee, 2019) and StarGAN-VC (Kameoka et al., 2018). The detailed implementation of these four methods are provided in the Supplementary Material. Table 1 shows the subjective and objective evaluation for the many-to-many VST task. Both methods with the encoder-decoder framework, AdaIN-VC and AUTOVC, have competitive results. However, our IDE-VC outperforms the other baselines on all metrics, demonstrating that the style-content disentanglement in the latent space improves the performance of the encoder-decoder framework.
For the zero-shot VST task, we use the same train-validation dataset split as in the many-to-many setup. The testing data are selected to guarantee that no test speaker has any utterance in the training set. We compare our model with the only two baselines, AUTOVC (Qian et al., 2019) and AdaINVC (Chou & Lee, 2019), that are able to handle voice transfer for newly-coming unseen speakers. We used the same implementations of AUTOVC and AdaIN-VC as in the many-to-many VST. The evaluation results of zero-shot VST are shown in Table 2, among the two baselines AdaIN-VC performs better than AUTOVC overall.Our IDE-VC outperforms both baseline methods, on all metrics. All three tested models have encoder-decoder transfer frameworks, the superior performance
3For Blow model, we use the official implementation available on Github (https://github.com/joansj/blow). We report the best result we can obtain here, under training for 100 epochs (11.75 GPU days on Nvidia V100).
of IDE-VC indicates the effectiveness of our disentangled representation learning scheme. More evaluation details are provided in the supplementary material.

heading: 5.4 DISENTANGLEMENT DISCUSSION
text: Besides the performance comparison with other VST baselines, we demonstrate the capability of our information-theoretic disentangled representation learning scheme. First, we conduct a tSNE (Maaten & Hinton, 2008) visualization of the latent spaces of the IDE-VC model. As shown in the left of Figure 2, style embeddings from the same speaker are well clustered, and style embeddings from different speakers separate in a clean manner. The clear pattern indicates our style encoder Es can verify the speakers’ identity from the voice samples. In contrast, the content embeddings (in the right of Figure 2) are indistinguishable for different speakers, which means our content encoder Ec successfully eliminates speaker-related information and extracts rich semantic content from the data.
We also empirically evaluate the disentanglement, by predicting the speakers’ identity based on only the content embeddings. A two-layer fully-connected network is trained on the testing set with a content embedding as input, and the corresponding speaker identity as output. We compare our IDE-VC with AUTOVC and AdaIN-VC, which also output content embeddings. The classification results are shown in Table 3. Our IDE-VC reaches the lowest classification accuracy, indicating that the content embeddings learned by IDE-VC contains the least speaker-related information. Therefore, our IDE-VC learns disentangled representations with high quality compared with other baselines.
5.5 ABLATION STUDY
Moreover, we have considered an ablation study that addresses performance effects from different learning losses in (11), with results shown in Table 4. We compare our model with two models trained by part of the loss function in (11), while keeping the other training setups unchanged, including the model structure. From the results, when the model is trained without the style encoder loss term Î1, a transferred voice still is generated, but with a large distance to the ground truth. The verification accuracy also significantly decreases with no speaker-related
information utilized. When the disentangling term Î3 is removed, the model still reaches competitive performance, because the style encoder Es and decoder D are well trained by Î1 and Î2. However, when adding term Î3, we disentangle the style and content spaces, and improve the transfer quality with higher verification accuracy and less distortion. The performance without term Î2 is not reported, because the model cannot even generate fluent speech without the reconstruction loss.

heading: 6 CONCLUSIONS
text: We have improved the encoder-decoder voice style transfer framework by disentangled latent representation learning. To effectively induce the style and content information of speech into independent embedding latent spaces, we minimize a sample-based mutual information upper bound between style and content embeddings. The disentanglement of the two embedding spaces ensures the voice transfer accuracy without information revealed from each other. We have also derived two new multi-group mutual information lower bounds, which are maximized during training to enhance the representativeness of the latent embeddings. On the real-world VCTK dataset, our model outperforms previous works under both many-to-many and zero-shot voice style transfer setups. Our model can be naturally extended to other style transfer tasks modeling time-evolving sequences, e.g., video and music style transfer. Moreover, our general multi-group mutual information lower bounds have broader potential applications in other representation learning tasks.

heading: ACKNOWLEDGEMENTS
text: This research was supported in part by the DOE, NSF and ONR.

Review summaries:
This paper proposes a zero-shot voice style transfer (VST) algorithms that explicitly controls the disentanglement between content information and style information. Experiments show that the proposed algorithm can achieve significant improvement over the existing state-of-the-art VST algorithms. There are two major strengths of this paper. First, it motivates the algorithm design from an information-theoretic perspective. Second, the performance improvement is significant.

This submission proposes a training approach for voice style transfer using encoder-decoder framework and content and style representations. The approach combines multiple mutual-information (MI) based terms into a single objective function. One of the MI based terms is the MI between content and style representations. By minimising mutual information between these representations, the training approach yields models where these representations are disentangled. Experimental results show that this approach leads to improved performance in speaker verification and speech similarity tasks. Experimental results in challenging zero-shot conditions also demonstrate improved performance in speaker verification, speech naturalness and speech similarity tasks.