Improving Zero-Shot Voice Style Transfer via Disentangled Representation Learning
Authors: Siyang Yuan
Pengyu Cheng
Ruiyi Zhang
Weituo Hao
Zhe Gan
Lawrence Carin
Venue: ICLR 2021 Poster (2021)
Keywords: Style Transfer
Mutual Information
Zero-shot Learning
Disentanglement

Abstract: Voice style transfer, also called voice conversion, seeks to modify one speaker's voice to generate speech as if it came from another (target) speaker. Previous works have made progress on voice conversion with parallel training data and pre-known speakers. However, zero-shot voice style transfer, which learns from non-parallel data and generates voices for previously unseen speakers, remains a challenging problem. In this paper we propose a novel zero-shot voice transfer method via disentangled representation learning. The proposed method first encodes speaker-related style and voice content of each input voice into separate low-dimensional embedding spaces, and then transfers to a new voice by combining the source content embedding and target style embedding through a decoder. With information-theoretic guidance, the style and content embedding spaces are representative and (ideally) independent of each other. On real-world datasets, our method outperforms other baselines and obtains state-of-the-art results in terms of transfer accuracy and voice naturalness.

Published as a conference paper at ICLR 2021
IMPROVING ZERO-SHOT VOICE STYLE TRANSFER VIA DISENTANGLED REPRESENTATION LEARNING
Siyang Yuan' Munay' Cheng': Ruiyi Zhang', Weituo Hao', Zhe Gan" and Lawrence Carin'
'Duke University, Durham, North Carolina, USA 2 Microsoff, Redmond, Washington, USA {siyang, yuan, pengyu. chenggeduke..c.. edu
ABSTRACT
Voice style transfer, also called voice conversion, seeks to modify one speaker's voice to generate speech as if it came from another (target) speaker. Previous works have made progress on voice conversion with parallel training data and pre-known speakers. However, zero-shot voice style transfer, which learns from non-parallel data and generates voices for previously unseen speakers, remains challenging problem. We propose novel zero-shot voice transfer method via disentangled representation learning. The proposed method first encodes speaker- related style and voice content of each input voice into separated oow-dimensional embedding spaces, and then transfers to a new voice by combining the source con- embedding and target style embedding through a decoder. With information- theoretic guidance, the style and content embedding spaces are representative and (ideally) independent of each other. On real-world VCTK datasets, our method outperforms other baselines and obtains state-of-the-art results in terms of trans- fer accuracy and voice naturalness for voice style transfer experiments under both many-to-many and zero-shot setups.
INTRODUCTION
Style transfer, which automatically converts data instance into a target style, while preserving its content information, has attracted considerable attention in various machine learning domains, including computer vision (aatss et al. 2016 Luan et al., 2017 Huang & Belongie (017)). video processing (Huang al. 20e7n Chen et al. (017). and natural language processing (Shen et al.) 2017 Yang et al. 2018 Lample et al. 2019 Cheng et al.] (020b). In speech processing, style transfer was earlier recognized as voice conversion (VC) (Muda al. 2010), which converts one speaker's utterance, as if it was from another speaker but with the same semantic meaning, Voice style transfer (VST) has received long-term research interest, due to its potential for applications in security (Sisman et al.) 2018), medicine (Nakamura et al.) 2006), entertainment (Villavicencio & Bonada 2010) and education (Mohammadi Kain| 2017). among others.
Although widely investigated, VST remains challenging when applied to more general application scenarios. Most of the traditional VST methods require parallel training data, iee. paired voices from two speakers uttering the same sentence. This constraint limits the application of such models in the real world, where data are often not pair-wise available. Among the few existing models that address non- ponraallllll data (Hsu et al. 2016 Lee & Wu |206 |206] Godoy et al., (001)) most methods cannot handle many-to-many transfer (Saito et al.) 2218;] Kaneko & Kameoka 2018 Kameoka et al. 2018). which prevents them from converting multiple source voices multiple target speaker styles. Even among the few non-paree non-parallel many- to
taany--t-manyy many transfer models, to the best of our knowledge, only models (Qian al. 2019 Chou & Leel 2019) allow zero-shot transfer, i.e., conversion from/to newly-coming speakers (unseen during training) without re-training the model.
The only zero-shot VST models (AUTOVC (Qian et al. 2019) and AdaIN-VC (Chou Lee 2019)) share a common weakness. Both methods construct encoder-decoder frameworks, which ex- tract the style and the content information into style and content embeddings. and generate voice sample by combining style embedding and a content embedding through the decoder. With the combination of the source content embedding and the target style embedding, the models generate
"qqual contribution.

---

Published as a conference paper at ICLR 2021
the transferred voice. based only on source and target voice samples. AUTOVC QQian et al. 2019) uses a GE2E (Wan et al. 2018) pre-trained style encoder to ensure rich ppeaker-relatee information in style embeddings. However, AUTOVC has regularizer to guarantee that the content encoder does not encode any style information. AdaIN-VC (Chou & Lee 2019) applies instance normaliza- tion (Ulannov al.] (0T6) the feature map of content representations, which helps eliminate the style information from content embeddings, However, AdaIN-VC fails to prevent content infor- mation from being revealed in the style embeddings. Both methods cannot assure that the style and content embeddings are disentangled without information revealed from each other.
With ceemmation-thhoreeee guidance, we propose a to enhance the encoder-decoder zero-shot VST framework, for both style and content informa- tion preservation. We call the proposed method Information-theoreiic Disentangled Embedding for Voice Conversion (IDE-VC). Our model successfully induces the style and content of voices into independent representation spaces by minimizing the mutual information between style and content embeddings. We also derive two new multi-group mutual information lower bounds, to further improve the representativeness of the latent embeddings. Experiments demonstrate that our method outperforms previous works under both many-to to-mmny and zero-shot transfer setups two objective metrics and two subjective metrics.
method
2 BACKGROUND
In information theory, mutual information (MI) is a crucial concept that measures the dependence between two random variables. Mathematically, the MI between two variables a and y is
I(x; (p)))))))) log p(a, y) p())p(()
(1)
where p(x:) and ply) are marginal distributions of and y. and p(æ, y) is the joint distribution.
Recently, MI has attracted considerable interest in machine learning as a criterion to minimize or
maximize the dependence between different parts of model (Chen al. 2016 Alemi al. [0166]
Hjelm et al.) 2018 Veličković et al.) 2018 Song et al. 2019). However, the calculation of exact MI values challenging practice, since the closed form of joint distribution p(z, y) in equation (T)
is generally unknown. To solve this problem, several MI estimators have been proposed. For MI maximization tasks. Nguyen, Wainwright and Jordan (NWJ) (Nguyen al. 2010) propose lower
bound by representing ① as an f-divergence (Moon & Hero 2014):
INwJ Ep( (f v)] c (e(())()))ll
with function f(x, y). Another widely-used sample-based MI lower bound is In- foNCE (Oord et al. 2018). which is derived with Noise Contrastive Estimation (NCE) (Gutmann & Hyvärinen| 2010). With sample pairs {(x.., y: drawn from the joint distribution p(æ, y), the
InfoNCE lower bound is defined as
ef (an.8e))
INCE E
log
(m..)))))
For MI minimization tasks, Cheng et al.
(200aa)
proposed contrastively learned upper bound that
requires the conditional distribution p(zyy)
N
I(
E
logp(z.
1
Σ
logp(æ 1
(4)
N
N
where the MI is bounded by the log-ratio of conditional distribution p(x between positive and negative sample pairs. In the following, we derive our information-theorees disentangled represen- tation learning framework for voice style transfer based on the MI estimators described above.
PROPOSED MODEL
We assume access to N audio (voice) recordings from M speakers, where speaker u has N,, voice samples X' = { The proposed approach encodes each voice input x E = LM ,,,,,, into speaker-related (style) embedding Faz(::) and content-related embedding F.(2r),
2

---

Published as a conference paper at ICLR 2021
using respectively a style encoder E.).) and content encoder E-(-). To transfer source Dui from speaker u to the target style of the voice of speaker v, Lej. we combine the content embed- ding Cui (x.)) and the style embedding Suj (Z,,)) to generate the transferred voice Lu-vn,i = (sj Cui) with a decoder ((, c). To implement this two-step transfer process, we in- troduce novel mutual information (MI)-based learning objective, that induces the style embedding and content embedding into independent representation spaces (i.e. ideally, contains rich style information of æ with content information, and vice versa). In the following, first describe our MI-based training objective in Section 3.1 and then discuss the practical estimation of the objective in Sections 3.2 and 3.3
3.1 MI- BASBA DISENTANGLING OBJECTIVE
From an information-theomm sirable maximize the mutual information between the embedding pair (s,c) and the input æ. Meanwhile, the style embedding and the content с are desired to be independent, so that we can control the style transfer process with different style and content attributes. Therefore, we minimize the mutual information Z(s; c) to disentangle the style embedding and content embedding spaces. Consequently, our overall
perspective, to learn representative latent embedding (s, c), it is de-
objective seeks to minimize
L = I(:; c) I(x; c) = I(s; - I(x;; c/s) I(x:8).
(5)
As discussed in Locatello et al. (Locatello et al. 2019), without inductive bias for supervision, the learned representation can be meaningless. To address this problem, we use the speaker identity u as variable with values {1..... 3 } to learn representative style embedding for speaker-related attributes. Noting that the process from speaker u to his/her voice zui to the style embedding 8.s (as
+ x 8) is Markov Chain, we conclude Z(s; a) > I(s; u) based on the MI data-processing inequality (Cover & Thomas 2012) (as stated in the Supplementary Material). Therefore, we replace I(s: x)
x( in 2 with ((; u) and minimize upper bound instead:
I(ssc) I(x; cls) I(u; s) 2 I(s; c) I(x: - I(x;
(6)
In practice, calculating the MI is challenging, as we typically only have access to samples, and lack the required distributions (Chen et al., 2016). To solve this problem, below we provide several MI estimates to the objective terms [(s; c). I(x; x c|s) and I(u;
3.2 MI LOWER BOUND ESTIMATION
To maximize [(u; s), we derive the following multi-gre group MI lower bound (Theorem 3.1 based on the NWJ bound developed in Nguyen et al. (Nguyen et al. 2010). The detailed proof is provided in the Supplementary Material. Let μ (-wi) n. represent the mean of all style embeddings in group Ny,, constituting the style centroid of speaker v: ph (-"")) is the mean of all style embeddings in group X. except data point Tui, representing leave-a., -out style centroid of speaker u. Intuitively, we minimize | p.m [- || to encourage the style embedding of voice Jui be more similar to the style centroid of speaker u, while maximizing |s... n² ="O to enlarge the margin between Sai and the other speakers' style centroids pap We denote the right-hand side of (7) as ΣT-
Theorem 3.1. Let p (-w)) 1 Exi 80k if # e; and μi (-v)
N. Ej Suj. Then, Σ N. exp(-11llsu
N,
M
Ilu; 5)
E
1
||sui
w [21]] ( (7)
N
To maximize I(x: cls), we derive a conditional mutual information lower bound below:
Theorem 3.2. Assume that given s.. samples {w cui) are observed. With variational distribution qo (xx sesss,o c), we have I c|s) B[i}, where
No.
No go ( Cuis sw))
i 1
ΣΣ
log (2 Cuss su) log
Σ
(8)
N
3

---

Published as a conference paper at ICLR 2021
Tek
Put
(-)
((:;;)
D
D ((.5
8
Cui
8u
Cuu
[(
(.)
E.(-)
E.(.)
E
F
Fui
Fr
Fn
Lus
(a)
(c)
Figure 1: Training and transfer processes. (a) Training style encoder E, with objective Σ,: All voice samples are encoded into style embedding space. For style embedding sai of Tui. we min- imize its distance with speaker u's style centroid ptu and maximize its distance other speaker style centroids pv- (b) Training for content encoder E.e and decoder 1 objectives Σz Iz: We encode content Cus from voice Eui from speaker u. The style of speaker is encoded from another speaker u's voice Tuj. The dependency of style and content embedding is minimized with Σz. With Cus and Su. the decoder reconstructs the voice Rui as &us (8 Cui). Then Σn calculated based on the original voice Cui and the reconstruction Cui- (c) Transfer process: for zero-shot voice style transfer, with Fui from speaker and Zuj from speaker v. we encode content Cus and style Su, and combine them together to generate transferred voice En- = D(s,,, Cui).
Based on the criterion for s in equation 7 a well-learned style encoder Eg pulls all style embed- dings Sui from speaker together. Suppose su is representative of the style embeddings of set X we parameterize the distribution (x) 8, e) ε exp(-I|xc D(s, e)ll?) with decoder D(s, c). then based on Theorem 3.2 estimate the lower bound of I(ær; c|s) with the following objective:
N
N, Σ ,=1
2,
|| D(Cui, 8u)l12 log
exp(-Ilaa D(Cuis 8, (1)))
N₂
When maximizing Σz, for speaker u with his/her given voice style Su, we encourage the content embedding Cui to well reconstruct the original voice Tui, with small I D(cus s,,)l. Addi- tionally, the distance |バıي - D( s,))l is minimized, ensuring Cus does not contain information to reconstruct other voices Tuy from speaker u. With Ż,, the correlation between Tui and Cui is amplified, which improves Cui in preserving the content information.
3.3
MI UPPER BOUND ESTIMATION
The crucial part of our framework is disentangling the style and the content embedding spaces, which imposes (ideally) that the style embedding s excludes any content information and vice versa. Therefore, the mutual information between s and is expected to be minimized. To estimate I(s; c), we derive sample-based MI upper bound in Theorem 3.3 base on
Theorem 3.3. !f p(s) c) provides the conditional distribution between variables s and e, then
M
N
M
1
log Dogp(sui (asd)cCo))) c.
C( < R
log.((..ccaa
The upper bound in requires the ground conditional distribution p(s(c), whose closed form is unknown. Therefore, we use a probabilistic neural network (s(s(c) to approximate p(s|c) by maximizing the log-likelihood F(O) >M ENn log (s With the learned (o(s)c), the
4

---

Published as a conference paper at ICLR 2021
objective for minimizing [[s; c) becomes:
1 M
M No
Is
log
1 N
log qo (susl (css)
(10)
u=1
When weights of encoders Ec, E, are updated, the embedding spaces 8, change. which leads to the changing of conditional distribution p(s[c). Therefore, the neural approximation qe (s|c) must be updated again. Consequently, during training. the encoders Ec, Eg and the approximation go (s(c)) are updated iteratively. In the Supplementary Material, we further discuss that with a good approximation qo (s|c), I3 remains an MI upper bound.
3.4 RECODER-DECODEE FRAMEWORK
With the aforementioned MI estimates Σ,, I2, and Ig, the final training loss of our method is
L' -I - 1g] - 3F(().
(11)
where 33 is positive number re-weighting the two objective terms. Term - Σz w.r. the parameters in encoders Ec, F.g and decoder D; term F(0) as the likelihood function of qa (a(s(c () is maximized w.r.t the parameter 0. In practice, the two terms are updated iteratively with gradient descent (by fixing one and updating another). The training and transfer processes of our model are shown in Figure I] We name this MI-guided learning framework as information-theorer Disentangled Embedding for Voice Conversion (IDE-VC).
- is minimized
4 RELATED WORK
Many-to-many Voice Conversion Traditional voice style transfer methods mainly focus on one-to- one and many-to-on conversion tasks, which can only transfer voices into target speaking style. This constraint limits the applicability of the methods. Recently, several many-to-many voice con- version methods have been proposed, to convert voices in broader application scenarios. StarGAN- VC (Kameoka et al. 2018) uses StarGAN (Choi et al.] 2018) to enable many-to-many transfer, in which voices are fed into unique generator conditioned on the target speaker identity. A discrimi- nator is also used to evaluate generation quality and transfer accuracy. Blow (Serrà et al. (2019) is flow-based generative model (Kingma & Dhariwal 2018). that maps voices from different speakers into the same latent space via normalizing flow (Rezende & Mohamed 2015). The conversion is accomplished by transforming the latent representation back to the observation space with the tar- get speaker's identifier. Two other many-to-many conversion models, AUTOVC (Qian et al. 2019) and AdaIN-VC (Chou Lee 2019), extend applications into zero-shot scenarios, iee, conversion from/to speaker (unseen during training), based only few utterances. Both AUTOVC and AdaIN-VC construct an encoder-decoder framework, which extracts the style and content of one speech sample into separate latent embeddings. Then when a voice from an unseen speaker comes, both its style and content embeddings can be extracted directly. However, as discussed the Introduction, both methods do have explicit regularizers reduce the correlation between style and content embeddings, which limits their performance.
Disentangled Representation Learning Disentangled representation learning (DRL. aims to code data points into separate independent embedding subspaces, where different subspaces rep- resent different data attributes. DRL methods can be classified into unsupervised and supervised approaches. Under unsupervised setups, Burgess et al. (2018), Higgins et al. (2016) and Kim & Mnih (2018) use latent embeddings to reconstruct the original data while keeping each dimension of the embeddings independent with correlation regularizers. This has been challenged by Locatello jet al. (2019), that each part of the learned embeddings may not be mapped to a meaningful data at- tribute. In contrast, supervised DRL methods effectively learn meaningful disentangled embedding parts by adding different supervision to different embedding components. Between the two embed- ding parts, the correlation is still required to be reduced to prevent the revealing of information to each other. The correlation-reducate reducing methods mainly focus on adversarial training between embed- ding parts (Hjelm et al. 2018 Kim & Mnih 2018), and mutual information minimization (Chen) et al. 2018 Cheng et al. 2020b). By applying operations such as switching and combining, one can improve empirical performance on downstream tasks, e.g. con- ditional generation (Burgess et al. 2018), domain adaptation (Gholami et al. 2020), and few-shot learning (Higgins et al.) 2017).
use disentangled representations
5

---

Published as a conference paper at ICLR 2021
5 EXPERIMENTS
We evaluate our IDE-VC on real-world voice a dataset under both many-to-many and zero-shot VST setups. The selected dataset CSTR Voice Cloning Toolkit (VCTK) (Yamagishi et al., 2019), which includes 46 hours of audio from 109 speakers. Each speaker reads a different sets of utterances, and the training voices are provided in a non-parallel manner. The audios are downsampled at 16kHz. In the following, we first describe the evaluation metrics and the implementation details, and then analyze our model's 'ss performance relative to other baselines under many-to-many and zero-shot VST settings.
5.1 EVALUATION METRICS
Objective Metrics We consider two objective metrics: Speaker verification accuracy (Verification) and the Mel-Cepstral Distance (Distance) (Kubichek]] 1993),. The speaker verification accuracy mea- sures whether the transferred voice belongs to the target speaker. For fair comparison, we used third-party pre-trained speaker encoder Resemblyzer to classify the speaker identity from the trans- ferred voices. Specifically, style centroids for speakers are learned with ground-trutt voice samples. For a transferred voice, we encode it via the pre- treirrd speaker encoder and find the speaker with the closest style centroid the identity prediction. For the Distance, the vanilla Mel-Cepstral Dis- tance (MCD) cannot handle the time alignment issue described in Section 2 To make reasonable comparisons between the generation and ground truth, we apply the Dynamic Time Warping (DTW) algorithm (Berndt & Clifford 1994) to automatically align the time-evolving sequences before cal- culating MCD. This DTW-MCD distance measures the similarity of the transferred voice and the real voice from the target speaker. Since the calculation of DTW-MCD requires parallel data, we select voices with the same content from the VCTK dataset as testing pairs. Then we transfer one voice in the pair and calculate DTW-MCD with the other voice as reference.
Subjective Metries Following Wester et al. (Wester et al., 2016), we use the naturalness of the speech (Naturalness), and the similarity of the transferred speech to target identity (Similarity) as subjective metrics. For Naturalness, annotators are asked to rate the score from 1-5 for each trans- ferred speech.For For the Similarity, the annotators are presented with two audios (the converted speech and the corresponding reference), and are asked to rate the score from to 4. For both scores, the higher the better. Following the setting in Blow (Serrà et al., 2019), as a total percentage from the binary rating. The evaluation of both subjective metrics is conducted on Amazon Mechanical Turk (MTurk More details about evaluation metrics are provided in the Supplementary Material.
report Similarity defined
5.2 IMPLEMENTATION DETAILS
Following AUTOVC Qian et al. 2019), our model inputs are represented via mel-spectrogram. The number of mel-frequee frequency bins set 80. When voices are generated. we adopt the WaveNet vocoder (Oord et al., 2016) pre-trained on the VCTK corpus to invert the spectrogram signal back to waveform. The spectrogram is first upsampled with deconvolutional layers to match the sampling rate, and then standard 40-layer WaveNet is applied to generate speech waveforms. Our model is implememted with Pytorch and takes I GPU day on an Nvidia Xp train.
Encoder Architecture The speaker encoder consists of a 2-layer long short-term memory (LSTM) with cell size of 768, and a fully-connected layer with output dimension 256. The speaker encoder is initialized with weights from a pretrained GE2E (Wan et al. 2018) encoder. The input of the content encoder is the concatenation of the mel-spectrogram signal and the corresponding speaker embedding. The content encoder consists of three convolutional layers with 512 channels, and two layers of a bidirectional LSTM with cell dimension 32. Following the setup AUTOVC (Qian) al. 2019), the forward and backward outputs of the bi-directional LSTM are downsampled by 16.
Decoder Architecture Following AUTOVC (Qian et al. 2019), the initial decoder consists of a three-layer convolutional neural network (CNN) with 512 channels, three LSTM layers with cell dimension 1024, and another convolutional layer to project the output of the LSTM to dimension of 80. To enhance the quality of the spectrogram, following AUTOVC (Qian et al. 2019). we use nest-network consisting of five convolutional layers with 512 channels for the first four layers, and
'https//gituu..com//eee http:/hhwwh.ccmm////mmmm////eeee hhtps://www.nn. http:///wwww.uuk.com////

---

Published as a conference paper at ICLR 2021
Table Many-to-many VST evaluation results. For all metrics except Distance, higher better.
Metric
Objective
Subjective
Distance
Verification((c]
Naturalness [1-5]
Similarity [%]
StarGAN
6.73
71.1
2.77
51.5
AdaIN-VC
6,98
85.5
2.19
50,8
AUTOVC
6.73
89,9
3.25
55.0
Blow
8.08
2.11
10.8
IDE-VC (Ours)
6.70
92.2
3.26
68.5
Table 2: Zero-Shot VST evaluation results. For all metrics except Distance, higher is better.
Metric
Objective
Subjective
Distance
Verification(%/
Naturalness [1-5]
Similarity [%]
AdaIN-VC
6.37
76.7
2.67
68.4 4
AUTOVC
6.68
60.0
2.19
58.6
IDE-VC (Ours)
6.31
81.1
3.33
76.4
80 channels for the last layer. The output of the post-network can be viewed as a residual signal. The final conversion signal is computed by directly adding the output of the initial decoder and the post-network. The reconstruction loss is applied to both the output of the initial decoder and the final conversion signal.
Approximation Network Architecture As described in Section 3.3 minimizing the mutual in- formation between style and content embeddings requires an auxiliary variational approximation (s)ccc. For implementation, we parameterize the variational distribution in the Gaussian distribu- tion family (s|c)c c) (c)
I), where mean (a(() and variance a-(-) are two-layer fully-connected networks with tanh(-) as the activation function. With the Gaussian parameteriza- tion, the likelihoods in objective In can be calculated in closed form.
5.3 STYLE TRANSFER PERFORMANCE
For the many-to many VST task, we randomly select 10% of the sentences for validation and 10% of the sentences for testing from the VCTK dataset, following the setting in Blow (Serrà et al. 2019). The rest of the data are used for training in non non-parallel scheme. For evaluation, pairs from the testing set, in which each pair of voices have the same content but come from different speakers. In each testing pair, we conduct transfer from one voice to the other voice's s speaking style, and then we compare the transferred voice and the other voice as evaluating the model performance. We test our model with four competitive baselines: Blow (Serrà al. 2019 AUTOVC (Qian) et al. 2019). AdaIN-VC (Chou & Lee 2019) and StarGAN- (Kameoka et al. 2018). The de- tailed implementation of these four methods are provided in the Supplementary Material. Table shows the subjective and objective evaluation for the many-to-many VST task. Both methods with the -ecoder framework, AdaIN-VC and AUTOVČ, have competitive results. However, our IDE-VC outperforms the other baselines on all metrics, demonstrating that the style-content disen- tanglement in the latent space improves the performance of the encoder-decoder framework.
select voice
For the zero-shot VST task, we use the same train-validation dataset split as in the many-to-many setup. The testing data are selected to guarantee that no test speaker has any utterance in the training set. We compare
model with the only two baselines, AUTOVC (Qian et al. 2019) and AdaIN- VC (Chou & Lee| 2019), that are able to handle voice transfer for newly-coming unseen speakers. We used the same implementations of AUTOVC and AdaIN-VC as in the many-to-many VST. The evaluation results of zero-shot VST are shown in Table 2 among the two baselines AdaIN- VC performs better than AUTOVC overall.Our IDE-VC outperforms both baseline methods, on all met- rics. All three tested models have encoder-decoder transfer frameworks, the superior performance
For Blow model, we use the official implementation available Github https://gthub..omjjans/hhtt We report the best result we obtain here, under training for 100 epochs (11.75 GPU days Nvidia V V100).
7

---

Published as a conference paper at ICLR 2021
240
Figure 2: Left: 1-SNE visualization for speaker embeddings. Right: t-SNE visualization for content
embedding. The embeddings are extracted from the voice samples of 10 different speakers.
of IDE-VC indicates the effectiveness of our disentangled representation learning scheme. More evaluation details are provided in the supplementary material.
5.4 DISENTANGLEMENT DISCUSSION
Besides the performance comparison with other VST baselines, we demonstrate the capability of our information-theoredt
disentangled representation learning scheme. First, we conduct t- SNE (Maaten & Hinton 2008) visualization of the latent spaces of the IDE-VC model. As shown in the left of Figure 2 style embeddings from the same speaker are well clustered, and style em- beddings from different speakers separate in a clean manner. The clear pattern indicates our style encoder E, can verify the speakers' identity from the voice samples. In contrast, the content embed- dings (in the right of Figure 2 are indistinguishable for different speakers, which means our content encoder F,, successfully eliminates speaker-related information and extracts rich semantic content from the data.
We also empirically evaluate the disentanglement, by predicting speakers' identity based on only the content embeddings. A two-layer fulle-comnected network is trained the testing set with content embedding as input, and the corresponding speaker identity output. We compare AdaIN-VC, which also output content embeddings. The classi- fication results shown in Table 3 Our IDE-VC reaches the lowest classification accuracy, indicating that the content embed- dings learned by IDE-VC contains the least speaker-related in- formation. Therefore, our IDE-VC learns disentangled represen- tations with high quality compared with other baselines.
Table 3: Speaker identity pre- diction accuracy on content em- bedding.
IDE-VC with AUTOVC and
Accuracy! [6]
AUTOVC AdaIN-VC
9.5
19.0
IDE-VC
8.1
5.5 ABLATION STUDY
Moreover, have considered an ablation study that ad- dresses performance effects from different learning losses in (11), with results shown in Table model with two models trained by part of the loss func- tion in [T]. while keeping the other training setups un- changed, including the model structure. From the results, when the model trained without the style encoder loss term 2, transferred voice still is generated, but with a large distance to the ground truth. The verification accu- racy also significantly decreases with no speaker-related information utilized. When the disentangling term Σą is removed, the model still reaches com- petitive performance, because the style encoder E, and decoder are well trained by Σ, and Σz. However, when adding term Σ,, we disentangle the style and content spaces, and improve the trans- fer quality with higher verification accuracy and less distortion. The performance without term 2, is reported, because the model cannot even generate fluent speech without the reconstruction loss.
Table 4: Ablation study with different training losses. Performance mea- sured by objective metrics.
We compare our
Distance Verification|
Without 7, 9.81
Without Ia
6.73
89.4
IDE-VC 5.66
92.2
8

---

Published as a conference paper at ICLR 2021
6 CONCLUSIONS
We have improved the encoder-decoder voice style transfer framework by disentangled latent rep- resentation learning. To effectively induce the style and content information of speech into inde- pendent embedding latent spaces, we minimize sample-based mutual information upper bound between style and content embeddings. The disentanglement of the two embedding spaces ensures the voice transfer accuracy without information revealed from each other. We have also derived two new multi-group mutual information lower bounds, which are maximized during training to enhance the representativeness of the latent embeddings. On the real-world VCTK dataset, our model out- performs previous works under both many-to-many and zero-shot voice style transfer setups. Our model can be naturally extended to other style transfer tasks modeling time-evolving sequences, e.g. video and music style transfer. Moreover, our general multi-group mutual information lower bounds have broader potential applications other representation learning tasks.
ACKNOWLEDGEMEENTS
This research was supported in part by the DOE, NSF and ONR.
REFERENCES
Alexander A Alemi, Ian Fischer, Joshua Dillon, and Kevin Murphy, Deep variational information bottleneck. arXiv preprint 161iv:1612.004100 2016.
Donald J Berndt and James Clifford. Using dynamic time warping to find patterns in time series. In KDD workshop, pp. 359-370. Seattle, WA, 1994.
Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Des- jardins, and Alexander Lerchner. Understanding disentangling in beta-vae. arXiv preprint arXiv:1804.03599, 2018.
Santosh V Chapaneri.
Spoken digits recognition using weighted mfce and improved features for dynamic time warping. International Journal of Computer Applications, 40(3):6-12, 2012.
Dongdong Chen, Jing Liao, Lu Yuan, Nenghai Yu, and Gang Hua. Coherent online video style transfer. Proceedings of the IEFE International Conference on Computer Vision, pp. 1105- 1114, 2017.
Tian Qi Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentan- glement in variational autoencoders. In Advances in Neural Information Processing Systems, pp- 2610-2620, 2018
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, pp. 2172-2180, 2016.
Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan, and Lawrence Carin. Club: A contrastive log-ratio upper bound of mutual information. In International Conference on Machine Learning. pp. 1779-1788. PMLR, 2020a.
Pengyu Cheng, Renqiang Min, Shen Dinghan, Christopher Malon, Yizhe Zhang, Li Yitong, and Lawrence Carin. Improving disentangled text representation learning with coommationthhoredte guidance. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, 2020b.
Yunjey Choi. Minje Choi, Munyoung Kim, Jung-Woo Woo Ha, Sunghun Kim, and Jaegul Choo. Star- gan: Unified generative adversarial networks for multi-domain image-to-imass translation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 8789-8797, 2018.
Ju-chieh Chou and Hung- Yi Lee. One-shot voice conversion by separating speaker and content representations with instance normalization. Proc. Interspeech 2019, pp. 664-668. 2019.

---

Published as a conference paper at ICLR 2021
Thomas M Cover and Joy A Thomas. Elements of information theory. John Wiley & Sons, 2012.
Shivanker Dev Dhingra, Geeta Nijhawan, and Poonam Pandit. Isolated speech recognition using mfec and dtw. International Journal of Advanced Research in Electrical, Electronics and Instru- mentation Engineering, 2(8):4085 4092, 2013.
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and puttern recog- nition, pp. 2414-2423, 2016.
Behnam Gholami, Pritish Sahu, Ognjen Rudovic, Konstantinos Bousmalis, and Vladimir Pavlovic. Unsupervised multi-targe' domain adaptation: An information theoretic approach. IEEE Trans- actions on Image Processing, 2020.
Elizabeth Godoy, Olivier Rosec, and Thierry Chonavel. Voice conversion using dynamic frequency warping with amplitude scaling. for parallel or nonparallel corpora. IEEE Transactions on Audio, Speech, and Language Processing, 20(4):1313-1323 2011.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 297-304, 2010.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vac: Learning basic visual concepts with a constrained variational framework. In International Conference Learning Representations, 2016.
Irina Higgins, Arka Pal, Andrei Rusu, Loic Matthey, Christopher Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner. Darla: Improving zero-shot trans- fer in reinforcement learning. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pp. 1480-1490. JMLR. org, 2017.
R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1800 1808.06670, 2018.
Chin-Cheng Hsu, Hsin-Te Hwang, Yi-Chiao Wu, Yu Tsao, and Hsin-Min Wang. Voice conver- sion from non-parallel corpora using variational auto-encoder. In 2016 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA), pp. 1-6. IEEE, 2016.
Haozhi Huang, Hao Wang, Wenhan Luo, Lin Ma, Wenhao Jiang, Xiaolong Zhu, Zhifeng Li, and Wei Liu. Real-time neural style transfer for videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 783-791, 2017.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance malization. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1501- 1510, 2017.
Hirokazu Kameoka, Takuhiro Kaneko, Kou Tanaka, and Nobukatsu Hojo. Stargan-ve: Non-parallel many-to-many voice conversion using generative adversarial networks. In 2018 IEEE Spoken Language Technology Workshop (SLT), pp. 266-273. IEFF, 2018.
Takuhiro Kaneko and Hirokazu Kameoka. Cyclegan-ve: Non-parallel voice conversion using cycle- consistent adversarial networks. In 2018 26th European Signal Processing Conference (EU- SIPCO), pp. 2100-2104. IEEE, 2018.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference on Machine Learning, pp. 2649-2658, 2018
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ix1 convolutions, In Advances in Neural Information Processing Systems, pp. 10215-10224, 2018.
10

---

Published as a conference paper at ICLR 2021
Robert Kubichek. Mel-cepstral distance measure for objective speech quality assessment. In Pro- ceedings of IEEE Pacific Rim Conference on Communications Computers and Signal Processing, volume 1. pp. 128-128. IEEE, 1993.
Guillaume Lample, Sandeep Subramanian, Eric Smith, Ludovic Denoyer, Mare' Aurelio Ranzato, and Y-Lan Boureau. Multiple-attribus text rewriting. In International Conference on Learning Representations, 2019.
Chung-Han Lee and Chung-Hsien Wu. Map-based adaptation for speech conversion using adap- tation data selection and non-parallel training. In Ninth International Conference on Spoken Language Processing, 2006.
Francesco Locatello, Stefan Bauer, Mario Lucie, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In International Conference on Machine Learning, pp. 4114 4124, 2019.
Fujun Luan, Sylvain Paris, Eli Shechtman, and Kavita Bala. Deep photo style transfer. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4990-4998, 2017.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):2579-266) 2008.
Seyed Hamidreza Mohammadi and Alexander Kain. An overview of voice conversion systems. Speech Communication, 88:65 82, 2017.
Kevin R Moon and Alfred O Hero. Ensemble estimation of multivariate f-divergence. In 2014 IEEE International Symposium on Information Theory, pp. 356-360. IEEE, 2014.
Lindasalwa Muda, Mumtaj Begam, and Irraivan Elamvazuthi. Voice recognition algorithms using mel frequency cepstral coefficient (mfec) and dynamic time warping (dtw) techniques. arXiv preprint arXiv:1003.4083 2010.
Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. Voxceleb: a large-scale speaker identifi- cation dataset. arXiv preprint arXiv::706.. 02.12, 2017.
Keigo Nakamura, Tomoki Toda, Hiroshi Saruwatari, and Kiyohiro Shikano. Speaking aid system for total laryngectomees using voice conversion of body transmitted artificial speech. In Ninth International Conference Spoken Language Processing, 2006.
XuanLong Nguyen, Martin J Wainwright, and Michael Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory, 56(11):5847-5861 2010.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03499 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.03744 2018.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE International Conference Acousties, Speech and Signal Processing (ICASSP), pp. 9206-52100 IEEE, 2015.
Kaizhi Qian, Yang Zhang, Shiyu Chang, Xuesong Yang, and Mark Hasegawa-Johnson. Autove: Zero-shot voice style transfer with only autoencoder loss. International Conference on Machine Learning, pp. 5210-5219, 2019.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. Interna- fional Conference on Machine Learning, pp. 1530-1538. 2015.

---

Published as a conference paper at ICLR 2021
Yuki Saito, Yusuke Ijima, Kyosuke Nishida, and Shinnosuke Takamichi. Non-parallel voice version using variational autoencoders conditioned by phonetic posteriorgrams and d-vectors. In 2018 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp- 5274-5278. IEEE, 2018.
Joan Serrà, Santiago Pascual, and Carlos Segura Perales. Blow: a single-scale hyperconditioned flow for non- con-parallel raw-audio voice conversion. In Advances in Neural Information Processing Systems, pp. 6790-6800, 2019.
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. Style transfer from non-parallel text by cross-alignment. In Advances neural information processing systems, pp. 6830-6841, 2017.
Berrak Sisman, Mingyang Zhang, Sakriani Sakti, Haizhou Li, and Satoshi Nakamura. Adaptive wavenet vocoder for residual compensation in gan-based voice conversion. In 2018 IEEE Spoken Language Technology Workshop (SLT), pp. 282-289. IEEE, 2018.
Jiaming Song, Pratyusha Kalluri, Aditya Grover, Shengjia Zhao, and Stefano Ermon. Learning controllable fair representations. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 2164 2173, 2019.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in- gredient for fast stylization. arXiv preprint arXiv:1607.08022 2016.
Petar Veličković, William Fedus, William Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. arXiv preprint arXiv: 10341, 2018.
Fernando Villavicencio and Jordi Bonada. Applying voice conversion to concatenative singing- voice synthesis. In Eleventh annual conference of the international speech communication association, 2010.
Li Wan, Quan Wang, Alan Papir, and Ignacio Lopez Moreno. Generalized end-to-end loss for speaker verification. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4879 4883. IEEE, 2018.
Mirjam Wester, Zhizheng Wu, and Junichi Yamagishi. Analysis of the voice conversion challenge 2016 evaluation results. In Interspeech, pp. 1637-1641, 2016.
Junichi Yamagishi, Christophe Veaux, Kirsten MacDonald, al. Cstr vetk corpus: English multi- speaker corpus for estr voice cloning toolkit (version 0.92). 2019.
Zichao Yang. Zhiting Hu, Chris Dyer, Eric P Xing, and Taylor Berg- Kirkpatrick. Unsupervised text style transfer using language models as discriminators. In Advances in Neural Information Processing Systems, pp. 7287-7298, 2018.
12


Reviews:
[{"review_id": "zkgiUlPfh-J", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This submission proposes a training approach for voice style transfer using encoder-decoder framework and content and style representations. The approach combines multiple mutual-information (MI) based terms into a single objective function. One of the MI based terms is the MI between content and style representations. By minimising mutual information between these representations, the training approach yields models where these representations are disentangled. Experimental results show that this approach leads to improved performance in speaker verification and speech similarity tasks. Experimental results in challenging zero-shot conditions also demonstrate improved performance in speaker verification, speech naturalness and speech similarity tasks. \n\nQuality: The quality of this submission appears to be generally OK. There are few cases I would like to draw your attention to:\n\n1) Although I understand the desire to give a theoretic flavour to this work, I am quite concerned that you boiled down all of the information theory into the mutual information. \n2) \"our ... outperforms the other baselines on all metrics\" is a very problematic statement to make. First of all, it is a very uninformative statement, I can see numbers myself. Second, it is a very poor analysis - I learnt nothing from this as you provide no more information. Third, some gains are marginal so such claim is not very strong. Fourth, I am more interested if what you expect to be better at compared to AUTOVC actually transpires into gains. \n3) Based on Table 3 you make a claim that your approach disentangles representations with higher quality compared to other baselines. I am sorry but I do not see that and am surprised why would you say that about accuracies of 8.1 and 9.5%?\n4) I am also surprised not to see discussion about work done on learning language, speaker, etc invariant representations in speech recognition. \n\nClarity: The clarity of this submission appears to be generally OK. There are few cases I would like to draw your attention to:\n\n1) Given statements such as \"good approximation\", you must make it absolutely unambiguous whether the final loss you are optimising is a bound or not;\n2) You are throwing \"based on the MI data-processing inequality, we conclude\" in a passing by manner on page 3. I do not consider it to be an appropriate line of argument or explanation. You need to discuss this aspect in a significantly more details. \n3) It is very unclear how much credit to give you for deriving these bounds given that they are based on other bounds. \n\nOriginality: I think this work is moderately original. \n\nSignificance: I think this work will have a moderate significance. \n\nPros: \n\n1) Using MI to learn disentangled representations is an interesting idea that may give rise to follow up works. \n2) Zero-shot experiments show that this approach clearly outperforms baselines based on 3 out of 4 metrics. \n\nCons: \n\n1) This submission does not really make a convincing case for using MI to learn disentangled representations generally. \n2) It also does not really strike a good balance in terms of theory versus engineering. The engineering part, which is way more important for this submission (at no point you go back to the theory in your submission), is left with a very limited space and very limited discussion about options available. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "wtmokxTtFy8", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a zero-shot voice style transfer (VST) algorithms that explicitly controls the disentanglement between content information and style information. Experiments show that the proposed algorithm can achieve significant improvement over the existing state-of-the-art VST algorithms. There are two major strengths of this paper. First, it motivates the algorithm design from an information-theoretic perspective. Second, the performance improvement is significant.\n\nHowever, since it is a resubmission from a previous machine learning conference, the previous concerns regarding limited novelty are not fully addressed. More specifically, if we view the proposed algorithm entirely from a technical perspective, there are two major innovations over AutoVC: 1) The style embedding is trained together with the content embedding, instead of being pre-trained; 2) The introduction of I3. The latter does not seem fully justified.\n\nFirst, it is shown in Table 4 that without I3, the drop in performance is not obvious. The authors ascribe this to that I1 and I2 already suffice to train the good model. Does it mean that the introduction of I3 is not as important an innovation as co-training?\n\nSecond and more importantly, in all the experiments, the proposed system retains AutoVC’s physical bottleneck design, which was the key to disentangling style in AutoVC. In order to justify that I3 is a better disentangling mechanism than AutoVC, it is necessary to perform an ablation study where the bottleneck is widened and see if I3 still guarantees disentanglement, without which it is hard to justify the value of I3.\n\nBesides the concern regarding novelty, there are a few other concerns. There lacks a back-to-back comparison in Figure 2. What do the embeddings look like for AdaINVC and AutoVC? Also, Figure 2 only shows that the content embedding does not include style information. It would also be helpful to show the style embedding does not include content information by showing the content embedding and style embedding cluster with respect to different phones.\n\nTo sum, without strong supporting evidence for the novel design in IDE-VC, it is hard to judge the contribution of this paper. I would look forward to more thorough evaluations in the rebuttal.", "extra_scores": {}, "extra_text": {}}, {"review_id": "h1jpFuUggwL", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a framework for \"disentangling\" style and content from audio samples, a very interesting topic. It's a good read until the math becomes a bit too hard/convoluted for me to follow -- it would help me, for one, if more intuitions were given and if the bottom line were explained more succinctly. Intuitions are actually given along the way, but I found the overall approach slightly beyond me -- or at least requiring considerable effort to piece together. My impression is that the approach is coherent, though, and the results are good. The citation of previous work is good.\n\nA number of specific comments follow.\n\nAbstract: \"On real-world datasets, our method outperforms other baselines and obtains state-of-the-art results in terms of\ntransfer accuracy and voice naturalness.\" \n\nIntro: \"Experiments demonstrate that our method outperforms previous works under both many-to-many and zero-shot transfer setups.\"\n\nBe more specific, what kinds of experiments, what datasets, what metrics?\n\n\"Among the a few existing models that address ...\" --> remove superfluous \"a\"\n\n\"To transfer a source xui from speaker u to the target style of the voice of speaker v, xvj ...\": --> \" ... to the target style of the voice of speaker v as reflected in a particular utterance j, from speaker v, ...\"? I could imagine the style of speaker v to be considered in aggregate, averaging somehow over all utterances from that speaker.\n\n\"(i.e., ideally, s contains rich style information of x with no content information, and vice versa)\" --> \"(i.e., ideally, s contains rich style\ninformation of x with no content information, and vice versa for c)\"\n\n\"... is the mean of all style embedding... \" --> embeddingS\n\n\"Under unsupervised setups, Burgess et al. (2018); Higgins et al. (2016); Kim & Mnih (2018) use... \" --> separate those references with commas, not semi-colons, i.e. --> \"... A, B and C use ...\"\n\nEq. (7), give intuition for the e^{-1} term?\n\n\"Based on the criterion for s in equation (7), a well-learned style encoder Es pulls all style embeddings sui from speaker u together\": I am not really following here, it starts out well with the intro to Theorem 3.1, but then I'm confused as to what is an upper/lower bound for what and that the actual criterion is. I don't quite know specifically what the authors mean by, \"a well-learned style encoder Es pulls all style embeddings sui from speaker u together\". I realize this work summaries derivations in the supplementary material, but ideally the central intuitions and actual specific bottom-line criteria used would be much clearer.\n\nFig 2b: show on the figure the use of I2 and I3, separate from I? Or at least, in the text, clarify how I2 is used? (I3 is mentioned).\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "a-_0N69IRCr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a way to do voice conversion by learning a disentangled representation of the style and content of the audio. This disentangled representation is enforced by minimizing the mutual information between these the style and content encodings of the audio. The paper shows maths to derive its loss functions.\n\nPros:\n1. The paper looks novel.\n2. The results section shows improvement over the chosen baseline.\n\nCons:\n1. Very difficult to understand/follow the central part of the paper.\n2. Associated samples would have been great to bolster the claims.\n\nHow does this work relate to other representation learning methods? \n\nI think the paper in the current form might be interesting to the ICLR community.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a new model to learn voice style transfer using an encoder-decoder framework with the aim of disentangling content and style representations.

The strengths of the paper are:
+ the method is well-motivated with sound theoretical justification
+ the authors improve up on the prior work by augmenting the loss with an information-theoretic term
+ empirical evaluations demonstrate performance improvements in speaker verification and speech similarity tasks
+ demonstrate improvements in the challenging zero-shot task

Several reviewers requested improvements in readability
+ “ideally the central intuitions and actual specific bottom-line criteria used would be much clearer.”
+ more clarify on empirical details including challenges that needed to be addressed

Decision: Accept (Poster)