ClsVC: Learning Speech Representations with two different classification tasks.
Authors: Huaizhen Tang
xulong Zhang
Jianzong Wang
ning Cheng
Jing Xiao
Venue: ICLR 2022 Submitted (2022)
Keywords: voice conversion
gradient reversal
adversarial learning
speech synthesis

Abstract: Voice conversion(VC) aims to convert one speaker's voice to generate a new speech as it is said by another speaker. Previous works focus on learning latent representation by applying two different encoders to learn content information and timbre information from the input speech respectively. However, whether they apply a bottleneck network or vector quantify technology, it is very difficult to perfectly separate the speaker and the content information from a speech signal. In this paper, we propose a novel voice conversion framework, 'ClsVC', to address this problem. It uses only one encoder to get both timbre and content information by dividing the latent space. Besides, some constraints are proposed to ensure the different part of latent space only contains separating content and timbre information respectively. We have shown the necessity to set these constraints, and we also experimentally prove that even if we change the division proportion of latent space, the content and timbre information will be always well separated. Experiments on the VCTK dataset show ClsVC is a state-of-the-art framework in terms of the naturalness and similarity of converted speech. 

Under review as a conference paper at ICLR 2022
CLSVC: LEARNING SPEECH REPRESENTATIONS WITH TWO DIFFERENT CLASSIFICATION TASKS.
Anonymous authors Paper under double-blind review
ABSTRACT
Voice conversion(VC) aims to convert one speaker's voice to generate new speech as it is said by another speaker. Previous works focus on learning latent representation by applying two different encoders to learn content information and timbre information from the input speech respectively. However, whether they apply bottleneck network or vector quantify technology. it is very difficult to perfectly separate the speaker and the content information from a speech sig- nal. In this paper, propose novel voice conversion framework, 'CIsVC', address this problem. It uses only one encoder to get both timbre and content in- formation by dividing the latent space. Besides, some constraints are proposed ensure the different part of latent space only contains separating content and tim- bre information respectively. We have shown the necessity set these constraints, and we also experimentally prove that even if we change the division proportion of latent space, the content and timbre information will be always well separated. Experiments on the VCTK dataset show CIsVC is a state-of-the-ae framework in terms of the naturalness and similarity of converted speech.
INTRODUCTION
Voice conversion (VC) is an exciting topic committed converting one utterance of source speaker into another utterance of target person by keeping the content in the original utterance while replacing with the vocal features from the target speaker. Up to now, many methods have been applied in VC successfully. Commonly, these methods can be roughly categorized into two classes, i... parallel VC and non- non-parallel VC Mohammadi & Kain (207)). Specifically, parallel VC means that model training requires parallel corpus, which is unnecessary for non-parallel VC. Recently, more researchers have focused on the solutions of non-parallel VC since is not easy for us to collect so many paired source-target speech datasets.
Early VC systems, like Gaussian Mixture Model (GMM), Styylanou et al. (1998); Toda et al. (2007) needed lot of parallel data for model training, and the generated speech quality was not good enough. With the advance of deep learning. a variety of novel VC methods have been proposed in recent years. Among them, GAN-based model is one of the most popular methods, Goodfellow let al. (2014); [ssu et al. (207a Kaneko & Kameoka (2018): Kaneko et al.] (2019a): Kameoka al. (2018); Kaneko et al. (2019b) which could learn global generative distribution of the target speech without explicit approximation. These GAN-based models jointly train a generator and a discriminator. An adversarial loss derived from the discriminator is used encourage the generator outputs to build indistinguishable from real speech. Thanks to the cycle consistency training, GAN- based VC models can be trained with non-parallel speech datasets.
Besides, learning discrete speech representations has also attracted much attention. Vector Quan- tization (VQ), an extremely important signal compression method, which can quantify continuous data into discrete data. Previous studies have confirmed that the quantized discrete data produced by the continuous speech data is closely related with the phoneme information Chorowski et al. (2019). Recently, VQVC Wu & Lee (2020) has been proposed to learn to disentangle the con- tent and speaker information with reconstruction loss only. Then, VQVC+ Wu et al. (2020) was presented to improve the conversion performance of VQVC by adding the U-Net architecture within an wee--ennndddrbaeeee VC system. To far improve the performance of disentangling content and speaker information, many other existing studies were introduced to combine with VQ, such as

---

Under review as a conference paper at ICLR 2022
VQ- Vec,, VQ- VAE and VQ-CPC. Baevski et al. (2019); Ding & Gutierrez-Osuna 2 van Nickerk et al. (2020)
There is also another line of research focus on learning latent representations with Autoencoder. In particular, Variational Auto Encoder(VAE)) is the most famous. The network structure of VAE contains encoder and decoder and the core idea is very clear: the encoder learns specific latent space from input speech and the decoder outputs a reconstructed speech from this latent space. In this process, VAE focuses on how force the encoder to learn specific latent space. So far, many VAE-based models have been successfully applied to HsJ et al. (2016); van den Oord et al. ;2017) Hssu al. (2017b); Ding Gutierrez- Osuna (2019;; Hsu et al. (2017a). In addition. Auto VC another successful application of Autoencoder. Qian et al. (2019) Through ingenious experimental design, VC uses two different encoders to learn content and speaker information, respectively, so that this model can achieve distribution-matching style transfer by only on self-reconstruction
loss.
Unfortunately, in the field of VC, all the models mentioned above have their inherent disadvan- tages. For example, GAN-based models can usually achieve a good conversion effect and ensure the matching between the generated data and the input data, but it is recognized that the training of GAN is very unstable. On the contrary, the training of VQVC is simple and fast enough, but the quantity of audio produced by this method is very poor. This may be because the discrete speech representations will inevitably lose some content information. In addition, although the VAF-based model also has great conversion effect, it can not guarantee distribution matching. VutoVC is great study, the training is very simple and achieves state-of-the-art results. However, in order to realize style conversion, it has to introduce a pre-trained speaker encoder.
Based on these existing methods, we naturally wonder if there is a new solution that can achieve the distribution matching as Vu and GAN, that trains as easily as VQVC and VAE, that can disentangle content and speaker information by only one encoder as VQ does, and also has better performance in voice conversion or in decoupling linguistic and timbre information from speech?
In this paper, we proposed a novel voice conversion framework to meet all the above requirements. Specifically, our model is similar to VAE, Autoencoder is the main framework of our model, and two different types of classification tasks are applied to force our model to separate the content and speaker information correctly. Here, the two classification tasks refer to general classification tasks and adversarial classification tasks respectively. The goal of the general classification task is to identify the features related to the speaker as accurately as possible, that is, the speaker informa- tion. While the latter is designed to climinate speaker information in latent space to get speaker- independent features, that is, content information. Experiment results are carried out on the VCTK dataset. Objective and Subjective evaluations demonstrate that the proposed method outperforms VQVC, Vuto VQ-VAE and StarGAN-VC in terms of naturalness and speaker similarity.
2 BACKGROUND
In mathematical statistics, if we already know the joint probability density functions of X and Y, we can easily find the marginal probability density functions of X and Y respectively. Formally, if (a,,)) y) ~ p(x, v) is known, we can get the marginal distributions of X and Y by the following formula:
fx fx( = f p(z, yd
(v = [ p(z, y)dx
(1)
Further, under the constraints of some setting conditions, although the closed-form of joint probabil- ity density functions p(x, y) in Eq. (1) is generally unknown, it is still feasible for neural network to learn the marginal distributions of and y from input samples when each corresponds to unique pair of (x. y).
Mutual information (MT), crucial indicator measure the dependence between two different variables. Recently many MI estimators have been successfully applied to constrain neural networks disentangle different components of the input data. which can be formulated as
P(X,Y) I(X,)) / P(X,Y) Y) log PYX)
(2)
2

---

Under review as a conference paper at ICLR 2022
Audio X
Audio X-j
Decoder
Decoder
Z'
Z
Common Classifier
Sx
Adversarial Classifier
Cx
Sx
Z
Z
Encoder
Encoder
Encoder
................................ ................................
................................ ................................
Audio X
Audio X)
Audio X
(a) Training of ClsVC
(h) Conversion of ClsVC
Framework of ClsVC. Z is the latent variable, which is divided into two parts, namely Cx and Sx. Here, we assume that Cx represents content information, which is speaker independent, and Sx represents speaker information, which is closely related to speaker identity.
Figure I:
Where P(X) and P(Y) are the marginal distributions of X and respectively, and P(X, Y) de- notes the joint distribution of X and Y. Since it is hard to obtain the required distribution formula P(X, Y), many studies focus on proposing sample-based MI lower bound (or upper bound) get
an approximation that can be calculated. Moon & Hero (2014); Hjelm al. (2018) 8): Cheng et al. 20200)
Recently, Yuan el al. Yuan et al. 2020)- introduced a new MI estimator to learn content and style information from speech for voice conversion. Specifically, they proposed a novel MI-based learning objective to encourage content encoder to output the content embedding and guide speaker encoder to output the speaker embedding. Inspired by this, we proposed a new, simple and more effective framework for learning latent speech representation.
Gradient Reversal Layer (GRL) Gradient Reversal Layer (GRL) was first proposed to address the issue of domain adaption Ganin & Lempitsky (2015), which aims to force model output domain- shared features which are independent with domains. Specifically, GRL often located between an encoder and domain classifier. During the forward propagation, GRL acts identity transform. During the backpropagation, GRL takes the gradient from the subsequent level, multiplies it by -1 and passes to the preceding layer that the encoder and domain classifier have completely opposite optimization objectives.
3 METHOD
Firstly, for every speech & we use content embedding C. to represent linguistic information and speaker embedding S. is proposed represent timbre and style information. And, U means the set of speakers. The following two theorems are the premise of our framework:
The content embedding C. and speaker identity U are independent for each other, In addition, the probability of each speaker's 's speech being selected the same, Formally, P(U u ClCa)) P(U =u) constant regardless of the speaker identity
Theorem 3.1
Theorem 3.2 The speaker embedding Sae and speaker identity U are in one-to-one correspondence.
That is, for a speaker u who produced speech x, P(U = u S.S.) = 1 and for other speakers v(v # u) P(U v u =
3.1 PROBLEM FORMULATION
Given a dataset of multi speakers and their audio recordings X' where speaker u has N,, audio recordings. Formally, for each speaker u, X. {x,, 1- For every input speech æ E X. we T to represent the number of frames of speech æs, noted that T' is constant, which means that
3

---

Under review as a conference paper at ICLR 2022
for any speech segments longer than T, we randomly select T frames, at the same time, for those speech segments with a length shorter than T, we pad them with constant. In this case, æ can also expressed x11 T) which is a random process randomly sampled from the speech distribution px (+|Cx = Ca. Sx = S.). Here, we assume that every speech æ can be expressed as a function f(+) of content embedding C. and speaker embedding S. That is, f(C== S.)) And, also assume that æ is uniquely determined by C2 and Sz. Formally, 21 = 22 only if C22 = Cæ2 and Saz Based on this assumption, for V 24, Jij X, the following formula must be true:
S²1=
[[Sm;:((Can, Sw;)) I(Smg: [(Cm; (ay); I(Cr, I(C,, Sm)) I(Cng J(Cnj: Sa)).
(3)
Now, assume two speech w and 22 from speaker u and speaker e respectively, #1 - s(C Sa1) and I2 f(Cæz: Saz) Our goal to design a speech conversion framework generate new speech 11-2 which preserves the content information of æ1 but the speaker information is matched with æ2 Formally, an ideal converted speech should satisfy the following forms:
I(Sag 31) I(S02 21-2),,
I(Cr2 22) =
(4)
Based on the above assumption, the formula in Eq. (4) can be equivalent to ē1++2 = -f f(2,, Sei) This conclusion is quite strong, and the formal proof will be presented in the appendix.
3.2 THE PROPOSED FRAMEWORK
In this section, we will introduce the core component of our proposed framework. As is illustrated in Fia) our framework contains four modules. The first module is an encoder E. which learns a latent variable Z from input speech æ. Here we expect Z to be specific function of content embedding C. and speaker embedding Sz. Which can be formulated as:
- E(x) E(f(C2,, Sa)) S @ C.
(5)
where 4 means concat operation. In this case, Z contains both linguistic information and timbre information and when we divide Z into two parts, the first part the estimated speaker embedding See while the second part is the estimated content embedding Cz
However, without any constraints, the encoder may learn meaningless speech representation. To address this problem, two different types of classifiers are used to encourage our encoder to output the target Z. Specifically, when we divide Z into two parts, the first part is put into a common speaker classifier C to identify the speaker as correctly as possible. At the same time, the latter part put into an adversarial classifier C2. Noted that C2 also expects to correctly recognize the speaker's ss identity like C1, the only difference is that GRL is included in Cg, which will make the encoder expect to fool the adversarial classifier so that it cannot classify correctly. This two different classifiers will finally converge when the first part of Z is very closely related with the speaker information and the second part of Z is sufficiently speaker-independer speaker-indeppedeet such that the adversarial classifier C2 is not able to identify the speaker. Then we regard the first part of Z estimated speaker embedding Sn and the second part of Z is regarded as estimated content embedding C..
The last module in our framework is decoder D, which will output reconstruct speech o from input latent variable Z'. It is worth noting that here the latent variable Z' is not same with Z since Z' is produced by concating with C2 and the vector norm of Sa rather than original speaker embedding S.
3.3 How AND WHY DOES IT WORK
Here we will discuss how and why our model can induce style embedding and content embedding into independent representation spaces simultancously.
In training phase, a speech Tu produced by a speaker u is selected to feed into the encoder E to output a latent variable Z,, E(w,, 0z) where Og are the parameters of the encoder. After that, the output of encoder is then divided into two parts, the vector norm of the first part of Zu and the speaker identity are fed into common speaker classifier C1 while the estimated content embedding C... which is the second part of Z,, and the speaker identity u are fed into an adversarial speaker classifier C2 to predict the probability of the speaker identity according estimated speaker
4

---

Under review as a conference paper at ICLR 2022
embedding Se., and content embedding Cs. respectively, i.e. P. Ci(Sau, u. de,)) P' C2(Cr.,) u, 0c,) Where 9e, and ee, denote the parameters of common speaker classifier and adversarial speaker classifier respectively. Subsequently, new latent variable Zz is then fed into decoder D to output a reconstruct speech x'= = D(Z', 0a). Where 02 refer to the parameters of decoder.
Our goal is to disentangle the content embedding C and speaker embedding S. from a speech æ. Formally, the ideal content embedding and speaker embedding should satisfy I(Cz, S.) In addition, since the process from speaker u to his speech Cu to speaker embedding S=,, is a MarKov Chain, we can say that (C.... S..) 2 I(C=,, F.u) 2 I(Co... u) based on the MI-data processing inequality Cover (1999) So, we use I(Ca.. u) instead of I(C Sa..) according to the illustration of 3.1 for the ideal content embedding. I(Cz,, u) = O. So our goal is to minimize I(C=.. 2.) and we use the iirersarialcllasfffaatoon loss function to minimize the upper bound, At the same time, the common-classifient speaker embedding Sw,, to be as closely related to speaker identity as possible. They can be expressed as:
loss function was designed to encourage the
Coom-cls (0e, 0c
E
I(u 1:) log Pk
(6)
k=1
K
Ladv-cis (ees 002) = Σ 1(u== = k) logr/
(7)
Where I(-) is the indicator function, K is the number of speakers and u denotes speaker who pro- duced speech æc and ps: is the probability of speaker k. During training, for Leom Oc and tc, all optimized to minimize the classification loss to better identify the corresponding speaker. But for Cadveels , Ocz are still optimized to minimize the classification loss, whereas Oe are optimized to max- imize the classification loss to fool the classifier. Ideally, under these two constraints, the first part of the output of encoder will be more closely related speaker information while the second part of the output of encoder will be sufficiently the speaker. And then, we can easily get the ideal content embedding and speaker embedding the same time.
ereaker-indepeneent so that the classifier can not identify
Unfortunately, the truth is, when there are only the above two classification constraints, three differ- ent results may occur, as is illustrated in Figure when estimated speaker embedding S. contain some content information, like FigureZ[a) correspondingly, for estimated content embedding Ca these content information will be lost. In this case, the estimated content embedding S., are still speaker-indes independent and the common classifier can still correctly identify the speaker from esti- mated speaker embedding. The only drawback is that some content information will eventually be lost because we replace S with it's vector norm Sz. which will lead imperfect reconstruction. That is why we propose the reconstruct-loss function between input speech æ and the reconstruct speech x' That is, Lreson E||| x x: x=1 The key logic behind setting this loss function is clear: to minimize the reconstruction loss, the estimated content embedding C₂, will be encouraged to carry all the content information to avoid the results of Figure((a)
Or, consider such case, the estimated content embedding C. may contain some speaker infor- mation that can not be completely recognized by the adversarial speaker classifier C2. When happens, both the common classifier C1 and the adversarial classifier C2 still work well, and even our model can perform well in the speech reconstruction task. In other words, the above three objective functions can not restrict the occurrence of such situation as Figure 2b). But, in this case, the decoder may ignore S₂ because the estimated content embedding C. has provided both content information and speaker information. As result, our model can only reconstruct speech but can not convert any speech. To address this problem, we provide the code-reconstruction loss function between the latent variable Z and reconstruct latent variable 2 which is produced from the reconstruct speech x' Leode-recon = E[Il Z ( This objective function will constrain the decoder to focus on the estimated speaker embedding Sz- At the same time, since the speaker information is provided by the estimated speaker embedding, Sa were also encouraged to carry all the speaker information to minimize Leude recon The full objective function can be computed as:
L(O es 0a) Lreeon L recun + 8 BLaomess XLutvccs
(8)
5

---

Under review as a conference paper at ICLR 2022
Cx
Cx
De
De
En
De
Sx
Sx
Z
X
Z
X
2
Z
X
(a)
(b)
(c)
Figure 2: Three different case of disentangle different parts. Here Z are the latent variable output from the input speech, C denote the estimated content embedding and S. refer to the estimated speaker embedding, both C and St part of the latent variable Z, in addition, x' is the reconstruct speech and 2 are regarded as reconstruct latent variable. 2 = E(x'). Whether CN and S are properly selected may lead the above three different situations
Where a 3, and refers to the weight Leade-reon- feom-cls and Ladv-els respectively. With this objective loss function, ideal case Figure 2 will eventually appear. Specifically, in this case, two important assumptions will be met.
The estimated content embedding C2 carry all content information but not contain any speaker information. On the contrary, the estimated speaker embedding S contain all speaker information but have no content information. In other words, now C. are the real content embedding and Sp are the real speaker embedding. So that for any input speech v, there must æ = s(C,, S.) ~ px (x = Cz Sx = Sx))
With Co and Sa, an perfect reconstruction achieved. i.e. x' Where x' is the reconstruction of æ through the likelihood po, Ca S.)
Futher, based on the above two conditions and our previous assumptions, we then say for any index there must be z; f( S,,)) xy Pa: (-1Ca., Sz,) That is, in the process of VC, if define source utterance æ1 f(Car Sa,), target utterance 22 f(Cr, Say), and ø1-+2 is regarded as the converted speech. Then We can finally find the best encoder parameters Oe and decoder parameters Os to satisfy the following formula: L(Os, 0a) -Cadvel -Nlog and
KL(px Sex = C2i, Sx = Swa)) =
where K is the number of speakers, KL(-II-) is the KL-divergence.
Which means the optimizer loss function in Eq.(8) would imperceptibly satisfy the ideal speech conversion in (4)) That's why we say method can achieve distribution-matchinn style transfer as GAN and VC.
4 EXPERIMENTS
In this section, we will evaluate the performance our CIsVC on traditional many-to-many VC tasks and one-shot VC tasks. Specifically, many-to-many VC means that in conversion phase, both the selected source speaker and the target speaker are all appeared in training process. In contrast, one- shot VC focuses on some more difficult tasks, in which both the source speaker and target speaker are unseen in training dataset, and only one utterance of source and target speakers are required. Besides, we also empirically validate the convenience and robustness of the CIsVC framework. We will present the audio demo and further details may be found in our implementation code.
4.1 DATASETS AND CONFIGURATIONS
To evaluate the performance of different model VC tasks, comparative experiments are conducted VCTK Veaux et al. (2016), a high-fidelity multi-speaker English speech corpus. This corpus

---

Under review as a conference paper at ICLR 2022
contains 46 hours of speech data produced by 109 English speakers from different countries. In our work, the entire dataset is randomly divided into 2 sets: 100 speaker's recordings for training and other 9 speaker's recordings for testing. Before training, the sampling rate of all recordings is sampled to 16KHz, and the mel-spectrograms are computed through a short-time Fourier transform (STFT) with Hann windowing, where 1024 for FFT size, 1024 for window size and 256 for hop size. The STFT magnitude is transformed to the mel scale using 80 channel mel filter bank spanning 90 Hz to 7.6 kHz.
ClsVC is trained with batch size of sixteen for 200K steps on one NVIDIA V100 GPU, using the ADAM optimizer with 3, 0.9, 3 0.98. The weight in Eq.(8) are set to a = 0.1, 8 0.1, À = 0.5. StarGAN-VC2l AutoVC VAB and VQVC+ are chosen as the baseline models.
4.2 COMPARISON
To compare the performance of different models in VC tasks, we use both objective and subjective tests. Specifically, the Mel-Cepstral DistortionMMCD) between converted speech and the ground truth target speech is used as our objective evaluation to measure the similarity of the transferred voice and the real voice from the target speaker. And, we invited 12 humans (7 male and 5 female) participants to evaluate the quality of some converted speech generated from different models. After hearing every speech, the subjects are asked to choose a score from 1-5 points of the naturalness of the converted speech. The higher the score, the better the audio quality of converted speech, which called the Mean Opinion Score(MOS) test. In addition, all participants are also asked to take Voice Similarity Score(VSS) test. Where groups of utterances are rated with a score of 1-5 on the voice similarity, in each group, there five converted utterances generated from Auto VC, VQVC+, VAE, StarGAN-VC2 and CIs respectively, and one real utterance from the target speaker. In VSS test, we calculate the score according the timbre similarity given by the tester. Specifically, the similarity score of 5 corresponds to the converted speech most similar to the ground truth speech, while the similarity score of indicates that the tester does not think that the converted speech and the ground truth speech come from the same speaker. The results appear as shown below:
Table 1: Comparison of different models in traditional VC and one-shot vc
Traditional VC
One-Shot VC
Methods
MCD
MOS
VSS
MCD
MOS
VSS
VQVC+
7.08 + 0.22
3.31
+ 0.90
3.42
+ 0.85
8.41
+ 0.08
2.75 + 0.84
3.11 + 0.88
AutoVC
4.34 + 0.12
3.81
1.14
3.45
0.76
7.66
0.17
2.61
- 0.73
2.91 0.72
VAB
5.63 1 0.21
3.33
1 0.87
3.07
1 0.84
StarGAN-VC2
6.28
+ 0.09
3.45
+ 1.01
3.59
+ 0.87
CIsVC
4.06 1 0.11
4.09
1 0.86
4.03
1 0.82
4.81
0.13
4.18
1 0.73
3.75 1 0.74
As quoted in Table T in the traditional VC task, compared with other baseline models, our CIsVC has achieved the best results in both subjective and objective tests, which indicates that the speech produced by our method is much better than baselines'. The results of VSS test show that compared with VQVC+, AutoVC, VAE and StarGAN-VC2, our method makes the converted speech learn better timbre information, which improves the conversion effect.
For ohe-shot VC task, since VAE and StarGAN- VC2 can not achieve voice conversion for unseen speakers, we just compare our method with VQVC+ and AutoVC. The result shows that with only one utterance available for unseen speakers, the performance of AutoVC is greatly reduced, previous studies have reported this phoneme Tan et al. (02T)) By comparison, even for unseen speakers and only one utterance is available, our ClsVC still achieved amazing results.
worth noting to say that from Table I the objective and subjective results are not completely consistent. For example, the mean score of MCD of VAE is 5.63, which is lower than that of StarGAN-VC2, which means that the quality of speech generated from VAE should be higher than
7

---

Under review as a conference paper at ICLR 2022
StarGAN-VC2, but the MOS results are just the opposite. This indicates that sometimes a speech with a lower score of MCD may still have inferior audio quality. To issue the problem, we add an- other objective experiment. By applying well-known open- source speech detection toolkit, Resem- blyzer (https: / //g//tttttt thub. com/ resembl e /Resemb 1 yzer), we conduct fake speech detection test to compare the similarity of 11 unknown utterances (6 real ones, 5 fakes which are generated from different models respectively) against ground truth reference audio. The results are shown in Figure 5
1.30
1.30
Peal
Real
(a) Female to Female conversion
(b) Femalc to Male conversion
Figure 3: Results of fake speech detection. The prediction threshold in our test is 0.84
The experimental results support our conclusion, for both same-gender VC (Female to Female) and cross-gender VC (Female Male), our ClsVC always outperforms other baseline models. All these objective and subjective experiments show that CIs is a state-of-the-ar framework for VC under both traditional and one-shot conditions.
4.3 NECESSITY OF SOME LOSS FUNCTIONS
In this section, we will prove experimentally that the loss function we proposed is necessary for our framework. Specifically, we design an ablation experiment to observe the effect of Leude-reson on our framework. We retrain CIsVC without Leode-reeon- called 'CIsVCs'. Based on our assumption, without the constrain of the code-reconstruccion loss function, the estimated speaker embedding Sp may be ignored by the decoder, and our model can finally achieve a good self reconstruction but can not convert any speech. To test this hypothesis, we reuse Resemblyzer to do Speaker diarization test. In this test, we need prepare some ground truth speech from the source speaker and target speaker respectively, then we input a converted speech produced by 'CIsVC" and 'CIsVCs' respectively, and this toolkit will automatically identify the probability of who the input speech belongs to. For further comparison, we divide the converted audio into two gender groups, the one is same-gender conversion, including male to male and female to female VC, the other one is cross-gender con- version, including female to male and male to female VC, results of Speaker classification test are summarized in Figure ((a)
Besides. We also select some self reconstruction speech of the source speaker to compare the per- formance of 'CIsVC' and 'ClsVCs' in the speech reconstruction task. In practice, 12 human par- ticipants were invited to hear real speech and two reconstructed speech produced by 'CIsVC' and 'CIsVCs' respectively, they need to evaluate the similarity between the reconstructed speech and real speech and choose one that more similar to the ground truth. In addition, if it is difficult to judge, they can also choose the 'Fair' option. We draw the result into the following Figure 4bb)
The Results show that in the speech reconstruction task, 'CIsVCs' performs slightly worse than our 'CIsVC', but in the VC task, the performance of 'CIsVCs' is quite poor. As shown in Figure 4(a) the converted speech produced by 'CIsVCs' still has high similarity with the source speaker and low similarity with the target speaker, which indicates that ClsVCss cannot achieve VC without the constraints of Lunde.recom
8

---

Under review as a conference paper at ICLR 2022
D Source speaker
furget speuker
ClsVCs
0.82
0.83
0.83
0.8
15.6%
0.72
43.75%
Fair
0.6
0.58
0.51
0.4
0.35
0.38
40.6%
Same-1
Same-2
Cross-1
Cross-2
ClsVC
(a) Comparison of converted speech
(b) Comparison of reconstruction speech
Figure 4: Results of ablation experiment.
I:ClsVC; 2:ClsVCs; Same: same-gender conversion;
Cross: cross- cross-gender conversion;
4.4 DIMENSIONS OF TWO PARTS LATENT VARIABLE
Here we will discuss how to devide the content embedding C and speaker embedding Sz from latent space Z. In AutoVC, it is important to select the size bottleneck carefully to make the estimated content embedding contain all content information but have timbre information. But in our model, as we discussed before, no matter how we divide it, the first part of tends to be the ideal content embedding, and the second part tends be the ideal speaker embedding. In this case, we can determine the dimensions of C and Sn at will and it will be very convenient for us to get content embedding and speaker embedding with only encoder.
In order to verify that the decoupling ability of our model is equivalent under different partition modes, retrain CIsVC by changing the dimensions of C. and Sa. Specifically, ClsVC, the dimensions of C and Sp are 32 and 256, respectively. Now we retrain CIsVC by changing them 32 and 64, called 'CIsVC-diml' or, 64 and 32, clled 'ClsVC-dim2'. In addition, another 'clsve-dim3' with both the dimensions of C and S₂ in this model are 64. We select some unseen speakers' utterances(100 utterances per speaker) to these models to obtain the estimated speaker embedding Sr. then we plotted S,, in 2-D space with t-SNE in Figure 5
trained
(a) ClsVC
(b) CIsVC-diml
(c) ClsVC-dim2
(d) CIsVC-dim3
Figure 5: The visualization of speaker embedding, None of these speaker appeared in training.
Results shown in Figure indicate that content and timbre information will always be well separated even we change the division proportion of latent space.
CONCLUSION
In this paper, we proposed ClsVC, a novel VC system learning latent speech representation. During training, common speaker classifier is proposed to encourage the estimated speaker embedding to become more and more related to the speaker identity and an adversarial classifier will focus the es- timated content embedding more beeaker-independene to make the encoder learn the ideal latent space. All subjective and objective experimental results show that the method proposed is state-of-the-art.
We also introduce other objective functions

---

Under review as a conference paper at ICLR 2022
REFERENCES
Alexei Baevski, Steffen Schneider, and Michael Auli. vq-wav2vec: Self-supervised learning of discrete speech representations. In International Conference on Learning Representations, 2019.
Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan, and Lawrence Carin. Club: A contrastive log-ratio upper bound of mutual information. In International Conference on Machine Learning, pp. 1779-1788. PMLR, 2020.
Jan Chorowski, Ron Weiss, Samy Bengio, and Aäron van den Oord. Unsupervised speech repre- sentation learning using wavenet autoencoders. IEEE/ACM transactions on audio, speech, and language processing, (((12)):0441225555 2019.
Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
Shaojin Ding and Ricardo Gutierrez-Osune Group latent embedding for vector quantized variational autoencoder in non-parallel voice conversion. In INTERSPEECH, pp. 724-728, 2019.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference on machine learning, pp. 1180-1189, PMLR, 2015.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Devon Hjelm, Alex Fedorov, Samuel Lavoie- Aavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations, 2018.
Chin-Cheng Hsu, Hsin-Te Hwang, Yi-Chiao Wu, Yu Tsao, and Hsin-Min Wang. Voice conver- sion from non-parallel corpora using variational auto-encoder. In 2016 Asia-Pucific Signal and Information Processing Association Annuul Summit and Conference (APSIPA), pp. 1-6. IEEE, 2016.
Chin-Cheng Hsu, He Hwang. Yi-Chiao Wu, Yu Tsao, and Hsin-Min Wang. Voice conver- sion from unaligned corpora using variational autoencoding wasserstein generative adversarial networks. In Francisco Lacerda (ed.), Interspeech 2017, 18th Annual Conference of the Interna- tional Speech Communication Association, Stockholm, Sweden, August 20-24, 2017, pp. 3364- 3368. ISCA, 2017a.
Wei-Ning Hsu, Yu Zhang, and James Glass. Unsupervised learning of disentangled and interpretable representations from sequential data. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 1876-1887, 2017b.
Hirokazu Kameoka, Takuhiro Kaneko, Kou Tanaka, and Nobukatsu Hojo. Stargan-ve: Non-parallel many-to monn-tto-mayy voice conversion using star generative adversarial networks. In 2018 IEEE Spoken Language Technology Workshop (SLT), pp. 266 273. IEEE, 2018.
Takuhiro Kaneko and Hirokazu Kameoka. Cyclegan-ve. Non-parallel voice conversion using cycle- consistent adversarial networks. In 2018 26th European Signal Processing Conference (EU- SIPCO), pp. 2100-2104. IEEE, 2018
Takuhiro Kaneko, Hirokazu Kameoka, Kou Tanaka, and Nobukatsu Hojo. Cyclegan vc2: Improved cyclegan-based non-parallel voice conversion. In ICASSP 2019-2019 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), pp. 6820-688 6824. IEEE, 2019a.
Takuhiro Kancko, Hirokazu Kameoka, Kou Tanaka, and Nobukatsu Hojo. Stargan- vc2: Rethinking conditional methods for stargan-based voice conversion. Proc. Interspeech 2019, pp. 679-683, 2019b.
Seyed Hamidreza Mohammadi and Alexander Kain. An overview of voice conversion systems. Speech Communication, 88:65-82, 2017.
10

---

Under review as a conference paper at ICLR 2022
Kevin R Moon and Alfred O Hero. Ensemble estimation of multivariate f-divergence. In 2014 IEEE International Symposium on Information Theory, pp. 256-360. IEEE, 2014.
Kaizhi Qian, Yang Zhang, Shiyu Chang, Xuesong Yang, and Mark Hasegawa-Johnson Autove: Zero-shot voice style transfer with only autoencoder loss. In International Conference on Machine Learning, pp. 5210-5219. PMLR, 2019.
Yannis Stylianou, Olivier Cappé, and Eric Moulines. Continuous probabilistic transform for voice conversion. IEEE Transactions on speech and audio processing, 6(2):131-14, 1998.
Zhiyuan Tan, Jianguo Wei, Junhai Xu, Yuqing He, and Wenhuan Lu. Zero-shot voice conver- sion with adjusted speaker embeddings and simple acoustic features. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2021, Toronto, ON, Canada, June 6-11, 2021, pp. 5964-5968. IEEE, 2021. doi: 10.1 10///CASSP399722200000 2021.999 4975. URL .or./.o.gg/.i..
(tt.0///CASSSP999P 2021 99144955
Tomoki Toda, Alan W Black, and Keiichi Tokuda. Voice conversion based on maximum- likelihood estimation of spectral parameter trajectory. IEEE Transactions on Audio, Speech, and Language Processing, 15(8):2222 22232,22222222222 2007.
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learn- ing. In Proceedings of the 31st International Conference on Neural Information Processing Sys- tems, pp. 6309-63 6318, 2017.
Benjamin Niekerk, Leanne Nortje, and Herman Kamper. Vector-quantized neural networks for acoustic unit discovery in the zerospeech 2020 challenge. In Helen Meng, Bo Xu, and Thomas Fang Zheng (eds.), Interspeech 2020, 21st Annual Conference of the International Speech Communication Association, Virtual Event, Shanghai, China, 25-29 October 2020, pp. 4836- 4840. ISCA, 2020.
Christophe Veaux. Junichi Yamagishi, Kirsten MacDonald, et al. Superseded-estr vetk corpus: En- glish multi-speaker corpus for cstr voice cloning toolkit. 2016.
Da-Y Yi Wu and Hung-yi Lee. One-shot voice conversion by vector quantization. In ICASSP 2020- 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7734-7738. IEEE, 2020.
Da-Yi Wu, Yen-Hao Chen, and Hung-yi Lee. Vqve+: One-shot voice conversion by vector quanti- zation and u-net architecture. Proc. Interspeech 2020, pp. 4691-4695, 2020.
Siyang Yuan, Pengyu Cheng, Ruiyi Zhang, Weituo Hao, Zhe Gan, and Lawrence Carin. Improving zero-shot voice style transfer via disentangled representation learning. In International Confer- ence on Learning Representations, 2020.
6 APPENDIX
Here we will present the formal proof of some equations which are appeared in the text of this paper.
About (3);
P(X.)) Given I(X,Y) -Ix Jy P(X,Y) log P(X)PTY) we can infer that
I(Sm,, Sm:)) J Ssc J(CCo.o.))
J P(Smg f(Ca, )2 log
P(S,,) f(C2, S2,)) P(S P(J[C Sz:;)
Where
P(Se,, ((i))
P(Sx = =x. Sa,) SS = Sw,)
P(S,, P(f(Car Swa))
P(Sx= = So, Sx) = f(O2, Sz.))
P(f[Cx, f(C, = Sz,)
P([(Cx, = -fCC,, S,,))
11

---

Under review as a conference paper at ICLR 2022
Note that we have assumed that the value of f(.) is unique. It is easily refer that the probability
of f(Cx. Sx) = f(CC.. Sz,) is equivalent to Cx CC. Sx = S2. so that
P( Sx) f(Cx; sn.)) P(Cx Cx: 5x -
( f(C2. Sa,) Sx = =2.)) P(Cx = C.: Sx = Sa,15x = Sw,)) P = =C+)
In this case
P(Sao s(C,, Sa.)) P(S,, P(F((C Sm,)) P(Cx Cz. Sx = Swa))
P(Cx =C,)
So the following formula must be true
JssJIIc.... P(Cx Ca. Sx Swx) log
P(Cx = Cz)
I(S f(C,; Sai)) JSs. (1C).
P(Cx Cz. S Sa,))
according to the conclusion of 3.1 and Thm 3.2 we can say that P(Cx Cn, Sx S..) P(Cx Ca. Sx = Say) for any index and So there must be
I I(S f(C, Se.)) = IIS f(C,, Sa,))
Similarly
f(e, Sa)) f(Ca, Sz)
About (4)(
We have shown when 11 f(Cm) Swz) there must be
x1) f(C22 Sa,)) I(S f(C21 Saz)) I(S 21,22 I(C2, D2) = f(C Szz)) = I(Cos: f(Co.: Swz)) I(Co, x1-42)
Now assume that ē142 f(Co, Swy) where Swy # Swz In this case
(1 I(S )ms)) 1
P(Smg [(Or)) Sr log P(S#2 f(Cz,, S2y))
Soz JiCo,
)(S,,) P([((()) S
Where
P(S..: Se)))
P(Sx S2,)) Sx) f(C,, Sa,) = Saz)
P(S.,) (f(Crr,,su)
P(Sx = =Saz) P(F()( Sx) f(Co,) =f Swy))
Since the probability of S(Cx,Sx) = f(Or, Sxy S1x = Swx is zero, We conclude that
f(S S2,)) while
f(C,,, P(fe)) Sw,))
P(S U.Smys((Cr [(Ony: Swa)) P(Cx = Sx = SaI) P(S P(f(C Swa)) P(Cx= =Cn)
Similarly, if assume that 31 f(Ca, Swy) where Cz, Ca, In this case
I(Cny,āt I(Can, Swa)) = 1 J
I(Cn, S,,)) (CCaif(Car Saz))
Co, S((Co)
P(C,,))([((C Saz))
12

---

Under review as a conference paper at ICLR 2022
Where
P(Caiif(Ca; Sw,)) ((f(,) Sw2))
= P =x
= Cz.)
P(Cx= = P(x Sx) = f((o.: Saz))
Then we can get the same conclusion, That is
P(Cm): ((C,, Swa)) 0 while P(C,,) ((f(Ce. Sx2))
P
(CCn: Smz)) P(Se,))((((C, Szz))
P(Cx = Car Sx = Swa)
P(Sx Swz)
Which indicates that the formula in Eq.(4) is equivalent to 31+2 f(Coz So2)
13


Reviews:
[{"review_id": "dFn4uuvlfin", "reviewer": "Reviewer_n7Fo", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n(1) In the proposed method, a single encoder is adopted to extract both timber (speaker) and content information from the reference speech, which simplifies the model architecture.\n(2) The latent speech representation is divided into two parts, for speaker and content embeddings repressively, which provides the flexibility to change the proportion of speaker and content embeddings given the total dimension of the latent representation.\n(3) Two classifiers are proposed to ensure the disentanglement of speaker and content information.  The common speaker classifier ensures the speaker embedding carries the timber information.  The adversarial speaker classifier ensures the content embedding does not contain timber information.\n(4) Vector norm is applied to the learned speaker classifier to eliminate the content information from the speaker embedding.\n(5) Code-reconstruction loss is proposed to force the content embedding not to contain the timber information.\n\nWeakness:\n(1) The method proposed by the paper is actually very simple and straightforward. The only difference between the proposed work with some state-of-the-art methods is the \"combining\" of nowadays content encoder and speaker encoder into a single encoder.  However, from the perspective view of the neural networks, the performance of a single encoder and the separate encoders should be the same.  Hence, the novelty of the proposed method is limited.\n(2) There might be some errors for the technical part.  For example, for equation (8), the adv-cls loss is subtracted from the total loss.  The minimizing of the total loss will actually increase the adv-cls loss.  However, the adv-cls loss should also be minimized to optimize the parameters (θ_{c2}) of the adversarial speaker classifier.  The optimization of the encoder (θ_e) tries to \"fool\" the adversarial speaker classifier by the gradient reversal layer only.\n(3) In addition to above (2), In the 3rd paragraph introducing code-reconstruction loss under equation 7, it is unclear how the Z_hat is derived?  If the input to the intermediate encoder (to derive Z_hat) is X', and X' is just reconstructed from the original speaker embedding (S_x) and content embedding (C_x), I do NOT think the introduction of code-reconstruction loss can solve the problem.  Or, the authors should clarify how the Z_hat (and X') is derived.\n(4) In the 2nd paragraph introducing reconstruct-loss under equation 7, it is said that \"In this case, the estimated content embedding S_x are still speaker-independent and the common classifier can still ...\".  Here, it seems that \"the estimated content embedding\" should be \"the estimated speaker embedding\"? \n(5) In the ablation study, only the effectiveness of \"code-reconstruction loss\" is experimented.  How about the other losses?\n(6) In section 4.4, the dimension of two parts of the latent variable has been experimented to validate the flexibility of the proposed method.  The question is will the selection of the dimensions affect the performance of VC and why?\n(7) The English writing of the paper should be proofread by a native speaker.  There also many topos in the paper.\n", "summary_of_the_paper": "The paper has proposed a VC system called ClsVC by learning the latent speech representation from reference speech.  To ensure the disentanglement of speaker and content inforamtion, the proposed method separates the latent speech representation into two parts, for speaker and content embedding respectively.  To ensure the performance of disentanglement, a common speaker classifier and an adversarial speaker classifier are proposed.  Furthermore, a new loss called code-reconstruction loss is proposed. Experimental results validate the effectiveness of the proposed method.", "summary_of_the_review": "Based on the above main review (especially the weaknesses of the paper), the method proposed by the paper is actually very simple and straightforward.  The novelty of the proposed method is limited.  There might also be some errors for the technical part.  The English writing of the paper needs improvement.  Hence, I do NOT think the paper should be accepted.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Vmb6Wl2Krd5", "reviewer": "Reviewer_fLnd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The four losses have been individually proposed in the past, and the paper does not try to argue why the combination of which is novel.\n\nThe experiments in Table 1 are weak and do not show any insights. The models in Table have different architectures and different training losses. It's also unclear if these models are tuned for this particular task, on this particular data set under this particular training.\n\nThe theory developed in section 3 is weak. The first and major assumption is that linguistic content can be separated from speakers. This assumption is clearly false, because different speakers clearly have preference over how sentences are constructed and have preference over how words are pronounced. The second assumption is that the content and the speaker embedding uniquely determine the speech, or in the paper x = f(C_x, S_x). This again is false, because no single person is able to produce the exact same speech utterance twice. Another assumption due to the information processing inequality I(C_x, S_x) \\geq I(C_x, x) is strong and is likely false. Based on equation (5), it really should be I(C_x, x) \\geq I(C_x, E(x)) \\geq I(C_x, S_x).\n\nThe approach has a few weaknesses. The first is about disentanglement. Even though adding a speaker classifier encourages pushes the speaker information to one embedding, it does not mean similar information is not in the other embedding unless a bottleneck is used. The gradient reversal layer also encourages similar effect, but again that the discriminator cannot identify speakers does not imply the speaker information is absent. Reconstructing the hidden vectors can also be also concerning, because the model can end up mapping everything to a constant vector if the weight of that loss is too strong. These weaknesses are not addressed in the paper.\n\nThe presentation has room for improvement. For example, listing the problems of other models in the introduction seems unnecessary, because the paper is not directly addressing these problems. The introduction also does not motivate the proposed approach. The background section is fragmented, jumping over multiple unrelated aspects. The theorems in section 3 should really be assumptions. There are also wording issues and typos throughout the paper. Below is a non-exhaustive list.\n\n> However, whether they apply a bottleneck network or vector quantify technology ...\n\nvector quantify technology --> vector quantization?\n\n> Mutual information (MI), a crucial indicator to measure ...\n\nThis sentence does not have a verb.\n\n> So our goal is to minimize ...\n\nRemove \"so\"?\n\n> Unfortunately, the truth is, when there are only ...\n\nRemove \"the truth is\"?\n\n> In this section, we will prove experimentally ...\n\nprove --> show? It's possible to use experiments to refute hypotheses, but I don't think this is the intended meaning.\n", "summary_of_the_paper": "The paper proposes a model for voice conversion. To achieve voice conversion, the approach is to separate linguistic content and speaker information into two embedding vectors. Voice conversion is achieved by swapping out the speaker embedding. There are four losses involved, one for reconstructing speech, one for reconstructing the embedding vectors, one for classifying speakers, and the last is a task to make speaker classification adversarially hard.", "summary_of_the_review": "The paper does not argue why the proposed approach is novel. The experiments do not show any insight. The theory in the paper is weak, if not wrong. Several weakness of the approach are not mentioned. The presentation has a lot of room for improvement.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Tzd88cv89Ew", "reviewer": "Reviewer_XqM2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes an approach for voice conversion. It key idea is to add classification tasks in an autoencoder approach. In particular, these classification tasks are aimed at disentangling the speaker and content representations. \n\n1. The paper seems to be  following AutoVC,  even on the details and descriptions. Here there are no separate speaker and content encoders and additional classifiers are added on top of speaker and content speakers. \n\n\n2. The paper is missing some important details. The details of the encoder and the decoder are missing. The details of the classifiers C1 and C2 are also missing. \n\n3. Why are Theorems 3.1 and 3.2 marked as “Theorem” ? Aren’t these just the assumptions/conditions under which the framework operates ? \n\n4.  How do we understand that the content embedding is capturing meaningful content related information ? The content embedding is trying to fool a speaker classifier, which it can do by producing any random output. The reconstruction losses can be claimed to avoid that but is there a way we can verify how much this holds empirically ? The paper talks about content and speaker information leakages into the embeddings. However, no attempt is made to study this empirically. \n\n5. What happens when Audio X_i and Audio X_j in Fig 1(b) are utterances of the same speaker.  How well is the reconstruction in this case. If the assumptions in the paper holds with training - then the framework should be able to reconstruct the audio well. \n\n6. Paper’s writing could be improved  substantially. Authors should proof-read the paper. There are quite a few typos as well as language errors. At times, the paper is not easy to read and understand. In several cases, the Figures (e.g 1 and 4) are not even properly discussed.\n\n7. The classifiers take the speaker/content embeddings as well the speaker identity (\\mathbf{u}) ? Is the speaker identity provided as a one-hot vector ? \n\n8. The experiments are also not very thorough and lacking in some aspects. Moreover, some basic details are also missing. First, it's not clear how the data is partitioned for experiments. The paper says the entire dataset is divided into two sets - 100 speakers for training and 9 speakers for testing ? But the “Traditional VC” has seen speakers. So how the training and testing partitioned for those experiments ? How are all the hyperparameters selected ? \n\n9. The MOS results have very high variance - close to 1 for all cases. This seems very high and makes it hard to interpret the performance. Also, what is the sample size for MOS study ? \n\n10. Its not clear what Fig 4 is trying to say. It is not properly discussed. \n\n11. For the change in dimensions, it would have been better if we could also see how the performances are affected by these changes along with the speaker embedding visualization. \n\n12. As mentioned in point 4, it would be great if some of the claims on speaker and content embeddings are empirically studied to verify how they hold after training. No interesting insight is provided in the paper.\n\n\n \n\n", "summary_of_the_paper": "This paper proposes methods for voice conversion. The proposed approach follows an encoder-decoder framework in which it adds classification tasks over the learned embeddings to enhance voice conversion. The representation space is divided into two parts - content and speaker, to enable disentangling of the content and the speaker. Experiments are shown on “traditional” and one-shot VC tasks. Objective as well as subjective scores are used to evaluate all systems. ", "summary_of_the_review": "The paper proposes a method for voice conversion which aims to use 2 classification tasks to learn content and speaker embeddings. The idea is simple and makes sense. However, the paper is not clear on several fronts. The claims are not justified properly through experiments and key details are missing as well. I think the paper in the current form is not ready for publication even though it appears to be a meaningful approach for VC. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "N0JrSKsuaIP", "reviewer": "Reviewer_DgA7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n1. The problem formulation is subtle and interpretative, given the premise theory constraints. The formal proof is also providing bonus points.\n2. The model is compared with open-sourced widely-acknowledge frameworks and outperformed them reliably.\n3. The ablation study is also a bonus point, especially analysis on latent variables.\n\nWeakness & Questions\n1. There are more than several grammatical and format-wise mistakes made in this paper. Please fix them via proof-reading, either for future publication or camera-ready version.\n2. Citation is poorly formed and arranged. Please check and re-write them.\n3. Problem formulation shall not be included as part of methodology. Please move them as a separate section or as part of the introduction.\n4. As a researcher who has some experience in speaker recognition, I question Theorem 3.2 - Embedding and Identity naturally shall not be an objective&surjective mapping. It should only be a surjective one.\n5. Does the parameters of Eq. 8 set by pilot experiments? Or from a pre-trained dataset? Not clear enough.\n6. The reviewer suggests to perform some tests on publicly available datasets for VC e.g. VC2018 challenge if applicable [1]. Does not have to be extensive but need one.\n7. One main weakness of this paper is during the evaluation time, multiple encoders are anyway also needed. Does not have to handle this for now but this shall be addressed and left as a future work.\n\n[1] http://www.vc-challenge.org/vcc2018/index.html", "summary_of_the_paper": "This paper presents a learning framework for Voice Conversion (VC) with single encoder containing both contextual and speaker information, along with some tricks on utilizing timbre and speaker-wise information in both training and inference phases. It is claimed to achieve state-of-the-art performance, outperforming most recently open-sourced models in the VC research community.", "summary_of_the_review": "In general, the author has mixed feeling about this work. On the one hand, the proposed framework is decent with clear mathematical proof (apart from Theorem 3.2, which is not valid but essentially trivial in the whole proof). However on the other hand, more than several statements are questionable, along with wording and grammatical mistakes. Besides, the speaker embedding background information is missed, along with Theorem 3.2 which is wrong in reviewer's personal knowledge.\n\nTherefore, the reviewer is recommend this paper to be on the margin. Even if it got accepted, significant work is expected to be done on re-phrasing, minor additional experiments, and speaker embedding related theoretical coverage.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a voice conversion framework, ClsVC, which is based on disentanglement of speaker and content information in some latent space.  The authors introduce two classification constraints (a common speaker classifier and an adversarial classifier) to improve the separation of the two embeddings.  Experimental results are reported on a few voice conversion tasks with objective and subjective scores.  Reviewers have reservation about the novelty of the work which is not considered overwhelmingly significant given existing techniques. The theory and arguments on the claimed effectiveness of the disentanglement of speaker and content also raise concerns, which need to be further verified.  The experimental results need to be more convincing.  Lastly, the exposition needs significant improvements.  The authors' rebuttal answers some of the comments but a few major concerns still stand.  This paper can not be accepted given its current form.
Decision: Reject