VoiceMixer: Adversarial Voice Style Mixup
Authors: Sang-Hoon Lee
Ji-Hoon Kim
Hyunseung Chung
Seong-Whan Lee
Venue: NeurIPS 2021 Poster (2021)
Keywords: Voice Conversion
Style Transfer
Speech Processing

Abstract: Although recent advances in voice conversion have shown significant improvement, there still remains a gap between the converted voice and target voice. A key factor that maintains this gap is the insufficient decomposition of content and voice style from the source speech. This insufficiency leads to the converted speech containing source speech style or losing source speech content. In this paper, we present VoiceMixer which can effectively decompose and transfer voice style through a novel information bottleneck and adversarial feedback. With self-supervised representation learning, the proposed information bottleneck can decompose the content and style with only a small loss of content information. Also, for adversarial feedback of each information, the discriminator is decomposed into content and style discriminator with self-supervision, which enable our model to achieve better generalization to the voice style of the converted speech. The experimental results show the superiority of our model in disentanglement and transfer performance, and improve audio quality by preserving content information.

VoiceMixer: Adversarial Voice Style Mixup
Sang-Hoon Leel
Ji-Hoon Kim?
Hyunseung Chung?
Seong-Wiin Whan Lee
{sh_lee, jihoon_kim, kim, hs_ hs_chung, sw ww..ee)kkorea.c ac.kr
Department of Brain and Cognitive Engineering, Korea University, Seoul, Korea Department of Artificial Intelligence, Korea University, Seoul, Korea
Abstract
Although recent advances in voice conversion have shown significant improvement, there still remains a gap between the converted voice and target voice. key factor that maintains this gap is the insufficient decomposition of content and voice style from the source speech. This insufficiency leads the converted speech containing source speech style or losing source speech content. In this paper, we present VoiceMixer which can effectively decompose and transfer voice style through novel information bottleneck and adversarial feedback. With self-supervised representation learning, the proposed information bottleneck can decompose the content and style with only small loss of content information. Also, for adversarial feedback of each information, the discriminator decomposed into content and style discriminator with self-supervision, which enable model to achieve better generalization to the voice style of the converted speech. The experimental results show the superiority of model in disentanglement and transfer performance, and improve audio quality by preserving content information.
Introduction
Voice conversion (VC) is the task of transferring the target voice style to the source speech while keeping the content information of the source speech. VC is also called voice style transfer (VST), and it shares long history with the objective to clone someone's voice. There is even potential risk of usage in crime such as a voice spoofing (Kinnunen et al. 2012), and also in various applications in entertainment (Nachmani and Wolf 2019), education (Sisman al. 2020), security (Wu and Li 2016), and voice restoring (Yamagishi et al. (012). Although deep learning made the breakthrough in the VC domain, there still remains challenging problems for real-world application such low audio quality or similarity to target voice style.
Usually, traditional VC systems require the same utterances for different speakers train properly. However, it is hard to collect such parallel data for many speakers, and extension to many-to-many VC systems becomes a laborious task. To overcome this problem, several methods have been developed. First, generative adversarial networks (GAN) based models (Kaneko and Kameoka 2018 Kaneko Kameoka al. 2018) adversarial feedback with cycle-consistent loss train with non-parallel data. However, it is hard to train these models, and they produce lacking audio quality and transfer performance. The flow-hased VC model, Blow (Serrà et al.) 2019). is also non-parallel VC model with normalizing flows using the hyperconditioning mechanism.
al. 2019 2020
Despite effort transfer the voice style in non-parallel settings, these models not able to sufficiently disentangle content and style from the source speech, and thus the converted speech
"Corresponding author
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

still contains the style of source speech. To overeome this limitation, AUTOVC (Qian et al. utilizes a simple autoencoder. The carefully designed fixed-length based information bottleneck disentangles the content and style information. For better disentanglement, IDE-VC (Yuan et al. 2021) followed the AUTOVC framework with fformation-theoretii guidance. AdaIN- (Chou et al. 2019) and AGAIN-VC (Chen et al., 2020) employs the instance normalization (Ulyanov et al. 2016) to remove the global style information. Additionally, AGAIN- makes use of the activation function as information bottleneck with small size of content embedding. However, these models have a trade-off between the audio quality and the disentanglement performance. In the process of disentanglement, the loss of content information results in low audio quality with missing linguistic information. Also, they have to find the proper size of information bottleneck heuristically.
2019)
Text transcriptions can be used to guide content embedding to learn only linguistic information (Biadsy et al. 2019 Zhang et al., 2019). These models have to be jointly trained with the text-to- speech (TTS) model to encode the linguistic information based on the attention alignment from autoregressive TTS system (Shen et al., 2018). However, they require text transcriptions for training.
Recently, self-supervised representation learning is adopted to extract important representation in speech representation learning task (Oord al. 2018, Wang et al.) 2020a). Predicting the future latent representation can make the model learn useful information without labeled data. However, such self supervisdd representation learning has not yet gotten the attention in voice conversion task.
In this paper, we present VoiceMixer, which can decompose and transfer voice style through a novel Aimilarity-based information bottleneck and adversarial feedback. We introduce self-supervised representation learning to disentangle and transfer voice style without any text transcription and additional information extracted from the external feature extractor. Self-supervised similarity based information bottleneck disentangles the content and style without effort to find the proper downsam- pling size. Also, we propose an adversarial voice style mixup learn the latent representation of the converted speech. We first disentangle the discriminator into content and style discriminator. The hidden representations of generator guide each discriminator as conditional information. Through adversarial feedback of disentangled discriminators, the generator has better generalization on the converted speech. The main contributions are as follows:
We propose the similarity- based information bottleneck with self-supervised representation learning. which can disentangle content and style with only small loss of content informa- tion. This preservation improves the audio quality of converted speech compared to previous methods.
For better generalization of the converted speech, we propose an adversarial voice style mixup. which learns the converted speech by adversarial feedback with sulf-supervised guidance, even though the converted speech does not have ground-truth audio.
Through various subjective and objective evaluations, we demonstrate that VoiceMixer has better disentanglement and transfer performance than other baselines in both many- toayyttyyyyyyyymmmmm and zero-shot voice style transfer scenarios on the real-world VCTK dataset.
2 Background
AUTOVC disentangles content and style information from the source speech, and transfers the voice style of target speech through information bottleneck (Qian et al. 2019). The simple autoencoder framework of AUTOVC consists of three modules; speaker encoder f-(.). content encoder fc(-), and decoder g(. .). During training, this model only requires self reconstruction with a fixed-length information bottleneck to disentangle the content and style information.
S - - C = (x(X XI-LA 9
(1)
Here, X1.A refers to the utterance "A" from the source speaker "1". S denotes speaker infor- mation in the speaker "1", CA denotes content information of the utterance "A", and X X 1.4 is self-reconstructed speech which contains the content information C'A and matches the speaker char- acteristics S,. Although it is a very simple way to decompose each information, a proper information bottleneck size is necessary. Formally, this can be represented as follows:
H [t/T]) Ca(i,
(2)
2

---

Real Fike
Real/ Fike
Real Fake
Festuee
Fomer
M(x+)
(+(+))
(7()))
A2D
(a) Reconstruction
(b) Conversion (c) Content discriminator
(d) Style discriminator
Figure 1 Overall framework of VoiceMixer.
where H denotes the downsampled feature for time indices / E (1....7). When the fixed-length information bottleneck size T is "too narrow", the model has higher reconstruction quality but has poor voice style transfer performance. On the other hand, when the "too wide", the model has higher voice style transfer performance but has lower reconstruction quality. In the process of separating content and style information, some content information lost even with proper bottleneck size. Therefore, missing some content information in converted voice is inevitable.
3 VoiceMixer
In this paper, we propose a similarity- information bottleneck by self-supervised representation learning. For adversarial feedback, we disentangle the discriminator to train content and style separately with self-supervised guidance. By disentangling the discriminator for each information, can be possible to train the converted speech which does not have ground-truth audio. It is worth noting again that using other supervised features (e.g., pitch contour or text transcription) help the model to disentangle each information, but model uses self-supervised representation learning without additional features. We describe the details of our architecture, similarity-basee information bottleneck, and the adversarial voice style mixup in the following subsections.
3.1 Generator
For the generator, follow the autoencoder framework of AUTOVC. As shown in Figure Ta the generator consists of a content encoder fe(-) which extracts the content embedding from speech, a speaker encoder f-(-) which extracts speaker embedding from speech, and decoder g(:-) which generates the speech from content and speaker embeddings represented in Equation
3.2 Similarity-basel information bottleneck
Unlike information bottleneck in Equation 2 we downsample the content embedding according to the similarity between the content embeddings. We assume that the content encoder produces similar content embedding from similar phoneme, and thus we downsample the adjacent phonetic information
to be mapped together. We calculate the similarity Q (q..... 4T) between content embedding
sequence C' (cI, , and shifted content embedding sequence C'hift = (C2, (C2,"* ,CT+1) as:
C, Cr+1 /p),
q. sig( [[c]]] ||c+|]|
where sig denotes the sigmoid function and p is the temperature parameter. Then, we extract the similarity- duration D (d1, .d) where d,, is cumulative sum until the similarity qi is under the average similarity, and the d, is computed again from 4++1 until time step T.
Gaussian down/up-sampling Assume that the center of same content have the largest information of that content, then we apply the Gaussian downsampling to focus attention the center. Given the content embedding to be downsampled C, duration D, and learnable range parameter o
(ai,".. (NN) like (Shen et al. 2020), we compute downsampled sequence
(h1,"* hN) as:
dn
N(t: an, o
T
Oh
2
dm:
W
Σ Σm=
N
Nit; ams 02,)²
hn
Σ
w C
(4)
m
3

---

2
Macrnio distance
ConvID
ConvID
Cc-2
C2-1
Cz-1
Cz+z
Ctek
(a)
(b)
(c)
Figure 2: (a) Fixed-length information bottleneck. (b) Similarity- based information bottleneck. (c) Context network for supervsd representation learning content embedding.
Afterwards, Gaussian upsampling TTS model following (Shen al. 2020 upscale H upsampled content sequence C (C,, (r) with the same duration of D, range parameter for upsampling o' = -oi. ,'y,, and then Ò is fed to g( generate the mel-spectrogram as:
W N(t any " 22)
N
c = Σ w
has
X1-1A = Č.)
(5)
n=l
3.3 Auxiliary losses for similarity
Contrastive loss To increase the similarity between the adjacent content embeddings, train the content encoder with self-supervised representation learning. The content embedding is fed context network fr learn a content representation illustrated in Figure 2c To train non- autoregressive manner, we utilize the masked convolutional blocks (Liu et al. 2020) to predict content embedding from the adjacent content embeddings, and the contrastive loss for positive sample defined minimize distance between predicted content embedding Ò (ê1. .rr and content embedding C:
T
(ee fr) = E
Σ
log sig(
|c, /P)
C;
T
||ll||
where sig denotes the sigmoid function and p represents the temperature parameter.
To remove style information on the content embedding in an unsupervised manner, we prevent context network to predict future representation of content embedding. While negative samples are uniformly sampled from the same utterance in (Baevski al. 2020), we only sample the k-th future content embedding negative sample prevent maximizing distance between the contents similar to each other. We maximize cosine distance between predicted content embedding and k-th future representation of content embedding, and the contrastive loss for negative sample is:
Cneg (Ses (o)
log sig(
Ci+* C |
(n)
cc+
Adversaria speaker classification To enforce speaker disentanglement on the content embedding, we apply adversarial speaker classification in a supervised manner (using speaker label y,) as:
T Σ (og (c:))
Laduse(Sc)
(
where fate denotes speaker classifier. To train the entired model jointly with fele, we use a gradient reversal layer before the content embedding is fed to fats following (Hsu et al. 2019).
3.4 Disentangled discriminator with self-supervised guidance
Unlike the previous GAN-based VC model which uses the cycle-consistency training to preserve linguistic information by two-way generation, follow the autoencoder based reconstruction method

---

Minimize distance
(attractive)
D*Gr)
Feature maps
Feature maps
DM6)
Speech
Maximize distance (repulsive)
Reconstructed Speech
81-1,
D(
Feature maps
Feature maps
D*
Minimize distance (attraction)
Converted Speech 8 82-2A
Target Speech Xza
Figure Style feature matching loss for reconstructed and converted mel-sppetrogram.
for training. For adversarial feedback, we divide the discriminator D into the content discriminator (- and style discriminator (*,-,) to disentangle content and style, respectively. To guide each discriminator for each attribute, we condition the content embedding to the content discriminator and style embedding the style discriminator self supervupe conditional information illustrated in Figure[] For the training objectives, we use the LSGAN (Mao et al. 2017) as followed:
D"; = E | L. Ca)
1 2l + | 1-11, CA)ll
(9)
+IID"(X 1A - 1 D X 1, Si)l
CA) - 1112
D' x 1-L,A,
--111 1||2]
(10)
Feature Matching for reconstruction We use the feature matching loss to train the generator, which minimizes the distance of the discriminator's features between ground truth and generated speech. For each discriminator, we use the content feature matching loss L ccntentt from the content discriminator for content and the style feature matching loss C Style from the style discriminator for style.
( = E
[X 1.4. CA) - DD CA)lli
(11)
K'
(G; =B GG (')
Σ
N [D - St)l
(12)
where and K' denote the number of blocks each discriminator, and N; is the number of features in i-th discriminator block. The total loss for reconstructed mel-spectrogram is defined as:
Lree CG (G; D D*) + Xe (G; DO) + N.C Styie (G; D*) + Ame! Emee
(13)
where Liel mean absolute error between X1,A and X1-11A
3.5 Adversarial Voice Style Mixup
By introducing the disentangled discriminator for each information, we can train the reconstructed speech for each disentangled feature. However, our goal to convert voice by disentangling source style and transferring the target style. To learn the latent representations of the converted speech, we propose an adversarial voice style mixup, which can train the converted speech by using the disentangled discriminator with self-supervised condition. Even though converted speech does not have ground-truth (GT) samples, the converted mel-spectrogram can be trained with adversarial feedback through each discriminator. It is worth noting that the model only uses a self-supervised hidden representation the generator as conditional features without any external feature extractor for conditional information. The GAN losses for the converted mel spectrogrga defined
D (D D": G)
E D 1 1-.2,4 Ca)l2
1- A
(14)
LG Gdder (G: D*) 1- CA) 11 12122A -11/2
(15)
5

---

Feature Matching for Mixup We can also use the feature matching loss for a converted speech by the disentangled discriminator. For the content representation, the feature distance of content discriminator between converted speech and source speech is minimized as:
Σ M ID
Crontene (G; =
CA) -;
(16)
For the style representation, the feature distance of style between converted and target speech is minimized as following:
K' Σ
-x S2)11a
(17)
N:
call "Attructive style loss" which minimizes the style feature distance between the same style of the same speaker. As shown in Figure we also introduce "Repulsive style loss" boss"dn to maximize the style feature distance between the different style of the converted speech and source speech as:
K' (G; - Σ
1 Si) -1(--))A-1 N
(18)
When the content encoder does not disentangle the source speaker, the converted speech may contain the style of the source speaker. Thus, this repulsive style loss restricts the converted speech from having the style of source speaker. The total loss for converted mel-spectrogram is defined as:
Leon Ca (G; D D Ac (G; D + A C stylet (G; D*) + >-- C Stylle- (G; D*) (19)
Our final objectives for the discriminators and generator are represented as:
L Ladu G); + Acan (D D);
(20)
C Lree AconLcon AposSpos( fr) AnegEneg (fer fe fr) + ddduec (udoscef)))) (21)
4 Experiment and result
We evaluated our model with the VCTK dataset, which has 46 hours audio from 109 speakers (Veaux et al., 2017). We divided the dataset into 98 speakers as base speakers for many-to-many VST and 10 speakers as the novel speakers for zero-shot VST. The base speaker is split into train and test sets. For the non-parallel dataset setting, the training set consists of different utterances for all of the speakers, and the test set consists of 25 same utterances. We transform the mel-spectrogram with 80 bins from the audio downsampled 22,050Hz. The spectrogram is inverted to a waveform by the pre-trained HiFi-GAN (Kong et al. 2020). For many-to-many VST, we randomly choose 20 speakers with equal distribution of male and female from the base speakers. For zero-shot VST, we randomly choose 10 speakers from the base speakers, and 10 speakers from the novel speakers. For each setting, single utterance selected from each speaker, and then all the possible pairs of utterances (20 x 20 400) are produced.
4.1 Implementation details
The generator consists of speaker encoder, content encoder, similarity-baa based information bottleneck. and decoder. We train the entire model jointly. The speaker embedding is extracted from the speaker encoder which has the same architecture as the reference encoder in (Skerry- Ryan et al. 2018). The source speech is fed to the content encoder consisting of a pre-net and three blocks of the multi-receptive field fusion (MRF) (Kong al.) 2020). The pre-net it two linear layers with 384 channels. The output concatenated with source speaker embedding is fed to a 1D convolutional layer with 384 channels, followed by the MRF. The MRF returns the sum of output from 384 channels of multiple convolutional layers with multiple dilations and multiple receptive fields. We use the combination of two dilations of [1, 3]. and two receptive fields of [3, 7] for the MRF, Before the features are fed to MRF, the instance normalization (IN) is applied.

---

The similarity-base based information bottleneck has two range predictors for down/up lown/up-sampling Both range predictors consist of three convolutional layers followed by linear layer with a softplus activation function, similar to (Shen et al. 2020). The range predictor for downsampling uses the similarity-ban duration as input. The range predictor for upsampling uses the same duration and downsampled content embedding as input. The content embedding is fed to both adversarial speaker classifier and contrastive encoder. The adversarial speaker classifier consists of five ID convolutional layers followed by linear layer to predict speaker identity. To remove the speaker identity content embedding, a gradient reversal layer is used before the first layer of the adversarial speaker classifier. For the contrastive encoder, channels, receptive field size of 23, and mask sizes of [5. 7, 9]. We set the as 24 (about 0.3s) which is over the average duration of consonant-vowel syllables (Steinschneider et al. 2013).
use three masked convolution blocks of (Liu et al. 2020) with 384
After the similarity- based information bottleneck, the upsampled feature concatenated with the target speaker embedding is fed to the decoder, which consists of a conditional layer, three blocks of MRF; and the linear layer. The conditional layer is single 1D convolutional layer with 384 channels. The MRF of the decoder has the same architecture with the MRF of the encoder without IN applied the feature before being fed to the MRF. Finally, the mel-spectrogram is predicted by the linear layer.
The content discriminator consists of four blocks which have a speech-side and content-condition side block following (Lee al. 202T). Each block has ID convolutional layers. The hidden representation of the condition-siin side block is added to the speech-side hidden representations of [256, 512, 1024. 1024]. The style discriminator consists of 4 blocks which have a speech-sine side block and style-condition side linear layers. Each output of the linear layer is added to the speech-side hidden representations of [256, 512, 1024, 1024). We report more details of hyperparameter Appendix
4.2 Information bottleneck alignment
To compare the similarity-basee information bottleneck using a different combination of auxiliary loss, Figure show the alignment between source speech and downsampled content embedding. The model without contrastive loss poe and Leeg) nearly shows the diagonal alignment, which implies that content embedding does not only represent content information. The model without Ladvec shows aligmment closer to attention alignment of Tacotron2 (Shen et al. (0188.. These results show that contrastive loss is more important disentangle content and speaker information than Cadvec- Without using any text transcript and target duration, our model with auxiliary losses produces an alignment similar to phonetic alignment, and shows better performance as shown in Table
200
200-
204-
17-
35
3
15
30
150
(a) w/o Cpos and Lieg
(b) w/o Ladese
VoiceMixer
(d) Phoneme alignment
Figure 4: Alignment of similarity- based information bottleneck
4.3 Evaluation metrics
Subjective metries We conduct the naturalness and similarity mean opinion score test. For the naturalness of speech (Naturalness), converted samples are evaluated by at least 20 raters on scale of 1 to 5. The Naturalness is reported with 95% contidence intervals. For the similarity of converted speech (Similarity), both converted and target speech are presented to at least 20 raters, and the raters evaluate on scale of 4. We also report the binary rating introduced in (Serrà et al. 2019).
Objective metrics We conduct three objective metrics; the equal error rate of the automatic speaker verification (ASV EER), the mel-cepstral distance (MCD13) (Kubichek) 1993), and the FO root mean square error fo). We use the pre-trained ASV model (Chung et al. 2020) trained by the large
7

---

Table 1: Many-to-many VST evaluation results.
Method
Naturalness
Similarity
ASV EER
MCD1a
RMSE
FDSD
GT
4.07++0.3
93.7%
5.5%
HiFi-GAN (Vocoder)
3.92:10.03
92.8%
7.6%
3.19
36.22
0.546
StarGAN-VC
3.48+00044
42.5%
22.7%
7.97
33.98
13.561
AGAIN-VC
3,60+0.03
52.2%
14.3%
41.39
3.185
AUTOVC (r=16)
3,65
52.1%
18.3%
44.05
5.753
AUTOYC (7=32)
3.6420.03
52.1%
14.0%
39.93
10.703
AUTOVC Ladvre (7=16)
3.03
54.6%
14.7%
39.40
6.036
Blow
3.12:10.04
33.2%
52.0%
44.55
15.112
VoiceMixer (Ours)
3.78++0.03
55.9%
12.5%
6.77
42.76
2,080
Table 2: Zero-shot VST evaluation results.
Method
Naturalness
Similarity
ASV EER
MCD13
RMSE
FDSD
GT
4.08+0.03
96.3%
4.4%
HiFi-GAN (Vocoder)
4.03++0.3
95.4%
6.0%
3.34
38.33
0.576
AGAIN-VC
3.29
58.2%
15.0%
6.96
44.81
3.261
AUTOVC (r=16)
3.40==0.03
46.3%
25.0%
6.92
46.33
5.227
AUTOVC (r=32)
3.24++0.03
59.7%
20.7%
6.65
39.93
9.741
AUTOVC Ladvse (7=16)
3.39++0.03
58.9%
21.9%
6.78
42.08
5.675
VoiceMixer (Ours)
3.75::0.03
63.4%
18.5%
7.04
44.70
2.416
scale dataset, VoxCeleb2 (Chung et al. 2018). We compute the BER from the converted and target speech (400 x 8000). We apply the dynamic time warping (DTW) to calculate the MCD1s 1s and RMSE between converted and target speech, which has different time alignment. For the naturalness mcasurement, we evaluate Fréchet DeepSpeech Distance (FDSD) (Binkowski et al.) 2020 Gritsenko et al. 2020 which is the distance between the high-level features of GT and generated audio from the pre-trained DeepSpeech2 (Amodei et al. 2016). Because DeepSpeech2 is a speech recognition model trained with connectionist temporal classification loss to classify the text sequence, the hidden representations are related to linguistic information. Thus, we use FDSD between the converted and source speech for objective naturalness measurement.
4.4 Audio quality and style transfer performance
For the many-to toayy-tt-mmmm many VST evaluation, we compared our model with four VC models; StarGAN-VC (Kameoka et al. 2018), AGAIN-VC (Chen al. 2020). Blow (Serrà al. 2019), and AUTOVC QQin et al. 2019). All models are trained on the same dataset VoiceMixer, and the implementation details are described in the Appendix A. We trained AUTOVC in various settings with each having different sizes of information bottleneck. For fair comparison, we also implemented the AUTOVC model with an adversarial speaker classifier. Table shows that our model outperforms other models in Naturalness and FDSD metrics. Our model also shows better transfer performance on the Similarity and ASV EER. When disentangling the content and style information, AGAIN- VC and AUTOVC models lose lot of content information, and thus the converted speech has lower Naturalness and higher FDSD score. The adversarial speaker classifier on the content embedding can help the disentangiement performance of models, but the naturalness can be degraded.
For the zero-shot VST evaluation, we compared our model with two VC models; AGAIN-V VC and AUTOVC shown in Table We also implemented various AUTOVC trained with different infor- mation bottleneck size and adversarial speaker classification loss. Our model has better performance in Naturalness and FDSD. In terms of similarity, even though the AGAIN-VC has higher performance ASV EER, our model has better performance in Similarity. terms of AUTOVC, is hard to select the proper down-sampling factor, which has trade-off between naturalness and similarity. Thus, is important note that our proposed similarity- -hased information bottleneck need not find the proper factor, which is learned by self-supervisee representation learning with small loss of content information. Our audio samples are available on the demo page
htttps:/anooymouust speech. speech. pappeechggihhub... sitthbb..o//oo.cciioe///neeee

---

Table 3: Ablation studies for zero-shot voice style transfer
Method
Naturalness
Similarity
ASV EER
MCD1a
RMSE,0
FDSD
VoiceMixer (Ours)
3.02
63.7%
18.5%
7.04
44.70
2.416
w/o Ladose
3.74
41.3%
32.2%
7.33
47.63
1.217
w/o Lpos and Lang
3.20
42.3%
38.0%
7.92
54.11
4.634
w/o Lnng
0.08
45.5%
24.2%
7.72
45.05
11.354
fixed-length (7=16)
3.35
51.8%
28.5%
7.38
46.14
2.827
w fixed-length (r=32)
3.28++0.03
56.8%
20.7%
6.97
44.16
2.823
w/o GAN
3.70+0.03
58.5%
20.5%
6.95
44.79
2.666
w/o disentangled discriminator
0.68
60.4%
18.3%
7.02
45.23
3.502
w/o style-
3.69+0.03
55.0%
21.6%
7.02
45.66
2.173
4.5 Ablation study
We conducted ablation studies for the information bottleneck and adversarial feedback in Table 3 We evaluate each model for the same zero-shot VST setting Table In terms of similarity-basee information bottleneck, absence of Cadvae using fixed-length information bottleneck (IB) makes harder to disentangle the content and style in content embedding. For better disentanglement, is essential to use Lyos with Lngg and it induces the information bottleneck to downsample the content embedding similar to the phoneme alignment as shown in Figure Additionally, when trained without any information bottleneck or all of the auxiliary losses, these models not able to convert any voice, but only reconstruct source speech. Using GAN makes the model have better performance all of the metries. The model trained with a single discriminator (instead of disentangled discriminator) shows lower performance in FDSD. In this regard, disentangled discriminator encourages better generalization for each attribute. Removing S Stylee- results in lower performance in both subjective and objective similarity metries even though the FDSD decreases.
4.6 Content and speaker disentanglement
We conduct speaker classification content embedding to evaluate the disentanglement per- formance compared AGAIN-VC and AU- TOVC. Table represent the classification re- sults. Our model shows the lowest accuracy, which mean our model has better disentangle- ment performance by removing speaker iden- tity on content embedding despite having the largest feature dimensions of 384 (The content embedding of AUTOVC has 64 dimensions). Even though AUTOVC trained with adversar- ial speaker classifier has comparable disentan- glement performance, AUTOVC loses lot of content information in their fixed-length based information bottleneck as shown in Table 1. AU- TOVC also has to heuristically find optimal downsampling size for a good balance between content and style. On the other hand, our pro- posed model finds proper downsampling size based the similarity of content embedding learned by self-supervised representation learning with only a small loss of content information. We additionally conducted word error rate (WER) evaluation the converted speech. The results show VoiceMixer has a lower WER of 20.92%, which support our model converts the voice with a small loss of content information. We used the Google Speech-to-Tess API for ASR model.
Table 4: Speaker classification accuracy on content embedding of the autoencoder based VC models and word error rate (WER) of converted speech.
Method
ACC
WER
GT HiFi-GAN (Vocoder)
8.18
10.12
AGAIN-VC
27.31
27.26
AUTOVC (r=16)
10.27
39.27
AUTOVC (r=32)
4.47
61.99
AUTOVC + Ladose (T=16)
3.11
39.31
VoiceMixer (Ours)
1.47
20.92
Broader Impact
We see voice conversion technology could used for various applications in many areas. First, the personalized voice systems for the entertainment industry can be utilized in game or voice chat.

---

This may help to secure the privacy of the player. Furthermore, it is also possible to read books with parents' voices for their children by converting the audiobook. The film industry can also use the system for dubbing the original actors' voice in different languages. Finally, voice conversion can also restore the voice for individuals who passed away or have lost their ability to speak by a neurological disorder or motor disabilities such as amyotrophic lateral sclerosis or spinal cord injury. However, we also acknowledge the potential harms, malicious use, and ethical concerns of voice conversion technology against the positive impacts.
Social negative impact and mitigation strategy As speech-related AI systems such as voice conversion and text-to-speech synthesize realistic audio. there is an increased risk of the potential harms, malicious use, and ethical concerns for TTS and VC systems. These systems can be used to deceive people in various ways; usage in crimes such voice spoofing (Kinnunen al. 201 2).. fake news (Singhal et al. 2019), and commercial use by cloning voice without consent (Wang et al. 2020b Also, there is potential use for fraud, theft, abuse, or harassment. Hence, it is necessary to consider a countermeasure for reducing these potential risks. The anti- anti-spoofing techniques (Wu et al. 2012) such a fake audio detection (Tak al.] [021] Mittal et al. 2020 Ma et al.. (021)) are developed to distinguish synthesized speech, and (Müller et al. (021) demonstrated that the Al outperforms the human in fake audio attacks. Moreover, recently, there is attack that uses partially fake audio clip, and some small fake audio clips are hidden in real speech audio. As (Yi et al. (021) indicated that partially fake audio detection is more challenging than fully fake audio, is also important to develop a partially fake audio detection model. For practical mitigation strategies, there are some potential approaches. First, voice conversion system should be deployed when conditioned on written consent from all parties involved (c.g., the speakers of source and target speech) if the dataset is not open-sourced or licensed to be open publicly. This can prevent inappropriate use of voice conversion without agreement from affiliated parties. If either of the two parties is dead or unavailable, there must be other measures taken such as consent from the direct family. Second, the converted speech or synthesized speech should be provided when conditioned on mandatory disclosure with the statement such as "this speech is synthesized by voice conversion or text-to-ppeech model". This can potentially prevent harmful and malicious conduct such as voice spoofing, phishing, or faking using voice conversion. Third, the user could synthesize the speech with the virtual voice by mixing the style vector from the multiple speakers where that voice doesn't 't exist in real-world. However, the user should use the data for style information, given the written consent from all of the parties. Finally, there is risk that the system trained with the leaked private data may be used to deceive people, so researcher and institution should enhance their privacy protection mechanism. To do this, (Hong eet al.l,, (021) introduced federated learning for text-to-speech. Although these mitigation strategies are still imperfect, we will continue to make improvements on mitigation strategies and dedicate thoughtful discussions on methods to prevent harmful use in our future works.
6 Conclusion
We presented VoiceMixer, which can decompose and transfer voice style by similarity based infor- mation bottleneck and adversarial feedback with self-supervised representation learning. Without effort to find the proper size of information bottleneck carefully, our model is able to learn proper downsampling factor with eeff-supervised representation learning. We successfully demonstrated that our novel information bottleneck can decompose content and style information from the source speech with a small loss of content information. Moreover, the alignment of information bottleneck is similar to phonetic alignment despite not using any text transcript and target phoneme duration. we also show the adversarial voice style mixup makes it possible to learn the latent representation of converted speech which does not have ground-truth speech, and it improves the overall generalization.
While there remains a gap between target and converted voice, we believe pre-trained speaker encoder with large-scale dataset (Jia et al.) 2018) could improve the VST performance. Moreover, we see our self-supervised learning based speech disentanglement extending to other tasks. For future work, we will apply our speech disentanglement to TTS without text (Dunbar et al.] 2019: TTan et al. 2021), which is a challenging task to synthesize the speech without any text transcript for use of the untranscribed large-scale speech data. We will also attempt to control the rhythm of speech like
(Qian et al. 2020 2021). We discussed the potential positive and negative impact of voice conversion in section 5 We hope that our method will be used in positive applications or extend to technical mitigation strategy such as fake audio detection.
10

---

Acknowledgements
This work was partly supported by Institute of Information communications Technology Planning & Evaluation (ITPP) grant funded by the Korea government (MSIT) (No. 2019-0-00079, Artificial In- telligence Graduate School Program (Korea University) and No. 2021-0-02068, Artificial Intelligence Innovation Hub) and Microsoft Research Asia (MSRA).
References
Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al. Deep speech 2: End- to-end speech recognition in english and mandarin. In International Conference Machine Learning, pages 173-182. PMLR, 2016.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-super learning of speech representations. Advances in Neural Information Processing Systems, 33, 2020.
Donald J Berndt and James Clifford. Using dynamic time warping to find patterns in time series, In KDD workshop, volume 10, pages 359-33 3700. Seattle, WA, USA:, 1994.
Fadi Biadsy, Ron J Weiss, Pedro J Moreno, Dimitri Kanevsky, and Ye Jia. Parrotron: An end-to-end speech-to-speed conversion model and its applications hearing -inriiir speech and speech separation. arXiv preprint arXiv: 904.04169, 2019.
Mikolaj Binkkwski, Jeff Donahue, Sander Dieleman, Aidan Clark, Erich Elsen, Norman Casagrande, Luis C. Cobo, and Karen Simonyan. High fidelity speech synthesis with adversarial networks. In International Conference on Learning Representations, 2020.
Weicheng Cai, Jinkun Chen, and Ming Li. Exploring the encoding layer and loss function end-to- end speaker and language recognition system. In Proc. Odyssey 2018 The Speaker and Language Recognition Workshop, pages 74 81, 2018.
Yen-Hao Chen, Da-Yi Wu, Tsung-Han Wu, and Hung-yi Lee. Again-ve: A one-shot voice conversion using activation guidance and adaptive instance normalization, arXiv preprint arXiv:2011.00316 2020.
Ju-chieh Chou, Cheng-chieh Yeh, and Hung-yi Lee. One-shot voice conversion by separating speaker and content representations with instance normalization. arXiv preprint (XX400900400742 2019.
Joon Son Chung, Arsha Nagrani, and Andrew Zisserman. Voxceleb2: Deep speaker recognition.
Proc. Interspeech 2018, pages 1086-1090, 2018
Joon Son Chung, Jaesung Huh, Seongkyu Mun, Minjae Lee, Hee Soo Heo, Soyeon Choe, Chiheon Ham, Sunghwan Jung, Bong-Jin Lee, and leksang Han. In defence of metric learning for speaker recognition. In Proc. Interspeech 2020, pages 2977-2981, 2020.
Ewan Dunbar, Robin Algayres, Julien Karadayi. Mathieu Bernard, Juan Benjumea, Xuan-Nga Cao, Lucie Miskic, Charlotte Dugrain, Lucas Ondel, Alan W. Black, Laurent Besacier, Sakriani Sakti, and Emmanuel Dupoux. The Zero Resource Speech Challenge 2019: TTS Without T. In Proc. Interspeech 2019, pages 1088-1092, 2019. doi: 00.21437//nerrpechh.299999999
Alexey A Gritsenko, Tim Salimans, Rianne van den Berg, Jasper Snock, and Nal Kalchbrenner. spectral energy distance for parallel speech synthesis. In Advances in Neural Information Processing Systems, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
Zhenhou Hong, Jianzong Wang, Xiaoyang Qu, Jie Liu, Chendong Zhao, and Jing Xiao. Federated learning with dynamic transformer for text to speech. arXiv preprint arXfv:2007.08795, 2021.

---

Wei-Ning Hsu, Yu Zhang, Ron J Weiss, Yu-An Chung. Yuxuan Wang, Yonghui Wu, and James Glass. Disentangling correlated speaker and noise for speech synthesis via data augmentation and adversarial factorization. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5901-5905 IEFF, 2019.
Ye Jia, Yu Zhang, Ron Weiss. Quan Wang, Jonathan Shen, Fei Ren, Patrick Nguyen. Ruoming Pang. Ignacio Lopez Moreno, Yonghui Wu, et al. Transfer learning from speaker verification to multispeaker text-to-speech synthesis. Advances in Neural Information Processing Systems, 31, 2018.
Hirokazu Kameoka, Takuhiro Kaneko, Kou Tanaka, and Nobukatsu Hojo. Stargan-vc: Non-parallel many-to-many voice conversion using star generative adversarial networks. In 20/8 IEEE Spoken Language Technology Workshop (SLT), pages 266-273. IEEE, 2018.
Takuhiro Kaneko and Hirokazu Kameoka. Cyclegan-ve: Non-parallel voice conversion using cycle- consistent adversarial networks. In 2018 26th European Signal Processing Conference (EUSIPCO), pages 1100-2104. IEEE, 2018.
Takuhiro Kaneko, Hirokazu Kameoka, Kou Tanaka, and Nobukatsu Hojo. Cyclegan-vc2: Im- proved cyclegan-based non-parallel voice conversion. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6820-6824, IEEE, 2019,
Takuhiro Kaneko, Hirokazu Kameoka, Kou Tanaka, and Nobukatsu Hojo. Cyclegan-ve3: Examining and improving cyclegan-ves for mel-spectrogram conversion. arXiv preprint 16X6, 2020.
Tomi Kinnunen, Zhi-Zheng Wu, Kong Aik Lee, Filip Sedlak, Eng Siong Chng, and Haizhou Li. Vulnerability of speaker verification systems against voice conversion spoofing attacks: The case of telephone speech. In ICASSP 2012-2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4401-4404. IEEE, 2012.
Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 17022-17033, 2020.
Robert F Kubichek. Mel-cepstral distance measure for objective speech quality assessment. In Pro- ceedings of IEEE Pacific Rim Conference on Communications Computers and Signal Processing, volume 1, pages 125-128, 1993.
Sang- Hoon Lee, Hyun-Wook Wook Yoon, Hyeong- Rae Noh, Ji-Hoon Kim, and Seong- Whan Lee. Multi- spectrogan: High-diversity and high-fidelity spectrogram generation with adversarial style com- bination for speech synthesis. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
Alexander H Liu, Yu-An Chung, and James Glass. Non-autoreereessvve predictive coding for learning speech representations from local dependencies. arXiv preprint arXiv:201 1.000.1..0040 1.00406, 2020.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International net.iiiwwweeeeeeeemmm wtwp://viiii.eeee/ttmumuiaaeaiii htt//fooummidd
Conference on Learning Representations, 2019. URL
BYy
Youxuan Ma. Zongze Ren, and Shugong Xu. Rw-resnet: novel speech anti-spoofing model using raw waveform. arXiv preprint arXiv:2108.03682 2021.
Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision, pages 2794-2802, 2017.
Trisha Mittal, Uttaran Bhattacharya, Rohan Chandra, Aniket Bera, and Dinesh Manocha. Emotions don't lie: An audio-visual deepfake detection method using affective cues. In Proceedings of the 28th ACM international conference multimedia, pages 2823-2832, 2020.
12

---

Nicolas M Müller, Karla Markert, and Konstantin Böttinger. Human perception of audio deepfakes. arXiv preprint N.XV:2207.00667, 2021.
Eliya Nachmani and Lior Wolf. Unsupervised singing voice conversion. pages 2583-2587, 2019.
Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. Voxceleb: A large-scale speaker identifica- tion dataset. Proc. Interspeech 2017, pages 2616-2620, 2017.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.037453 2018.
Jongseok Park, Kyubyong Kim.
2019.
Kaizhi Qian, Yang Zhang, Shiyu Chang, Xuesong Yang, and Mark Hasegawa-Johnson. Autove: Zero-shot voice style transfer with only autoencoder loss. In International Conference on Machine Learning, pages 5210-5219. PMLR, 2019.
Kaizhi Qian, Yang Zhang, Shiyu Chang, Mark Hasegawa-Johnson, and David Cox. Unsupervised speech decomposition via triple information bottleneck. In International Conference on Machine Learning, pages 7836-7846. PMLR, 2020.
Kaizhi Qian, Yang Zhang. Shiyu Chang, Jinjun Xiong, Chuang Gan, David Cox, and Mark Hasegawa- Johnson. Global prosody style transfer without text transcriptions. In International Conference on Machine Learning, pages $650-8660 PMLR, 2021.
Joan Serrå, Santiago Pascual, and Carlos Segura. Blow: a single-scale hyperconditioned flow for non-parallel raw-audio voice conversion. arXiv preprint arXiv: 1906.00994 2019.
Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al. Natural tts synthesis by conditioning wavenet mel spectrogram predictions. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4779-4783 IEFE, 2018
Jonathan Shen, Ye Jia, Mike Chrzanowski, Yu Zhang, Isaac Elias, Heiga Zen, and Yonghui Wu. Non- attentive tacotron: Robust and controllable neural tts synthesis including unsupervised duration modeling. arXiv preprint arXiv:2010.04330 2020.
Shivangi Singhal, Rajiv Ratn Shah, Tanmoy Chakraborty, Ponnurangam Kumaraguru, and Shin ichi Satoh. Spotfake: multi-modal framework for fake news detection. In 2019 IEEE fifth interna- lional conference mullimedia big data (BigMM), pages 39-47. IEEE, 2019.
Berrak Sisman, Junichi Yamagishi, Simon King, and Haizhou Li. An overview of voice conversion and its challenges: From statistical modeling to deep learning. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:1 29:132-157, 2020.
RJ Skerry-Ryan, Eric Battenberg, Ying Xiao, Yuxuan Wang, Daisy Stanton, Joel Shor, Ron Weiss, Rob Clark, and Rif A. Saurous. Towards end-to-end prosody transfer for expressive speech synthesis with tacotron. In International Conference on Machine Learning, pages 4700 4709, 2018.
Mitchell Steinschneider, Kirill V. Nourski, and Yonatan I. Fishman. Representation of speech in human auditory cortex: Is it special? Hearing Research, 305:57-73, 2013. ISSN 0378-5955. doi: https://doi.or//11 1e..110.1116.6... ht1l//..eaare.2213... Directions and Perspectives.
Communication Sounds and the Brain: New
Hemlata Tak, Jose Patino, Massimiliano Todisco, Andreas Nautsch, Nicholas Evans, and Anthony Larcher. End-to-end anti-spoofing with rawnet2. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6369-6373. IEEE, 2021.
Xu Tan, Tao Qin, Frank Soong, and Tie- Yan Liu. A survey on neural speech synthesis. arXiv preprint arXiv:2/06. 15561, 2021.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08000 2016.
13

---

Christophe Veaux, Jumichi Yamagishi, Kirsten MacDonald, et al. Superseded vetk corpus: English multi-speaker corpus for estr voice cloning toolkit, 2017.
Li Wan, Quan Wang, Alan Papir, and Ignacio Lopez Moreno. Generalized end-to-end loss for speaker verification. In 2018 IEEE International Conference Acouslics, Speech and Signal Processing (ICASSP), pages 4879 4883. IEEE, 2018.
Luyu Wang, Kazuya Kawakami, and Aaron van den Oord. Contrastive predictive coding of audio with adversary. Proc. Interspeech 2020, pages 826 826-830, 2020a.
Run Wang, Felix Juefei-Xu, Yihao Huang, Qing Guo, Xiaofei Xie, Lei Ma, and Yang Liu. Deepsonar: Towards effective and robust detection of ai-synthesized fake voices. In Proceedings of the 28th ACM International Conference on Multimedia, pages 1207-1216, 2020b.
Zhizheng Wu and Haizhou Li. On the study of replay and voice conversion attacks text-dependent speaker verification. Multimedia Tools and Applications, 75(9):5311 (119:)511--3327, 2016.
Zhizheng Wu, Eng Siong Chng, and Haizhou Li. Detecting converted speech and natural speech for anti-spoofing attack in speaker recognition. Thirteenth Annual Conference of the International Speech Communication Association, 2012.
Junichi Yamagishi, Christophe Veaux, Simon King, and Steve Renals. Speech synthesis technologies for individuals with vocal disabilities: Voice banking and reconstruction. Acoustical Science and Technology, 33(1):1-5, 2012.
Jiangyan Yi, Ye Bai, Jianhua Tao, Zhengkun Tian, Chenglong Wang. Tao Wang, and Ruibo Fu. trlt-truth: A partially fake audio detection dataset. arXiv preprint arXiv:2104.330 2021.
Siyang Yuan, Pengyu Cheng, Ruiyi Zhang, Weituo Hao, Zhe Gan, and Lawrence Carin. Improving zero-shot voice style transfer via disentangled representation learning. International Conference on Learming Representations, 2021.
Jing- Xuan Zhang, Zhen-Hua Ling, and Li-Rong Dai. Non-parallel sequence-to-sequene voice conversion with disentangled linguistic and speaker representations. /EEE/ACM Transactions on Audio, Speech, and Language Processing, 28:540-555 2019.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section S 6
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Section 1
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] See Section
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as URL)? [Yes] We attach the training and inference code at the supplemental material
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4 and Appendix A
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix A
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [Yes] See Appendix C
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [Yes] See Appendix
15


Reviews:
[{"review_id": "I_RQNkpLVkK", "reviewer": "Reviewer_LWwC", "summary": "This paper describes a few new additions to voice conversion. These include the 1) use of contrastive self-supervision and 2) and adversarial training component to disentangle content and style.  The impact of these modifications are verified on VCTK demonstrating improvements over AUTOVC -- a reasonable comparison.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall this paper is well motivated and the contributions are mostly well described.  Comments below mostly aim to improve the presentation of the work, but in some cases look for clarity in the presentation.\n\nLines 45-48: Also \"Parrotron: An End-to-End Speech-to-Speech Conversion Model and its Applications to Hearing-Impaired Speech and Speech Separation\" Biadsy et al 2019\n\nLine 89: \"...some content information is lost even with proper bottleneck size. Therefore, missing some content information in converted voice is inevitable.\" Why is this loss inevitable? Certainly there is some bottleneck size that does not lose content information.\n\nLine 95: why is \"pitch contour\" considered to be a \"supervised feature\"? F0 is a quality of the acoustic signal that can be extracted without human supervision.\n\nLine 106: without phoneme information, how do you know that this is corresponding to phoneme boundaries?\n\nLine 111: \"cumulative sum\" -> cumulative sum of what?  \nLine 111: is N the same as D?\nLine 112: \"average similarity\" -> over what unit? average over the utterance? average over the batch? average over the whole dataset?\n\nSection 3.3 Why is it important to train in a non-autoregressive manner?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Voice conversion -- making one speaker sound like another -- with minimal supervision has privacy and disinformation implications.  These are not addressed in the paper or mitigated by the approach.  ", "review_text": "", "extra_scores": {"needs_ethics_review": "Yes", "ethics_review_area": ["Inappropriate Potential Applications & Impact  (e.g., human rights concerns)"], "time_spent_reviewing": "2"}, "extra_text": {"ethical_concerns": "Voice Conversion's potential negative societal impact is described in https://neurips.cc/public/EthicsGuidelines under\n\n7. Deceive people in ways that cause harm. For example: could the approach be used to facilitate deceptive interactions that would cause harms such as theft, fraud, or harassment? Could it be used to impersonate public figures to influence political processes, or as a tool of hate speech or abuse?\n\n\"In summary, we expect NeurIPS submissions to include discussion about potential harms, malicious use, and other potential ethical concerns arising from the use of the proposed approach or application. We also expect authors to include a discussion about methods to mitigate such risks. Moreover, authors should adhere to best practices in their handling of data. Whenever there are risks associated with the proposed methods, methodology, application or data collection and data usage, authors are expected to elaborate on the rationale of their decision and potential mitigations. Submissions will be evaluated also in terms of the depth of such ethical reflections.\"\n\nFollowing these guidelines, this paper should include a discussion of these risks and mitigation strategies. "}}, {"review_id": "EzBKxb6IQiD", "reviewer": "Reviewer_T6dF", "summary": "This paper presents a new method for voice conversion. In particular, it overcomes limitations in previous methods by focusing on the disentanglement of content and style information through several new losses and similarity measures.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall the ideas in the paper were quite good, and the results reflect that. I was impressed with the quality of the voice conversion results in the supplementary results, which are largely artifact free and better than the baselines.\n\nI particularly liked the idea of replacing the fixed-length bottleneck found in AutoVC and other methods with a learned similarity based bottleneck. Finding the right bottleneck can be a big pain point when working with those systems, so this is a welcome contribution. With that in mind, Section 3.2 reasonably describes the similarity based bottleneck given the assumptions that content embeddings are similar to those around them and that the most important part of an utterance is generally in the middle. \n\nThe contrastive losses in section 3.3 are also reasonable. A question for the authors, for the negative loss in eq 7, is it possible that the (i+k) content is similar to the ith content (for example a word could be repeated)? In that case this loss would be maximizing the distance between two embeddings that should actually be similar.\n\nThe use of the style and content discriminators further improves the performance of the generator and make it more aware of the disentanglement between the two pieces of information.\n\nThe experiment section is particularly thorough, with a wide variety of qualitative and quantitative comparisons including content alignment, mean opinion scores, and ablation studies. The classification experiment in section 4.6 is particularly compelling to show that the style-content disentanglement of the proposed method is better than the baselines.\n\nThe downsides of the paper: Firstly, the work is largely based on AutoVC which is acknowledged in the introduction. The high level formulation of a content and style encoder with a generator (left two panels of figure 1) follow directly from AutoVC. Additionally, one of the main benefits of AutoVC was how simple the training objectives and losses were, whereas VoiceMixer has far more objectives and heuristics built into the training. However, the strong experimental section seem to show that the additional loss functions are justified and provide sufficient novelty.\n\nOne more note, the paper \"Unsupervised Speech Decomposition via Triple Information Bottleneck\" by Qian et al. also shows improvement on AutoVC with carefully crafted information bottlenecks for pitch and rhythm. It would be useful to mention this in the related works. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The societal impact is not sufficiently addressed, except for a short paragraph in the introduction. The paper does not contain a broader impact statement even though it is required, and this work in particular has many potential negative applications such as phishing attacks or cloning voices without individuals' consent.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "8rHO_51NhPA", "reviewer": "Reviewer_SxHX", "summary": "This paper is concerned with many-to-many and zero-shot voice conversion. In particular, this paper focuses on the learning of disentangled representation learning to improve the conversion quality. To address this, this paper proposes a framework called VoiceMixer, which can efficiently separate the content and speaker style by using an information bottleneck and adversarial feedback. For the information bottleneck, self-supervised learning is used to find and bundle similar content frames in a data-driven manner. For adversarial feedback, two discriminators, i.e., content- and style-conditional discriminators, are used to learn content- and style-specific representations, respectively. The experimental results on the VCTK dataset demonstrate that the proposed method outperforms some baselines in terms of objective and subjective evaluations.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\n\nThe main technical contributions of this paper are twofold:\n1. A similarity-based information bottleneck. As described in Figure 2, this information bottleneck is different from that of the previous study (i.e., AutoVC). The previous one chunks frames in a fixed length, whereas the proposed one chunks frames based on the similarity. In particular, this similarity is learned in a data-driven manner in a self-supervised way; therefore, no supervision is required.\n2. Adversarial voice style mixup. To learn the disentangled representations, content- and style-conditional discriminators are introduced. In addition, the feature matching loss is used to make the same contents close. The attractive and repulsive adversarial losses are used to encourage the same speakers and different speakers to approach and leave, respectively. All losses are designed in a self-supervised learning framework; therefore, no supervision is required.\n\nTo the best of my knowledge, these two techniques are new, and I consider that this work has sufficient originality.\n\nQuality:\n\nIn the experiments, the proposed method is evaluated using multiple metrics, including subjection evaluation on naturalness and similarity, and objective evaluation on ASV EER, MCD, RMSEf0, and FDSD. The proposed method is compared to multiple baselines, including StarGAN-VC, AGAIN-VC, AUTOVC, and Blow. In addition, the ground-truth and synthesized-and-analyzed speeches are provided; hence, we can know the anchor performance. Ablation studies are also provided. From the results, we can understand the effectiveness of each component. Information bottleneck alignment is visualized in Figure 4. Although it would be better to provide more samples in the supplementary material, the presented results are helpful for understanding how a similarity-based information bottleneck works.\n\nOverall, I think that the authors evaluate the proposed method comprehensively. However, I have some concerns.\n- In the discussion of the results (Section 4.4), the authors pick up the metrics (i.e., naturalness, similarity, ASV EER, and FDSD) that are favorable for the proposed method and ignore the metrics (i.e., MCD and RMSEf0) that are favorable for some baselines. I think the description is unfair and the authors should describe both merits and demerits clearly.\n- I am not sure whether calculating FDSD between source and converted speeches is reasonable. Another possible calculation method is to calculate FDSD between converted and target speeches. When cFDSD (conditional FDSD) is used, the first calculation method is relatively reasonable because it is possible to calculate the distance conditioned on the linguistic features. In contrast, FDSD (unconditional FDSD) is used, it is required to handle speaker difference and linguistic difference simultaneously. I suspect that when FDSD is used, the latter calculation method (i.e., FDSD between converted and target speeches) is an available alternative metric.\n- In the experiments, StarGAN-VC (version 1) is applied to mel-cepstrum conversion, and the WORLD vocoder is used to synthesize waveform. However, the improved variant called StarGAN-VC2 [A] has been proposed. In addition, the techniques to apply CycleGAN-VC to mel-spectrogram conversion have been also presented [Kaneko et al., 2020][B]. As the same neural vocoder (i.e., HiFi-GAN) is used except for StarGAN-VC, the combination of StarGAN-VC2-based mel-spectrogram conversion and HiFi-GAN is more appropriate as a baseline.\n  - [A] Kaneko et al., “StarGAN-VC2: Rethinking conditional methods for StarGAN-based voice conversion,” Interspeech 2019.\n  - [B] Kaneko et al., “MaskCycleGAN-VC: Learning Non-Parallel Voice Conversion with Filling in Frames,” ICASSP 2021.\n- On the demo page, the volumes of audio samples sound different. However, it is known that the volume difference influences scoring. The authors should normalize the audio clips before subjective evaluation.\n\nClarity:\n\nOverall, this paper is well written and easy to follow. An exception is Section 4.2, where the network architectures are explained with just words over two-thirds of a page. Although the detailed figures are provided in the appendix, it would be better to devise this part to improve the readability.\n\nSignificance:\n\nThe significance of the proposed method is demonstrated through the comparative study and ablation study. The visualized results in Figure 4 demonstrate the effectiveness of the similarity-based information bottleneck well.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- Voice conversion technique can be used for creating deepfakes. This is a potential negative social impact of this study.\n- As I mentioned above, the proposed method deteriorates MCD and RMSEf0 compared with some baselines. This is a clear demerit of the proposed method.\n- This paper focuses on converting speeches in a frame-by-frame manner, and the proposed method cannot be used to convert global speaker characteristics, such as rhythm and prosody. This is also a limitation of the proposed method.\n- I suspect that the proposed method requires larger-scale training data to learn sufficiently disentangled representations, comparing with StarGAN-VC that can learn using a relatively small amount of data (e.g., five min. utterances multiplied by four persons). I suspect that the proposed method cannot learn using such a small amount of data.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}, {"review_id": "8LF8SrtUZQu", "reviewer": "Reviewer_yW2U", "summary": "This paper adds many new loss terms to the AutoVC voice conversion algorithm, resulting in improved performance.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\nVoiceMixer achieves better perceptual results than four state of the art voice conversion algorithms.\n\nThe comparison of MCD, RMSEF0, and FDSD is enlightening: AutoVC is better on the former two measures, but VoiceMixer is better on the latter measure.  This result suggests that VoiceMixer's similarity-based bottleneck layer, and its masked-reconstruction hidden layer, are better than AutoVC's regularly-spaced bottleneck at maintaining phoneme identity through the conversion, possibly at the expense of some of the non-phonemic spectral detail.  Perceptual results suggest that the phonemic detials maintained by VoiceMixer are of more importance to human listeners than the non-phonemic details  that it loses.\n\nWeaknesses\n\nThe notations X1, X1->1, and so on are introduced without definition.  I imagine that this paper would be impossible to follow if one did  not already know the voice conversion literature quite well.\n\nThere are a very large number of loss functions, presented with only brief justification.  One  has  the sense that these loss terms are collected from every previous publication on voice conversion, without separate theoretical justification.  The masked convolutions are from (Liu et al., 2020); adversarial speaker classification is from (Hsu et al., 2019); the GAN parts are from (Mao et al., 2017).   Only the similarity-based bottleneck and the voice style mixup are apparently new to this paper.\n\nThe style- loss seems theoretically incorrect.  I understand that you want X1 and X1->2 to sound like different people, but Eq. (18) doesn't measuring the difference between the two voices -- it measures the difference between their discriminator outputs.  Since both discriminators should output 1, the difference between them should be 0.  \n\nAblation studies are very hard to follow; it seems that each one of the loss terms has an impact on all of the different performance metrics.  Does that  mean that  the loss terms work together in some way?\n\n\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Voice conversion is an extremely active research field, with significant potential societal impacts, both good and ill.  Authors have listed some of the potential impacts in the paper, both good and ill.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: UPDATE: The revision was reviewed and paper accepted.

----

This paper was discussed at length between the SACs, ACs, ethics reviewers, ethics review chairs, and program chairs.  In the end, a decision was made to conditionally accept the paper.  The list of conditions for acceptance is as follows:

1. Meaningful broader impacts statement. Moving beyond superficial discussion of obvious harms into a more detailed and thorough reflection on ethical issues, especially possible broader impact and potential for misuse. 

2. Restricted release of model through some type of licensing or form-restricted access (ie. private repo accessed via request, model code and data use restricted by licenses, etc.)

3. Discussion of possible theoretical and practical mitigation strategies for minimizing harm of such technologies in the future. If this is not possible to discuss, include a clear articulation of the limits of such models in the absence of mitigation approaches. It is not necessary to implement the mitigation strategies discussed though some current work in this area should be highlighted by authors in the main text. 

The original meta-review from the AC follows.

---

The reviewers have found this work technically sound overall, with a solid experimental section and thorough evaluation. The proposed approach is quite complex relative to the natural baseline (AutoVC), but the convincing experimental results adequately justify this additional complexity.

With regards to ethics however, there were some major concerns. I personally believe voice conversion has legitimate applications, and I do not think this work should be rejected simply because it genuinely improves on the state of the art in an area that is unfortunately fraught with potential misuse. It is imperative, however, that the authors appreciate the ethical implications of their work, and that it is correctly framed in this light in the manuscript.

The authors have responded to the ethics reviewers' concerns with a draft paragraph to include in the revised manuscript. The proposed changes include an overview of various potential nefarious uses, as well as a discussion of some potential countermeasures. Nevertheless, this has left some reviewers unconvinced. 

After giving this a lot of thought however, I think this should be an opportunity for the authors to genuinely engage with the potential impact of their work, and therefore I prefer a positive approach. I would like to recommend acceptance, but this will be conditional on a further expansion of this section of the manuscript, demonstrating an exploration of the larger ethical concerns and implications of inappropriate use, as well as a clear, objective and exhaustive analysis showing that the benefits of publishing this work outweigh the risks. Please refer to the ethics reviewers' comments for further details on what should be included.
Decision: Accept (Poster)